Posts made by dgmcdona

dgmcdona

@ben20ben I see that they merged the relevant pull request into the master branch 3 days ago, so if you're running XO from the sources, and you do an update, the patch should be applied.

dgmcdona

I don't, but I can get one set up for Monday. Thanks!

dgmcdona

I think the eth5 NIC has the correct properties, the only thing that I'm unclear on is whether it needs an IP address itself or if xenbr5 having the IP address is good enough. When I look at each host's network information in Xen Orchestra, I do see an IP address assigned to eth5.

We tried to start with as clean of a slate as possible. We restarted all of the physical hosts, restarted xo-server, reloaded the sdn-controller plugin, recreated the vxlan, and then recreated the VMs, and we still have the same problem with the traffic intended for the vxlan flowing through eth2. Looking at the logs, at the exact moment that we attempt to create the vxlan, we see the following two types of errors, for all of the hosts:

Sep 18 03:00:18 XenO xo-server[37374]: 2020-09-18T08:00:18.611Z xo:xo-server:sdn-controller:ovsdb-client ERROR No bridge found for network { network: 'vxlan', host: 'cyrange2' }
Sep 18 03:00:18 XenO xo-server[37374]: 2020-09-18T08:00:18.611Z xo:xo-server:sdn-controller:ovsdb-client ERROR No result for select {
Sep 18 03:00:18 XenO xo-server[37374]:   columns: [ '_uuid', 'name' ],
Sep 18 03:00:18 XenO xo-server[37374]:   table: 'Bridge',
Sep 18 03:00:18 XenO xo-server[37374]:   where: [ [ 'external_ids', 'includes', [Array] ] ],
Sep 18 03:00:18 XenO xo-server[37374]:   host: 'cyrange5'
Sep 18 03:00:18 XenO xo-server[37374]: }

dgmcdona

@BenjiReis We have corrected our configuration so that eth5 is on all of the hosts, and has an adequate MTU (9000). It does have a static IP address on xenbr5, and I believe it satisfies the requirements you listed. Here are the network interfaces for one of the hosts, which has a vxlan and a VM connected to both the private network and another network.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether f4:03:43:cb:09:e0 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:30:e0:3b:d5:90 brd ff:ff:ff:ff:ff:ff
4: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether f4:03:43:cb:09:e8 brd ff:ff:ff:ff:ff:ff
5: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 80:30:e0:3b:d5:91 brd ff:ff:ff:ff:ff:ff
6: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:30:e0:3b:d5:92 brd ff:ff:ff:ff:ff:ff
7: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether 80:30:e0:3b:d5:93 brd ff:ff:ff:ff:ff:ff
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 76:8b:d6:46:ba:ec brd ff:ff:ff:ff:ff:ff
9: xenbr5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether f4:03:43:cb:09:e8 brd ff:ff:ff:ff:ff:ff
    inet 10.30.0.12/24 brd 10.30.0.255 scope global xenbr5
       valid_lft forever preferred_lft forever
10: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:30:e0:3b:d5:90 brd ff:ff:ff:ff:ff:ff
11: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:30:e0:3b:d5:92 brd ff:ff:ff:ff:ff:ff
    inet 137.30.124.82/24 brd 137.30.124.255 scope global xenbr2
       valid_lft forever preferred_lft forever
12: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether f4:03:43:cb:09:e0 brd ff:ff:ff:ff:ff:ff
13: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:30:e0:3b:d5:93 brd ff:ff:ff:ff:ff:ff
14: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
20: xapi5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:30:e0:3b:d5:90 brd ff:ff:ff:ff:ff:ff
25: xapi2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:c1:41:78:32:6f brd ff:ff:ff:ff:ff:ff
26: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 5a:d3:89:b8:7b:8f brd ff:ff:ff:ff:ff:ff
27: vif2.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
28: vif2.1: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1550 qdisc mq master ovs-system state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

And here are the interfaces from within the VM:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 9e:e9:1a:4e:e4:ba brd ff:ff:ff:ff:ff:ff
    inet 137.30.122.109/24 brd 137.30.122.255 scope global dynamic eth0
       valid_lft 4696sec preferred_lft 4696sec
    inet6 fe80::9ce9:1aff:fe4e:e4ba/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    link/ether f6:ce:ed:a7:18:68 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.32/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::f4ce:edff:fea7:1868/64 scope link
       valid_lft forever preferred_lft forever

Even after configuring eth5 on all of the hosts, the VxLAN traffic continues to be routed through eth2, and I don't see any traffic flowing through the xapi interface at all. There are a lot of errors appearing related to openvswitch in the Xen Orchestra logs as well:

Sep 17 12:51:17 XenO xo-server[25920]: 2020-09-17T17:51:17.324Z xo:xo-server:sdn-controller:ovsdb-client ERROR No bridge found for network { network: 'vxlan', host: 'cyrange4' }
Sep 17 12:51:17 XenO xo-server[25920]: 2020-09-17T17:51:17.326Z xo:xo-server:sdn-controller:ovsdb-client ERROR No result for select {
Sep 17 12:51:17 XenO xo-server[25920]:   columns: [ '_uuid', 'name' ],
Sep 17 12:51:17 XenO xo-server[25920]:   table: 'Bridge',
Sep 17 12:51:17 XenO xo-server[25920]:   where: [ [ 'external_ids', 'includes', [Array] ] ],
Sep 17 12:51:17 XenO xo-server[25920]:   host: 'cyrange2'
Sep 17 12:51:17 XenO xo-server[25920]: }
Sep 17 12:51:17 XenO xo-server[25920]: 2020-09-17T17:51:17.326Z xo:xo-server:sdn-controller:ovsdb-client ERROR No bridge found for network { network: 'vxlan', host: 'cyrange2' }
Sep 17 12:51:18 XenO xo-server[25920]: 2020-09-17T17:51:18.691Z xo:xo-server:sdn-controller:ovsdb-client ERROR No result for select {
Sep 17 12:51:18 XenO xo-server[25920]:   columns: [ '_uuid', 'name' ],
Sep 17 12:51:18 XenO xo-server[25920]:   table: 'Bridge',
Sep 17 12:51:18 XenO xo-server[25920]:   where: [ [ 'external_ids', 'includes', [Array] ] ],
Sep 17 12:51:18 XenO xo-server[25920]:   host: 'cyrange1'
Sep 17 12:51:18 XenO xo-server[25920]: }
Sep 17 12:51:18 XenO xo-server[25920]: 2020-09-17T17:51:18.691Z xo:xo-server:sdn-controller:ovsdb-client ERROR No result for select {
Sep 17 12:51:18 XenO xo-server[25920]:   columns: [ '_uuid', 'name' ],
Sep 17 12:51:18 XenO xo-server[25920]:   table: 'Bridge',
Sep 17 12:51:18 XenO xo-server[25920]:   where: [ [ 'external_ids', 'includes', [Array] ] ],
Sep 17 12:51:18 XenO xo-server[25920]:   host: 'cyrange0'
Sep 17 12:51:18 XenO xo-server[25920]: }

The following lines also stand out when we try to create VMs that use the private network:

Sep 17 13:02:38 XenO xo-server[25920]: 2020-09-17T18:02:38.925Z xo:xo-server:sdn-controller INFO Unable to add host to network: no tunnel available { network: 'vxlan', host: 'cyrange1', pool: 'cyrange-pool' }
Sep 17 13:02:43 XenO xo-server[25920]: 2020-09-17T18:02:43.947Z xo:xo-server:sdn-controller INFO Unable to add host to network: no tunnel available { network: 'vxlan', host: 'cyrange1', pool: 'cyrange-pool' }
Sep 17 13:02:49 XenO xo-server[25920]: 2020-09-17T18:02:49.563Z xo:xo-server:sdn-controller INFO Unable to add host to network: no tunnel available { network: 'vxlan', host: 'cyrange2', pool: 'cyrange-pool' }
Sep 17 13:02:59 XenO xo-server[25920]: 2020-09-17T18:02:59.592Z xo:xo-server:sdn-controller INFO Unable to add host to network: no tunnel available { network: 'vxlan', host: 'cyrange2', pool: 'cyrange-pool' }

Sorry for the wall of text, just wanted to provide you with as much debugging information as possible. Additionally, when I run iftop on either of the xapi interfaces while doing a test with iperf, I don't see any traffic flowing through at all, leading me to suspect that the VMs are not even using the tunnels.

dgmcdona

It seems like the simplest fix, for now, would be to use a different PIF that runs across all 5 hosts and supports jumbo frames. We actually have one (eth0) that fits these requirements, but it isn't offered in the dropdown list when I go to create the private network: Screen Shot 2020-09-16 at 2.27.25 AM.png
Any thoughts on why this might be?

dgmcdona

Digging deeper, I think I am developing a hunch for why this isn't working. Our pool currently has 5 servers, two of which were unfortunately purchased at a later date and are not a perfect match with the other 3. eth5 only runs on hosts 1, 2, and 3. We though that this wouldn't matter, and that Xen Orchestra would simply keep the hosts that use the VxLAN on the hosts which use eth5. I took a look at the Xen Orchestra logs, and saw this:

Sep 16 01:11:10 XenO xo-server[130952]: 2020-09-16T06:11:10.116Z xo:xo-server:sdn-controller INFO Private network registered { privateNetwork: '234880c5-c3fd-466d-9e6a-f9fbaa36700a' }
Sep 16 01:11:10 XenO xo-server[130952]: 2020-09-16T06:11:10.182Z xo:xo-server:sdn-controller INFO New network created {
Sep 16 01:11:10 XenO xo-server[130952]:   privateNetwork: '234880c5-c3fd-466d-9e6a-f9fbaa36700a',
Sep 16 01:11:10 XenO xo-server[130952]:   network: 'vxlan',
Sep 16 01:11:10 XenO xo-server[130952]:   pool: 'cyrange-pool'
Sep 16 01:11:10 XenO xo-server[130952]: }
Sep 16 01:11:10 XenO xo-server[130952]: 2020-09-16T06:11:10.187Z xo:xo-server:sdn-controller ERROR Can't create tunnel: no available PIF {
Sep 16 01:11:10 XenO xo-server[130952]:   pifDevice: 'eth5',
Sep 16 01:11:10 XenO xo-server[130952]:   pifVlan: '-1',
Sep 16 01:11:10 XenO xo-server[130952]:   network: 'vxlan',
Sep 16 01:11:10 XenO xo-server[130952]:   host: 'cyrange4',
Sep 16 01:11:10 XenO xo-server[130952]:   pool: 'cyrange-pool'
Sep 16 01:11:10 XenO xo-server[130952]: }
Sep 16 01:11:10 XenO xo-server[130952]: 2020-09-16T06:11:10.189Z xo:xo-server:sdn-controller ERROR Can't create tunnel: no available PIF {
Sep 16 01:11:10 XenO xo-server[130952]:   pifDevice: 'eth5',
Sep 16 01:11:10 XenO xo-server[130952]:   pifVlan: '-1',
Sep 16 01:11:10 XenO xo-server[130952]:   network: 'vxlan',
Sep 16 01:11:10 XenO xo-server[130952]:   host: 'cyrange5',
Sep 16 01:11:10 XenO xo-server[130952]:   pool: 'cyrange-pool'
Sep 16 01:11:10 XenO xo-server[130952]: }

eth2, however, runs on all 5 hosts. Maybe the sdn-controller is defaulting back to eth2 once it sees that eth5 is unavailable on hosts 4 and 5?

dgmcdona

@BenjiReis I think that I have gotten to the bottom of the issue, although I'm not sure why its happening or how to fix it.
When creating the VxLAN (MTU 1550), I am presented with the option to use 2 PIFs: eth2 and eth5. eth5 is the desired interface, as it is configured to use jumbo frames (9000 MTU), and it is the one that I selected. I destroyed and recreated the VXLAN just to be sure of this. I noticed that when the two VMs are on the same host, they can use the VXLAN with their default MTU of 1500 transparently. However, once I migrate one of the VMs to a different host, packets are dropped until I set the MTU to 1450 within the VMs. Knowing this, I suspected that there was an MTU of 1500 somewhere along the line causing the packets to be lost.
Here is how I discovered the issue: I set the VM MTUs to 1450, and monitored each PIF on the host consoles using iftop while running iperf on the VMs to test bandwidth, and found that the VXLAN is using the wrong PIF. It is using eth2, which has a 1500 MTU, thus causing the packets to be lost. I have no idea why, but Xen Orchestra or XCP-ng is assigning the VXLAN to use the wrong PIF.

dgmcdona

@ben20ben @nraynaud has created a pull request with the fix, but if you're not using XO from the sources, it won't be available until the next release of XOA, which will hopefully be soon. Also, not sure if you're problem has been caused by the same issue.

dgmcdona

When I create the vxlan and look at the host consoles, I don't see a tunnel interface appear. Here is the diff between the interfaces before and after VXLAN creation:
Screen Shot 2020-09-11 at 2.25.48 PM.png

And the full list of interfaces post-VXLAN creation, just in case you want to see it:
Screen Shot 2020-09-11 at 2.28.40 PM.png

I tried assigning an IP address to vxlan_sys_4789 on both hosts, and pinging between them. This was unsuccessful. I repeated this method with the xapi15 interface, and was able to ping with packets well over 1500 bytes (I think the largest value that I tried was 2000 bytes), although the first 4 ICMP packets were lost every time.

dgmcdona

@BenjiReis The VMs are using a default MTU of 1500

dgmcdona

I just tried configuring the VxLAN with an 1800 MTU and this didn't fix the problem, unfortunately. Screen Shot 2020-09-09 at 2.17.11 PM.png

dgmcdona

Yeah, both the eth5 interface as well as the switch are configured with an MTU of 9000. These screenshots reflect the current configuration: Screen Shot 2020-09-09 at 11.09.11 AM.png

dgmcdona

Using the latest version of XO, compiled from sources. XCP-ng 8.1 running on 5 servers.

@BenjiReis, maybe you can help us out with this.

We have been working on setting up VxLANs, and have gotten close, but are running into a problem with MTU. When we create a VxLAN, we associate it with eth5, which is configured for jumbo frames (this network has been tested, and works as expected with large packets).

Expected behavior: Create a VxLAN with an MTU of 1550, so that VMs that default to a 1500 MTU can use the network transparently.

Actual behavior: Any outgoing packets larger than 1450 bytes are dropped.

So, our network switch is correctly configured on eth5 for jumbo frames and the VxLAN has been given the correct MTU value, but somewhere along the VxLAN network path, there is a 1500 MTU causing packets to drop (1450 + 50 for VxLAN). My suspicion is that it is somewhere within OpenVSwitch, which I'm not that familiar with, and I'm afraid of messing with the internal OVS settings lest I break something and make a mess of our networks.

Any recommendations for tracking this problem down?

dgmcdona

error
bubbles: false
cancelBubble: false
cancelable: false
composed: false
currentTarget: null
defaultPrevented: false
eventPhase: 0
explicitOriginalTarget: FileReader
    error: DOMException
        code: 0
        columnNumber: 0
        data: null
        filename: ""
        lineNumber: 0
        message: "File could not be read"
        name: "NotReadableError"
        result: 2154102785
        stack: ""
        <prototype>: DOMExceptionPrototype { name: Getter, message: Getter, INDEX_SIZE_ERR: 1, … }
    onabort: null
    onerror: null
    onload: null
    onloadend: null
    onloadstart: null
    onprogress: null
    readyState: 2
    result: null
    <prototype>: FileReaderPrototype { readAsArrayBuffer: readAsArrayBuffer(), readAsBinaryString: readAsBinaryString(), readAsText: readAsText(), … }
isTrusted: true
lengthComputable: true
loaded: 3400646144
originalTarget: FileReader { readyState: 2, result: null, error: DOMException, … }
returnValue: true
srcElement: FileReader { readyState: 2, result: null, error: DOMException, … }
target: FileReader { readyState: 2, result: null, error: DOMException, … }
timeStamp: 59158
total: 3400646144
type: "error"
<get isTrusted()>: function isTrusted()
<prototype>: ProgressEventPrototype { lengthComputable: Getter, loaded: Getter, total: Getter, … }
index.js:225:12

Link to the OVA

dgmcdona

@olivierlambert I'd be happy to share the .ova that I'm working with, just let me know how you'd like me to do that. Where can I find logs for this type of error?

dgmcdona

We are continuing to see this issue on our system, despite running the latest version. I'm trying to figure out what causes some uploads to succeed and others to fail with the error message I mentioned above. Besides being a .ova file, what are the requirements for a virtual appliance? Does the type of virtual disk make a difference (VHD, VMDK, VDI)? Is this a problem with the XML in the .ovf component?

dgmcdona

Reloading the SDN controller plugin fixed the problem. Thanks!

dgmcdona

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_input_chain  tcp  --  anywhere             anywhere             tcp dpt:nbd
xapi-INPUT  all  --  anywhere             anywhere
ACCEPT     gre  --  anywhere             anywhere
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_output_chain  tcp  --  anywhere             anywhere             tcp spt:nbd

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere             icmp any
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW udp dpt:ha-cluster
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:21064
ACCEPT     udp  --  anywhere             anywhere             multiport dports hpoms-dps-lstn,netsupport
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain xapi-INPUT (1 references)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW udp dpt:4789
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:6640
RETURN     all  --  anywhere             anywhere

Chain xapi_nbd_input_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain xapi_nbd_output_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

dgmcdona

cert-dir is left blank, and override-certs is set to ON.

sdnController.createPrivateNetwork
{
  "poolIds": [
    "847ffc73-e664-9dd6-87bf-c7561ea0c44b"
  ],
  "pifIds": [
    "8bb22ef0-208e-7ab4-737f-b44855da54b0"
  ],
  "name": "VxLAN",
  "description": "Cross-server private network",
  "encapsulation": "vxlan",
  "encrypted": false,
  "mtu": 1550
}
{
  "code": "ECONNRESET",
  "host": "137.30.124.81",
  "port": 6640,
  "message": "Client network socket disconnected before secure TLS connection was established",
  "name": "Error",
  "stack": "Error: Client network socket disconnected before secure TLS connection was established
    at connResetException (internal/errors.js:610:14)
    at TLSSocket.onConnectEnd (_tls_wrap.js:1546:19)
    at Object.onceWrapper (events.js:421:28)
    at TLSSocket.apply (events.js:327:22)
    at TLSSocket.patchedEmit (/opt/xo/xo-builds/xen-orchestra-202027081251/@xen-orchestra/log/src/configure.js:95:17)
    at TLSSocket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
}