SDN Controller Errors

gb.123

I am unable to connect to VMs using VxLAN.
I am unable to reproduce the exact problem, but sometimes this works:

Disable the SDN Controller
Delete Configuration (SDN Controller)
Set Override Certs to 'ON'
Save Config
Enable Controller
Start VM

However, even if the VM is able to connect at start, re-starting the VM without touching/changing any config leads to VM not being able to connect.

I get following errors:

2024-04-15T19:39:59.933Z xo:xo-server:sdn-controller:ovsdb-client ERROR No result for select {
   columns: [ '_uuid', 'name' ],
   table: 'Bridge',
   where: [ [ 'external_ids', 'includes', [Array] ] ],
   host: 'xxxxx-1'

2024-04-15T19:41:51.55XX xo:xo-server:sdn-controller ERROR Error in _objectsUpdated {
error: TypeError: Cannot read properties of undefined (reading 'network')
       at find (/home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:241:38)
       at Array.find (<anonymous>)
       at getHostTunnelForNetwork (/home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:241:26)
       at SDNController._addHostToPrivateNetwork (/home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:1134:20)
       at SDNController._addHostToPrivateNetworks (/home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:1198:18)
       at SDNController._hostMetricsUpdated (/home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:994:19)
       at /home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:846:22
       at /home/node/xen-orchestra/node_modules/lodash/lodash.js:4967:15
       at baseForOwn (/home/node/xen-orchestra/node_modules/lodash/lodash.js:3032:24)
       at forOwn (/home/node/xen-orchestra/node_modules/lodash/lodash.js:13082:24)
       at SDNController._objectsUpdated (/home/node/xen-orchestra/packages/xo-server-sdn-controller/src/index.js:838:11)
       at Collection.emit (node:events:529:35)
       at Collection.emit (/home/node/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
       at /home/node/xen-orchestra/packages/xo-collection/src/collection.js:291:16
       at /home/node/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
       at baseForOwn (/home/node/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
       at forOwn (/home/node/xen-orchestra/node_modules/lodash/forOwn.js:33:20)
       at /home/node/xen-orchestra/packages/xo-collection/src/collection.js:289:13
       at Xapi._processEvents (file:///home/node/xen-orchestra/packages/xen-api/index.mjs:1066:5)
       at Xapi._watchEvents (file:///home/node/xen-orchestra/packages/xen-api/index.mjs:1226:14),
  object: host_metrics {
     uuid: '9xxx',
     memory_total: 1xxxx83072,
     memory_free: 1xxxx41824,
    live: true,
     last_updated: '20240413T08:39:XX',
     other_config: {}
  }
 }

olivierlambert

Question for @BenjiReis

gb.123

@BenjiReis

I have also noticed that the same host gets 'added' after a while (restarts of VMs on that Host)

2024-04-16T04:23:54.252Z xo:xo-server:sdn-controller:private-network INFO Host added {
privateNetwork: '',
network: 'VxLAN',
host: 'xxxx-2',
pool: 'XXXX'
}
2024-04-16T04:24:33.462Z xo:xo-server:sdn-controller:private-network INFO duplicates of the previous log were hidden { nDuplicates: 2 }
2024-04-16T04:24:39.152Z xo:xo-server:sdn-controller:private-network INFO Host added {
privateNetwork: 'xxx-xxx-xxx-xxx-xxx',
network: 'VxLAN',
host: 'xxxx-2',
pool: 'XXXX'
}

BenjiReis

@gb-123 That's weird

is there another SDN controller connected to your pools? (not necessarily an XO SDN controller) that would override the certificates of XO?

Are there any XAPI error (in /var/log/xensource.log) when you restart your VM?

gb.123

@BenjiReis

is there another SDN controller connected to your pools? (not necessarily an XO SDN controller) that would override the certificates of XO?

None. I have just done a clean install of Xcp-NG 8.3 beta 2

Are there any XAPI error (in /var/log/xensource.log) when you restart your VM?

I just ran cat /var/log/xensource.log | grep error and got the following :

xapi: [error||376 |xapi events D:exxcfeaxxbxx|xenops] events_from_xapi: missing from the cache: [ 1xx05x9-x65x-xxxxxxx]

Are the certs copied to the xcp-ng hosts in a pool ?
Is there way to delete all SDN certificates from hosts so that new ones are copied automatically ?

Update :
I don't think certificates are at fault. Checked /etc/stunnel/certs and the certificates are same in all hosts

New Error while re-creating VxLAN (this time non encrypted)

xo:xo-server:sdn-controller:ovsdb-client ERROR No result for select {
columns: [ 'name', 'options' ],
table: 'Interface',
where: [ [ '_uuid', '==', [Array] ] ],
host: 'xxxxx-2'
}

I'll try the GRE networks now

gb.123

GRE Encrypted Network Seems to work fine.

VxLAN still not connecting

gb.123

@BenjiReis

Update 2:

I added another host, in which the tunnel for GRE is showing as tunnel0 and for VxLAN is showing as tunnel1. However, in the pool, tunnel0 is VxLAN and tunnel1 is GRE.

So now GRE has also stopped working on the new Host.

Any way I can change the tunnel numbers manually ?

I think the code allots tunnel0 and tunnel1 as per name of the connection in the first time.

gb.123

@BenjiReis

Update 3:

I managed to sync the tunnels by deleting the network and then recreating all networks one by one. However, I think the code allots tunnel numbers as per alphabetical name sorting rather than taking it from the pool.

(VxLan - not working
GRE - working)

gb.123

@BenjiReis

Update 4 :
The GRE connection works, but us spotty at times. Sometimes, the VM doesn't connect, but if I restart the VM several times, it starts to connect. This is when I have not changed any setting between starts.

(From what I remember, it was working perfectly in 8.2.1. Do you think this is a problem with 8.3 beta ? )

BenjiReis

@gb-123 The tunnel are numbered by order of creation by XAPI, you can't change them.

I don't XCP-ng 8.3 has an issue since we use it internally and the SDN is working fine on our side.
I think some events are triggering the SDN controller to add the host over and over and during the adding of the host the connection might be perturbated. You should take a look a look at XO's logs to see if you can find something that might trigger the SDN controller.

Does the host added frequently has another weird behavior?

gb.123

@BenjiReis said in SDN Controller Errors:

@gb-123 The tunnel are numbered by order of creation by XAPI, you can't change them.

Bug Report: At the time of adding a host to existing GRE/VxLANs in the network, the XAPI takes into account the Alphabetical name while creating the tunnel numbers. Specific Eg.:

Add 2 Hosts to a pool.
Create VxLAN called "VxLAN" -> This will be created as tunnel0
Create GRE called "GRE" -> This will be created as tunnel1 (since tunnel0 already exists)
Add new Host.

Error : New Host will have GRE as tunnel0 and VxLAN as tunnel1

I don't know if the above happened to me as a one-time case scenario or there is a bug, but worth a check.

You should take a look a look at XO's logs to see if you can find something that might trigger the SDN controller.

Does the host added frequently has another weird behavior?

Can you let me know the file I should see the log or should I look for journalctl ?
The weird behavior is that after several restarts, the VM gets connected and then after restart looses connection again. But I noticed whenever a VM is started on a host, the Host gets added to the SDN as per the logs above.

Just a last question which I need to further diagnose this-
Does the Host ever call on XO or is it always XO that calls on the Host (Server-Client)?
Or do they use WebSockets connection (so that the host does not need to call XO) ?

What would happen if I turn off XO after creating a SDN network ?

BenjiReis

@gb-123 AFAIK the host doesn't call XO, if you shut XO off, the private network would continue to work but its state wouldn't change.
It means if you restart an host etc the connection to this host would be lost.

The logs depends on how your installed XO or XOA etc so you might want to go through our doc!

Cheers