Xen Orchestra netbox sync error

sb2014

@pdonias
xo-server: 5.120.2
xo-web: 5.122.2

XCP-NG:
NAME="XCP-ng"
VERSION="8.2.1"
ID="xenenterprise"
ID_LIKE="centos rhel fedora"
VERSION_ID="8.2.1"
PRETTY_NAME="XCP-ng 8.2.1"
ANSI_COLOR="0;31"
HOME_URL="http://xcp-ng.org/"
BUG_REPORT_URL="https://github.com/xcp-ng/xcp"

pdonias

@jpasher-work Thanks, I'm adding that too!

@sb2014 I may have found the issue. If you want to test it, you can build the branch pierre-fix-addresses.

sb2014

@pdonias
Thank you.
Now everything seems to work.

yann

@sb2014 this address reporting looks buggy, can you please share some guest information (OS type and version, guest agent version reported by XO)?

sb2014

@yann

Here are the infomations.
Guest OS:
Management agent version: 6.2.0-76888
OS name: FreeBSD 13.2-RELEASE-p2 (OpnSense 23.7.1_3)
OS kernel: 13.2-RELEASE-p2

Is it possible that the reporting of IPv6 addresses works only sometimes or not at all?
Also under other operating systems, like Windows Server, Debian and AlmaLinux.

yann

@sb2014 The status of the guest tools on FreeBSD is really bad at the moment, and they finally marked this port as unmaintained 2 weeks ago.
Looks like such a result could only result in ifconfig output changing; it would be useful to get a look at what your ifconfig output looks like, and likely update /usr/local/sbin/xe-update-guest-attrs and propose a patch to the BSD ports maintainers.

For a better solution, I'll try to allocate more time to xen-guest-agent to get it fully working in FreeBSD soon.

sb2014

@yann

Here is the output from ifconfig.
I replaced the public IPs with:
192.168.178.0/24
fe00::/112

enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
	groups: pflog
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
	pfsync: syncdev: xn2 syncpeer: 10.0.255.2 maxupd: 128 defer: off
	syncok: 1
	groups: pfsync
xn0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN (wan)
	ether 32:92:38:a6:82:37
	inet 192.168.178.3 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.2 netmask 0xfffffff0 broadcast 192.168.178.255 vhid 3
	inet 192.168.178.4 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.5 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.6 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.7 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.8 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.9 netmask 0xfffffff0 broadcast 192.168.178.255
	inet6 fe00::3 prefixlen 64
	inet6 fe80::30da:86ff:fea6:e181%xn0 prefixlen 64 scopeid 0x5
	inet6 fe00::2 prefixlen 64 vhid 4
	carp: MASTER vhid 3 advbase 1 advskew 100
	carp: MASTER vhid 4 advbase 1 advskew 100
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN (lan)
	ether 43:86:34:67:34:75
	inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 1
	inet 10.0.3.1 netmask 0xffffff00 broadcast 10.0.3.255
	inet6 fd00::2 prefixlen 112
	inet6 fe80::c45c:dbff:feb5:62b9%xn1 prefixlen 64 scopeid 0x6
	inet6 fd00::1 prefixlen 112 vhid 2
	inet6 fd00::3:1 prefixlen 112
	carp: MASTER vhid 1 advbase 1 advskew 100
	carp: MASTER vhid 2 advbase 1 advskew 100
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: pfsync (opt1)
	ether 32:90:32:43:21:89
	inet 10.0.255.1 netmask 0xffffff00 broadcast 10.0.255.255
	inet6 fd00::ffff:1 prefixlen 112
	inet6 fe80::6809:c1ff:fe14:b82d%xn2 prefixlen 64 scopeid 0x7
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet6 fd00::1:1 prefixlen 112
	inet 10.0.1.1 netmask 0xffffff00
	groups: wg wireguard
	nd6 options=101<PERFORMNUD,NO_DAD>
wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet6 fd00::2:1 prefixlen 112
	inet 10.0.2.1 netmask 0xffffff00
	groups: wg wireguard
	nd6 options=101<PERFORMNUD,NO_DAD>

yann

@sb2014 OK no fundamental change here.

Looking at this old agent code, though, I note that the IPs are all written space-separated in a single write to attr/eth0/ip, which is a place I've never heard about (the standard xenstore structure is documented here.

Still, this xenstore key is documented as "old style" in the XAPI code. We can easily check that writing just 2 IP space-separated addresses to this xenstore key had this exact behaviour:

host# xenstore ls /local/domain/443/attr
eth0 = ""
 ip = "1.2.3.1 1.2.3.2"
PVAddons = ""
 MajorVersion = "6"
 MinorVersion = "2"
 MicroVersion = "0"
 BuildVersion = "76888"
 Installed = "1"

XO then displays 3 entries, the first 2 ones being the individual IPs from the string, and the last one being the full unparsed string. This looks like a bug in XAPI, where the old way of publishing IPs is likely not that well tested.

EDIT Reported upstream - it looks like this old protocol simply did not support multiple IPs, and the old guest-agent is just buggy, it should have protected against multiple IPs and only reported one.

yann

@pdonias looks like the problem occurs because there are 2 kinds of data in the guest metrics: the "old style" which was never meant for multiple IPs but can be (and apparently has been) used for that, and OTOH the "new style" which XAPI created without checking it puts a valid value there.

Would it be that XO was parsing both, splitting 0/ip?
It would look like in the presence of both styles, given the existence of this issue, it would be sufficient to just ignore the "new style" one?

pdonias

@yann Yes, XO does parse it to flatten x/ip into multiple x/ipv4/x. The issue was that we assumed that only x/ip fields could have multiple space-separated IPs so we weren't flattening x/ipv4/x fields.
So when we got something like this from XAPI:

{
  '0/ip': '1.1.1.1 2.2.2.2',
  '0/ipv4/0': '1.1.1.1 2.2.2.2'
}

it used to be transformed into this:

{
  '0/ipv4/0': '1.1.1.1',
  '0/ipv4/1': '2.2.2.2',
  '0/ipv4/2': '1.1.1.1 2.2.2.2'
}

which is pretty much what @sb2014 was getting here.
And now we made a change to also flatten x/ipv4/x fields (with deduplication, etc.) so now it would be transformed into:

{
  '0/ipv4/0': '1.1.1.1',
  '0/ipv4/1': '2.2.2.2'
}

So if XAPI isn't supposed to return space-delimited IPs in x/ipv4/x fields, better make sure it's fixed upstream, but from XO's point of view, we're supporting it anyway, now

yann

@pdonias said in Xen Orchestra netbox sync error:

So if XAPI isn't supposed to return space-delimited IPs in x/ipv4/x fields, better make sure it's fixed upstream, but from XO's point of view, we're supporting it anyway, now

Yes I'd think XAPI should not blindly forwarding information from guest without a minimum of sanitation.