Xen Orchestra netbox sync error

sb2014

Good day,
I have noticed that Xen Orchestra does not sync the IPs with Netbox.
When I manually trigger the sync, I get the following error message.

netbox.synchronize
{
  "pools": [
    "550e8400-e29b-41d4-a716-446655440000"
  ]
}
{
  "message": "ipaddr: the address has neither IPv6 nor IPv4 format",
  "name": "Error",
  "stack": "Error: ipaddr: the address has neither IPv6 nor IPv4 format
    at Object.parse (/opt/xo/xo-builds/xen-orchestra-202306050256/packages/xo-server-netbox/node_modules/ipaddr.js/lib/ipaddr.js:929:19)
    at Netbox.#synchronize (/opt/xo/xo-builds/xen-orchestra-202306050256/packages/xo-server-netbox/src/index.js:555:35)
    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202306050256/packages/xo-server/src/xo-mixins/api.mjs:417:20)"
}

Does anyone have an idea how to get to the source of the error message.

olivierlambert

Hi!

Let me ping @pdonias about this

ThasianXi

I recall a forum thread about this error. It should be fixed: (if using the latest XOA build or commit from source)
https://xcp-ng.org/forum/topic/4810/netbox-plugin-error-ipaddr-the-address-has-neither-ipv6-nor-ipv4-format

If on the latest, then there may be an issue with an IP in the pool.

sb2014

@ThasianXi
I have updated xoa.
However, I get the same error message.

netbox.synchronize
{
  "pools": [
    "550e8400-e29b-41d4-a716-446655440000"
  ]
}
{
  "message": "ipaddr: the address has neither IPv6 nor IPv4 format",
  "name": "Error",
  "stack": "Error: ipaddr: the address has neither IPv6 nor IPv4 format
    at Object.parse (/opt/xo/xo-builds/xen-orchestra-202308162147/packages/xo-server-netbox/node_modules/ipaddr.js/lib/ipaddr.js:933:19)
    at Netbox.#synchronize (/opt/xo/xo-builds/xen-orchestra-202308162147/packages/xo-server-netbox/src/index.js:631:35)
    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202308162147/packages/xo-server/src/xo-mixins/api.mjs:417:20)"
}

I took a look at the other post.
I get 2 noticeable things that may not have been considered.

first:

{
  "addresses": {}
},

second:

{
  "addresses": {
    "0/ipv4/0": "192.168.178.1",
    "0/ipv4/1": "192.168.178.2",
    "0/ipv4/2": "192.168.178.3",
    "0/ipv4/3": "192.168.178.4",
    "0/ipv4/4": "192.168.178.5",
    "0/ipv4/5": "192.168.178.6",
    "0/ipv4/6": "192.168.178.7",
    "0/ipv4/7": "192.168.178.9",
    "0/ipv4/8": "192.168.178.2 192.168.178.3 192.168.178.4 192.168.178.5 192.168.178.6 192.168.178.7 192.168.178.8 192.168.178.9",
    "1/ipv4/0": "10.0.0.2",
    "1/ipv4/1": "10.0.0.1",
    "1/ipv4/2": "10.0.0.2 10.0.0.1",
    "2/ipv4/0": "10.0.255.1"
  }
},

I replaced the public IPs with some from the 192.168.178.0/24 network.

Are there any suggestions?

sb2014

I also noticed that the doct talks about virtualization > vminterface.
However, in my netbox there is only virtualization > interface.

https://xen-orchestra.com/docs/advanced.html#netbox
next to: Add a UUID custom field:

ThasianXi

@sb2014 In the 5.85 release of XO there were updates to the Netbox plugin that now requires three objects in the UUID field.
https://xen-orchestra.com/blog/xen-orchestra-5-85/

Here are the ones you need:

Virtualization > cluster
Virtualization > virtual machine 
Virtualization > interface

pdonias

@sb2014 and @ThasianXi Thanks for the report and investigation
An empty IP shouldn't be an issue but the multiple IP addresses in the same field is probably what's causing the error. However, this should be handled by XO already, so I'm surprised we're still seeing this. What version of XCP-ng/XS are you running @sb2014?

Regarding the custom fields, that's most likely not the issue here because if you don't have the correct custom fields, it will fail at the very beginning with a clear error message. But indeed:

with the new version of the plugin, you need to assign the uuid custom field to a third object type: Virtualization > interface
the doc currently says Virtualization > vminterface but you're right, it should say Virtualization > interface instead. I'm fixing that

jpasher-work

@pdonias One more doc note. With the recent changes to store the VM platform, you also need to grant full permission on DCIM > Platform

sb2014

@pdonias
xo-server: 5.120.2
xo-web: 5.122.2

XCP-NG:
NAME="XCP-ng"
VERSION="8.2.1"
ID="xenenterprise"
ID_LIKE="centos rhel fedora"
VERSION_ID="8.2.1"
PRETTY_NAME="XCP-ng 8.2.1"
ANSI_COLOR="0;31"
HOME_URL="http://xcp-ng.org/"
BUG_REPORT_URL="https://github.com/xcp-ng/xcp"

pdonias

@jpasher-work Thanks, I'm adding that too!

@sb2014 I may have found the issue. If you want to test it, you can build the branch pierre-fix-addresses.

sb2014

@pdonias
Thank you.
Now everything seems to work.

yann

@sb2014 this address reporting looks buggy, can you please share some guest information (OS type and version, guest agent version reported by XO)?

sb2014

@yann

Here are the infomations.
Guest OS:
Management agent version: 6.2.0-76888
OS name: FreeBSD 13.2-RELEASE-p2 (OpnSense 23.7.1_3)
OS kernel: 13.2-RELEASE-p2

Is it possible that the reporting of IPv6 addresses works only sometimes or not at all?
Also under other operating systems, like Windows Server, Debian and AlmaLinux.

yann

@sb2014 The status of the guest tools on FreeBSD is really bad at the moment, and they finally marked this port as unmaintained 2 weeks ago.
Looks like such a result could only result in ifconfig output changing; it would be useful to get a look at what your ifconfig output looks like, and likely update /usr/local/sbin/xe-update-guest-attrs and propose a patch to the BSD ports maintainers.

For a better solution, I'll try to allocate more time to xen-guest-agent to get it fully working in FreeBSD soon.

sb2014

@yann

Here is the output from ifconfig.
I replaced the public IPs with:
192.168.178.0/24
fe00::/112

enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
	groups: pflog
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
	pfsync: syncdev: xn2 syncpeer: 10.0.255.2 maxupd: 128 defer: off
	syncok: 1
	groups: pfsync
xn0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN (wan)
	ether 32:92:38:a6:82:37
	inet 192.168.178.3 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.2 netmask 0xfffffff0 broadcast 192.168.178.255 vhid 3
	inet 192.168.178.4 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.5 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.6 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.7 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.8 netmask 0xfffffff0 broadcast 192.168.178.255
	inet 192.168.178.9 netmask 0xfffffff0 broadcast 192.168.178.255
	inet6 fe00::3 prefixlen 64
	inet6 fe80::30da:86ff:fea6:e181%xn0 prefixlen 64 scopeid 0x5
	inet6 fe00::2 prefixlen 64 vhid 4
	carp: MASTER vhid 3 advbase 1 advskew 100
	carp: MASTER vhid 4 advbase 1 advskew 100
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN (lan)
	ether 43:86:34:67:34:75
	inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 1
	inet 10.0.3.1 netmask 0xffffff00 broadcast 10.0.3.255
	inet6 fd00::2 prefixlen 112
	inet6 fe80::c45c:dbff:feb5:62b9%xn1 prefixlen 64 scopeid 0x6
	inet6 fd00::1 prefixlen 112 vhid 2
	inet6 fd00::3:1 prefixlen 112
	carp: MASTER vhid 1 advbase 1 advskew 100
	carp: MASTER vhid 2 advbase 1 advskew 100
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: pfsync (opt1)
	ether 32:90:32:43:21:89
	inet 10.0.255.1 netmask 0xffffff00 broadcast 10.0.255.255
	inet6 fd00::ffff:1 prefixlen 112
	inet6 fe80::6809:c1ff:fe14:b82d%xn2 prefixlen 64 scopeid 0x7
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet6 fd00::1:1 prefixlen 112
	inet 10.0.1.1 netmask 0xffffff00
	groups: wg wireguard
	nd6 options=101<PERFORMNUD,NO_DAD>
wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet6 fd00::2:1 prefixlen 112
	inet 10.0.2.1 netmask 0xffffff00
	groups: wg wireguard
	nd6 options=101<PERFORMNUD,NO_DAD>

yann

@sb2014 OK no fundamental change here.

Looking at this old agent code, though, I note that the IPs are all written space-separated in a single write to attr/eth0/ip, which is a place I've never heard about (the standard xenstore structure is documented here.

Still, this xenstore key is documented as "old style" in the XAPI code. We can easily check that writing just 2 IP space-separated addresses to this xenstore key had this exact behaviour:

host# xenstore ls /local/domain/443/attr
eth0 = ""
 ip = "1.2.3.1 1.2.3.2"
PVAddons = ""
 MajorVersion = "6"
 MinorVersion = "2"
 MicroVersion = "0"
 BuildVersion = "76888"
 Installed = "1"

XO then displays 3 entries, the first 2 ones being the individual IPs from the string, and the last one being the full unparsed string. This looks like a bug in XAPI, where the old way of publishing IPs is likely not that well tested.

EDIT Reported upstream - it looks like this old protocol simply did not support multiple IPs, and the old guest-agent is just buggy, it should have protected against multiple IPs and only reported one.

yann

@pdonias looks like the problem occurs because there are 2 kinds of data in the guest metrics: the "old style" which was never meant for multiple IPs but can be (and apparently has been) used for that, and OTOH the "new style" which XAPI created without checking it puts a valid value there.

Would it be that XO was parsing both, splitting 0/ip?
It would look like in the presence of both styles, given the existence of this issue, it would be sufficient to just ignore the "new style" one?

pdonias

@yann Yes, XO does parse it to flatten x/ip into multiple x/ipv4/x. The issue was that we assumed that only x/ip fields could have multiple space-separated IPs so we weren't flattening x/ipv4/x fields.
So when we got something like this from XAPI:

{
  '0/ip': '1.1.1.1 2.2.2.2',
  '0/ipv4/0': '1.1.1.1 2.2.2.2'
}

it used to be transformed into this:

{
  '0/ipv4/0': '1.1.1.1',
  '0/ipv4/1': '2.2.2.2',
  '0/ipv4/2': '1.1.1.1 2.2.2.2'
}

which is pretty much what @sb2014 was getting here.
And now we made a change to also flatten x/ipv4/x fields (with deduplication, etc.) so now it would be transformed into:

{
  '0/ipv4/0': '1.1.1.1',
  '0/ipv4/1': '2.2.2.2'
}

So if XAPI isn't supposed to return space-delimited IPs in x/ipv4/x fields, better make sure it's fixed upstream, but from XO's point of view, we're supporting it anyway, now

yann

@pdonias said in Xen Orchestra netbox sync error:

So if XAPI isn't supposed to return space-delimited IPs in x/ipv4/x fields, better make sure it's fixed upstream, but from XO's point of view, we're supporting it anyway, now

Yes I'd think XAPI should not blindly forwarding information from guest without a minimum of sanitation.