XCP-ng 8.3 updates announcements and testing

semarie

"xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\")"

This is related to new code in the upgrade. the trunks attribute is used for VLAN filtering (new feature).

it seems that the running code is the new one (it is knowning about trunks) but the VM doesn't have the trunks attribute.

I would be interested to have a clean picture of the situation:

on which host this VM is resident on
version of the master
version of the host where the VM is resident

flakpyro

@semarie In my case the pool i tested these updates on was a 2 host pool, i patched each host using yum and then kicked off a rolling pool reboot. So the VMs would have been resident on the pool master since it is the first rebooted when running a rolling pool reboot. I was unable to manually migrate the VMs resident on this host as well, attempting to do so would show the same error.

In my case though restarting the tool stack on both hosts allowed migrations to complete and a rolling pool reboot to run successfully.

gduperrey

@acebmxer This error does not appear to be linked to the update, but rather to a repo that has not yet had time to synchronize or is in the process of doing so at the time of your update.

semarie

@flakpyro it seems to me the proper upgrade path is:

put the master in maintenance mode (xe host-disable host=$MASTER)
evacuate the master (xe host-evacuate host=$MASTER)
yum update the master
reboot the master (xe host-reboot host=$MASTER)
once done, do the same of the others hosts

the VM would have been updated with the new trunks attribute when migrating to some updated host (in your case, when migrating to the master).

flakpyro

@semarie If that's the case then the update will likely go smoothly once its released to the stable repos and users use Xen Orchestra to run a rolling pool update which follows that workflow. Since this was a test pool and using the test repos i ran the update via yum without evacuating the host first.

semarie

@flakpyro yes. thanks for your test and to reporting the problem anyway. it is helping us to see what kind of problems users could have.

gduperrey

Thank you everyone for your tests and your feedback!

The updates are live now: https://xcp-ng.org/blog/2026/08/18/august-2026-updates-1-for-xcp-ng-8-3-lts/

Andrew

@gduperrey Rolling pool update worked with released production patches.

MajorP93

Hello, so I got access to the test environment back and was able to install this set of patches.

I actually installed them on the XCP-ng host before you guys released them to the stable repository.

I ran:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates

and rebooted the system.

Unfortunately I have to say that this is the first time that patches broke my system.

While the XCP-ng host is still able to boot, it is not longer able to mount my SRs.

I have 2 SR in this test environment:

1x Linstor vSAN iSCSI configured as QCOW2
1x TrueNAS Core NFSv3 configured as VHD

I spent some hours troubleshooting this and appearently it is caused by jumbo frames no longer working after applying these XCP-ng patches.

The hypervisor / storage network in this testing environment is using jumbo frames everywhere (all switches involved, all storage systems).
Prior to installing the updates everything was working fine.

Now I can not ping the storage systems anymore using jumbo frames.
(ping -M do -s 8972 ...)

Hence the tasks that are meant to mount the SRs are stuck forever:

[14:38 xcpng-test01 ~]# xe task-list
uuid ( RO)                : a6ba1324-bfe1-8aca-c5d1-d7cc57d37cca
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : f9c76606-e516-553f-706c-ff52bc303e2d
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

The interesting thing is that "ip a" is still showing MTU 9000:

[14:44 xcpng-test01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:1c:22:40:4a:d2 brd ff:ff:ff:ff:ff:ff
8: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
9: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
10: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
11: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
12: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
13: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
    inet 10.10.160.24/24 brd 10.10.160.255 scope global xapi1
       valid_lft forever preferred_lft forever

Can't start XO VM right now as it lives on one of the SR but XO Lite is also still showing jumbo frames being enabled:

Was something changed in this set of patches that could cause this issue?

Thanks and best regards

bufanda

@gduperrey said:

Thank you everyone for your tests and your feedback!

The updates are live now: https://xcp-ng.org/blog/2026/08/18/august-2026-updates-1-for-xcp-ng-8-3-lts/

Installed on my prod pool and instantly got an SMART error. What a timing But other than that updates working withut issues.

anthoineb

@MajorP93 Try first to fix the ping issue with jumboframe before anything. If the issue persists then:
what are the types of your SRs unable to be mounted? Is there any error in SMlog or xensouce.log?

gduperrey

Hello @MajorP93,

Several things in your message raise questions for me.

First, this SR:

1x Linstor vSAN iSCSI configured as QCOW2

What is this storage and how is it configured?
Currently, we do not support QCOW2 on Linstor/Xostor. So, is it simply an iSCSI device to which you have applied the QCOW2 format, or is it a Xostor on the XCP-ng pool?

Regarding jumbo frames, we do not use them internally and are not currently testing them. It is often recommended not to use them for the management interface, as indicated in the XCP-ng documentation: https://docs.xcp-ng.org/networking/

Non-standard MTUs (such as jumbo frames) are not supported on management interfaces. Using them can lead to serious issues, including failed pool member joins or unexpected network outages.

Is that the case here?

bleader

@majorp93 no, these updates should not change anything on the MTU behavior. From your screenshots and ip a output, I assume eth4 is the only link used, than your vlan1060 network is indeed your management network. I guess with some confidence that:

eth4 is your actual NIC
xenbr4 is the pool-wide network without vlan using eth4
xapi1 is the bridge for vlan1060
If that's right indeed on your host side, the MTUs are properly set.

As Gaël said, jumbo on management network is not officially supported, because we always end up in situation similar to yours, where something stops working for "some" reason

But as you said, when everything is setup properly, it does work.

I have test hosts at home up to date but no similar setup to yours for SR. I tried with a pool-wide and a pool-wide + vlan as management with 9000 and the ping -M do -s 8972 does work fine in both cases, so nothing I can see here.

When it fails, what do you see? Message too long? mtu=1500 ? something else?

Do you have multiple hosts in that pool? Can you try that between the hosts as well and not toward the storage systems? I would also check on ovs with ovs-vsctl list interface that each of eth4, xennbr4 and xapi1 report 9000 mtu.

I know you said it was properly setup everywhere, but I would still be tempted to think there is a setup issue somewhere.

MajorP93

@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

marcoi

for the latest updates. the test pool didnt move the vms when i went to reboot. so i moved them manually. rebooted master, move vms back patched and reboot 2nd test server in the pool.
Skipped trying rolling reboot in prod and followed the same process. both pools went well without other issues.

gleh

New maintenance update candidates for XCP-ng 8.3 LTS

This batch of updates focuses on storage bug fixes and performance improvements.

What changed

Storage

blktap:
When CBT is enabled, an issue occurred where tapdisk opened cbtlog instead of the primary QCOW2 image. This previously caused commit operations (ENOTSUPP) and leaf-coalesce routines to fail, resulting in excessive storage consumption on the SR. Tapdisk now correctly bypasses the cbtlog image to allow proper coalescing.
sm: Fix a bug where the coalesce on a secondary host of a QCOW2 VDI on LVMSR could corrupt LVM metadata and would need manual intervention to restore them.
xapi:
- Migration speeds are now improved for sparse QCOW2-backed VDIs, as empty sectors are no longer transferred.
- Preserve VDI tags on migration. This is a foundational fix for kubernetes integration.

Versions

blktap: 3.55.5-9.3.xcpng8.3 -> 3.55.5-9.4.xcpng8.3
sm: 3.2.12-23.4.xcpng8.3 -> 3.2.12-23.5.xcpng8.3
xapi: 26.1.16-1.1.xcpng8.3 -> 26.1.16-1.2.xcpng8.3

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

As usual, normal use and anything else you want to test.

Test window before official release of the updates

~24 hrs

We would like to thank users who shared feedback since our last call for testing: @Andrew, @JeffBerntsen, @MajorP93, @XCP-ng-JustGreat, @acebmxer, @bufanda, @flakpyro

bufanda

@gleh installed on my lab pool. Haven’t seen any issues so far.

acebmxer

@gleh

Installed...

Updated:
  blktap.x86_64 0:3.55.5-9.4.xcpng8.3                forkexecd.x86_64 0:26.1.16-1.2.xcpng8.3                
  message-switch.x86_64 0:26.1.16-1.2.xcpng8.3       qcow-stream-tool.x86_64 0:26.1.16-1.2.xcpng8.3         
  rrdd-plugins.x86_64 0:26.1.16-1.2.xcpng8.3         sm.x86_64 0:3.2.12-23.5.xcpng8.3                       
  sm-cli.x86_64 0:26.1.16-1.2.xcpng8.3               sm-fairlock.x86_64 0:3.2.12-23.5.xcpng8.3              
  squeezed.x86_64 0:26.1.16-1.2.xcpng8.3             varstored-guard.x86_64 0:26.1.16-1.2.xcpng8.3          
  vhd-tool.x86_64 0:26.1.16-1.2.xcpng8.3             wsproxy.x86_64 0:26.1.16-1.2.xcpng8.3                  
  xapi-core.x86_64 0:26.1.16-1.2.xcpng8.3            xapi-nbd.x86_64 0:26.1.16-1.2.xcpng8.3                 
  xapi-rrd2csv.x86_64 0:26.1.16-1.2.xcpng8.3         xapi-storage-script.x86_64 0:26.1.16-1.2.xcpng8.3      
  xapi-tests.x86_64 0:26.1.16-1.2.xcpng8.3           xapi-xe.x86_64 0:26.1.16-1.2.xcpng8.3                  
  xcp-networkd.x86_64 0:26.1.16-1.2.xcpng8.3         xcp-rrdd.x86_64 0:26.1.16-1.2.xcpng8.3                 
  xenopsd.x86_64 0:26.1.16-1.2.xcpng8.3              xenopsd-cli.x86_64 0:26.1.16-1.2.xcpng8.3              
  xenopsd-xc.x86_64 0:26.1.16-1.2.xcpng8.3

MajorP93

@gleh Installed those RC packages in my test environment.
Tested basic operation. Looks good so far.
(Booted, all 3 SR connected, VMs able to start, both VHD and QCOW2)
Best regards

flakpyro

Installed on my usual hosts without issues. I am only making limited use of QCOW2 however.