XCP-ng 8.3 updates announcements and testing

TeddyAstie

@john.c said in XCP-ng 8.3 updates announcements and testing:

I don’t have AMD based hosts for XCP-ng. However may I suggest an additional validation test of this change, against Debian 13 when stable is released during or following tomorrow. I recon it should work - newer Linux Kernel version 6.12 series, though can’t be sure! Best check to avoid nasty surprises.

The performance fix support is related to the kernel version. All kernel >= 5.19 work with it (or that have https://lore.kernel.org/all/20220530082634.6339-1-jgross@suse.com/), this includes Debian 13.

Andrew

@gduperrey Updated and running. Single hosts were fine. No AMD testing.

Upgrading a busy pool seems to have had some odd issues with VM migration, but all seems to be running fine now. I had upgraded the pool from 8.2.1 to 8.3 last month and everything has been fine. This time (after rebooting pool master) while trying to migrate guests so I could reboot hosts got a few XO errors like:

xo:api WARN admin | vm.migrate(...) [1s] =!> XapiError: INTERNAL_ERROR(Object with type VM and id bd38ee46-2701-3022-ec61-03bf3ffbdcc9/config does not exist in xenopsd)
xo:api WARN admin | vm.migrate(...) [2s] =!> XapiError: INTERNAL_ERROR(Object with type VM and id 235de1d7-832e-f1a7-fa1c-a45877aab8f6/config does not exist in xenopsd)

It was only a few on one host. I can NOT confirm this is related to the updates as I'm not having problems now. After manually rebooting the host and stuck guests, things were ok again.

olivierlambert

Updated and enabled (but on an Intel machine so far). I think I will enable it on our prod with the new config, this won't hurt (full EPYC)

bufanda

Installed on a 2 Node pool consisting of 2
HP EliteDesk 300 G3 Mini with

1x i7 6700T & 1x i5 6500T
32GB RAM each
NFS VM SR
iSCSI VM SR
various SMB ISO SR

VMs migrated during reboot withput issues, also no issues with migration after updates completed with all nodes rebooted. Alos no issues found with VMs. Primarily AlmaLinux 9 and FreeBSD 13.

gb.123

@gduperrey
@olivierlambert

First of all ...... a BIG THANK YOU for this patch !

Tested on Minisforum AMD 7945HX based pc; and I can confirm that XCP-ng boots with ACPI PSS & C Cores as mentioned in this post and it works with both options enabled.

So excited that I have tested this only and replied before testing other things..

UPDATE :
While updating this, I actually had to do 2 updates (last stable update patch of 7 files + this one)

After this installation, GPU (Nvidia) passthrough completely seems to be broken. Before the update(s) the host had to be turnoff completely, power cable had to be removed and then replugged to reset the graphics card (which used to work as I was able to run the card); but now attaching the graphics card to VM makes the VM hang on VM boot and at times, the host is also abruptly restarted (as if someone as pressed the HW reset button).

I am not sure whether the problem is in this update or the last stable one; but graphics card reset for Nvidia ( I think it has to do with ACPI reset ) still remains a problem.

Would be great if you guys can look into this.

olivierlambert

Updated on our prod cluster, works very well (full EPYC)

Andrew

@gduperrey I updated my little AMD Ryzen 5 5600U (Zen3) and it's running great!

As for the important VM to VM network performance, using Debian 13 and iperf3 single thread, before (update not enabled) is about 7.2-8Gb/sec. After the update (xen-platform-pci-bar-uc=false) I get about 10.1-13Gb/sec. So that's about a 40-60% improvement. This brings it in line with similar small Intel systems.

I did not see any change (or problem) setting it on my small test Intel system (getting about 10.1-10.8Gb/sec).

FYI: Remember, after the config change and xe-toolstack-restart, restart the VMs!

olivierlambert

Yeah I had to stop then start the VM to enjoy the new performance. On my end, iperf (not iperf3) bring even more perf on my setup, especially with multiple threads (-P4 and -P8 gave more than 100% boost)

gduperrey

New update candidate for you to test!

A new non-urgent update is ready for user testing before a future collective release. Below are the details.

A bug was found in the Emergency Network Reset due to desynchronisation between xsconsole and XAPI. This issue prevented the Emergency Network Reset from working at all. This update includes the fixes from the upstream xsconsole project to fix it.

Maintenance updates

xsconsole
- Backport sync of network reset trigger file path with XAPI to fix emergency network reset
- Backport fix for pool.conf IPv6 to avoid IPv6 truncation

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

Reboot is not strictly necessary, but the xsconsole instance running on the first virtual terminal of your host won't be restarted otherwise. If you do not reboot, make sure to start xsconsole from another terminal after the update.

The usual update rules apply: pool coordinator first, etc.

Versions:

xsconsole: 11.0.8-1.2.xcpng8.3

What to test

Normal xsconsole usage, is still useful feedback. However, if possible, the most helpful test would be performing an Emergency Network Reset through xsconsole, making actual configuration changes and verifying that they are correctly applied after reboot.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

gduperrey

New update candidate for you to test!

A new non-urgent update is ready for user testing before a future collective release. Below are the details.

Maintenance updates

broadcom-bnxt-en: Update driver to version 1.10.3_232.0.155.5

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

A reboot is preferable to load the new version of the driver.

The usual update rules apply: pool coordinator first, etc.

Versions:

broadcom-bnxt-en: 1.10.3_232.0.155.5-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

gduperrey

Updates published: https://xcp-ng.org/blog/2025/09/01/september-2025-maintenance-update-for-xcp-ng-8-3/

Thank you for the tests!

manilx

@gduperrey Installed at HomeLab. No issues.
Running via
yum clean metadata ; yum update

john.c

@manilx said in XCP-ng 8.3 updates announcements and testing:

@gduperrey Installed at HomeLab. No issues.
Running via
yum clean metadata ; yum update

You must have been looking forward to this improvement for quite a while. Once it reaches the point where it can be rolled into production, your AMD Epyc servers will get to see a boost, the Linux guests any way.

manilx

@john.c Will apply to business EPYC servers right away

manilx

@john.c Updated our 2 production pools, with RPU.

RPU emptied the master, rebooted BUT then nothing else.
It should have moved all VM's to the other host, patched/rebooted and then migrating the VM's where they were.
This did not happen. I had to manually empty the other hosts, patch, reboot and migrate the VM's.

flakpyro

@gduperrey Installed on about 50 servers across various pools and remote sites. No issues. Ran a couple backup jobs as well which completed without issue.

Greg_E

@manilx

Once in a while my rpu will do this, then I handle it manually. Been happening more often since the 8.3 upgrade, but not enough to post about it yet since we are only a couple of updates into the LTS. Still watching though and will probably do this patch on wednesday.

Greg_E

I got my production system updated yesterday, no issues or oddities with the RPU.

I'm still surprised by how much faster the VMs migrate host to host than they did with 8.2.x, it's like a 4:1 or 5:1 change on my production system. Haven't had time to fool with my lab and see what's what.

gduperrey

New security update candidates for you to test!

News XSAs (Xen Security Advisory) were published on the 9th of September, and updates to Xen & XAPI address them.

xapi:
- Fix XSA-474 — A Denial of Service can be caused by buggy or malicious inputs to XAPI (CVE-2025-58146). There are several vulnerabilities identified in XAPI:
  - Input sanitisation mismatch in notifications — While updates to the XAPI database correctly sanitise input strings, the system generates notifications using the unsanitised version. This flaw causes the database’s event thread to crash, halting further processing.
  - Inconsistent UTF-8 handling — XAPI’s UTF-8 encoder follows version 3.0 of the Unicode specification, whereas some of the libraries it relies on enforce the stricter version 3.1 standard. As a result, certain strings may be accepted as valid UTF-8 by XAPI but rejected by other components. If such strings are entered into the database, the database can subsequently fail to load.
  - Lack of sanitisation in Map/Set updates — When updating Map/Set objects in the XAPI database, no sanitisation is applied to the inputs, which introduces additional risks.
xen-*:
- Fix XSA-472 — Potential risks include Denial of Service (DoS) impacting the whole host, information exposure, or escalation of privileges. There are several vulnerabilities associated with the way guest memory pages are handled and accessed in the Viridian code:
  - NULL pointer dereference during reference TSC area update — This issue occurs when the system tries to update the reference TSC area but encounters a NULL pointer. (CVE-2025-27466)
  - NULL pointer dereference when delivering synthetic timer messages — This happens if the code assumes the SIM page is already mapped when a synthetic timer message must be delivered. (CVE-2025-58142)
  - Race condition in reference TSC page mapping — A guest system can trigger Xen to release a memory page while it is still referenced in the guest’s physical-to-machine (p2m) page tables. (CVE-2025-58143)

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-candidates
yum update --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xapi: 25.6.0-1.12.xcpng8.3
xen: 4.17.5-15.3.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the updates

~2 days.

Andrew

@gduperrey 8.3 Pools updated and running. RPU worked 99%...failed with a host out of memory error on the last migrations (pool is N+2, so no reason). Single hosts are updated as usual.