XCP-ng 8.3 updates announcements and testing

robertblissitt

@Pilow I've been wondering lately if I should do a Rolling Pool Reboot before any Rolling Pool Update. This might allow me to identify problems in advance and I would also be installing the patches on freshly-rebooted hosts.

Pilow

@robertblissitt yup, afterward this seems to be a good best practice...
my hosts were up for 4 month, and because of DNS resolution problem had 77 patches to catch up (80 for one with advanced telemetry enabled)

a rolling reboot would have probably put in front the initial migration/evacuation problem (and subsequent zombies VMs)

and no patches applied, and no pool in a semi upgraded state

note to my future self, try a rolling reboot first.

rzr

New security and maintenance update candidates for you to test!

The whole platform has been hardened with crypto libraries updates.
We also publish other non-urgent updates which we had in the pipe for the next update release.

Important notice

️ Xen Orchestra's sdn_controller users should be aware that OpenSSL was updated to major version 3, causing XCP-ng to reject previously generated self-signed certificates for the SDN Controller: they must be updated manually, accordingly to the guide's procedure.

User feedback is valuable to all, feel free to report success or ask for clarification in the related forum thread.

What changed

OpenSSL and OpenSSH major version update

openssl: Update to 3.0.9
- The OpenSSL 3 upgrade is improving the security and maintainability of the system, but has impact regarding certificates generation in sdn_controller, as documented above.
- To enable backward compatibility with older deprecated APIs, a new package, openssl-compat-10 has been introduced.
openssh: Update to 9.8p1
- Note that older ssh-clients (with weak ciphers) will need to update, if connection is rejected.
libssh2: Update to 1.11.0

Maintenance updates

Virtualization & System

xen: Update to 4.17.6
- Xen sources updated to v4.17.6 and synchronization of previously released patches for XSA-477 and XSA-479.
qemu: Bug fixes
- qemu would crash when a framebuffer is relocated on a migrated HVM guest.
- A race condition could cause events to be sent before capabilities negotiation.
varstored: Update to 1.3.1
- No further functional change from 1.2.0-3.5 (Fixes for XSA-478 / CVE-2025-58151 were backported).
- Just syncing with XenServer, rebuilt with openssl-3.

Control plane

xapi: Update to 26.1.3
- User agents of clients are now tracked. Fetchable by using Host.get_tracked_user_agents.
- Now it's possible to delete a VM with a snapshot that has a vTPM associated.
- Speed up exports for mostly empty disks.
- Now the tags of VDIs are copied when they are cloned or snapshotted done.
- Fixed RPU scenario where pool members don't get enabled.
- Added API for controlling NTP.
- Fixed falling back to full backups instead of delta backups in cases where a VM was hosted in a local SR with more than 256 disks. This could also cause migrations to fail.
- Added API to limit the number of VNC connections to a single VM.

UI

xolite: Update to 0.19.0
- [VM/New] Added vTPM support.
- [VM/New] Fix wording in "Memory" section.
- [TreeView] Scroll to current item in list view.
- ChangeLog

Storage

sm: Bug fixes
- Improve Robustness FileSR GC when a host is offline.
- Ensure LVM VDI is always active before relink.
- Remove GC flag DB_GC_NO_SPACE when necessary to avoid errors.
- Improve error messages when vdi_type is missing on LVM VDIs.
blktap: Bug fix
- Fixes a crash happening when scanning a SR with corrupt VHDs.
lvm2: Update to 2.02.180
- Add scini device support (Dell PowerFlex).

Network

netsnmp: Update to 5.9.3
openvswitch:
- Rebuild with openssl-3 plus minor maintenance change.
gnutls: Remove dane tool

Misc

xcp-ng-release: UX improvement
- The shell command history now record timestamps to improve consumer support.
createrepo_c: Update to 0.21.1
krb5: Synchronized with XenServer 8.4 and rebuilt for OpenSSL 3.
ipmitool: Update to 1.8.19
libarchive: Update to 3.6.1
trousers: Update to 0.3.15 and rebuild for OpenSSL 3.
- This version includes security fixes for known vulnerabilities in earlier upstream version, deemed not exploitable realistically on XCP-ng.
wget: Update to 1.21.4

Note that libraries updates (libopenssl, notably) impacted several other packages which had to be rebuilt (some had to be patched too). Refer to the package list below.

Drivers updates (check details below)

More information about drivers and current versions is maintained on the drivers wiki page.

broadcom-bnxt-en: Update to v1.10.3_237.1.20.0
- No functional changes expected.
intel-i40e : Update to 2.25.11
- PTP-related kernel crash bugfixes for Intel i40e driver version 2.25.11.
- ️ Google for the "intel <model-name> compatibility matrix" and make sure to update the non-volatile memory in NIC with the matching NVM version, after updating the driver.
- This is also applicable for the intel-i40e-alt flavour of the driver package.
intel-ixgbe: Update to 6.2.5
- More Ethernet PCI Express 10 Gigabit Intel NIC devices are handled (E600 et E610 series).

XOSTOR

In addition to the changes in common packages, the following XOSTOR-specific packages received updates:

drbd: Reduce the I/O load and time during resync.
drbd-reactor: Misc improvements regarding drbd-reactor and events.
linstor:
- Resource delete: Fixed rare race condition where a delayed DRBD event causes "resource not found.
- Misc changes to improve robustness LINSTOR API calls and checks.
sm:
- Wait for DRBD UpToDate state during LINSTOR VDI resize.
- Improve LINSTOR error messages in the case of an excessively long VDI resize.
- Simplify LINSTOR SR scan logic removing XAPI calls.
- Use worker threads during LINSTOR SR's scan to improve performance.
- Ensure a XOSTOR volume can't be destroyed if used by any process (outside of the SMAPI environment).
- Use ss to obtain the controller IP: it's a significant improvement to avoid relying on DRBD commands or XAPI plugins.
- Avoid issuing errors if the size of a LINSTOR volume cannot be fetched after a bad delete call.
python-linstor: updated to version 1.27.1. LINBIT's changelog:
- "Added api method to check the controller’s current encryption state (locked/unlocked/unset)"
linstor-client: updated to version 1.27.1. LINBIT's changelog:
- "Added new alias --drbd-diskless to command r td to mimic the option from r c.
- "Added new sub-command encryption status to show the current locked-state of the controller.

Versions:

bind: 32:9.9.4-61.el7_5.1 -> 32:9.9.4-63.1.xcpng8.3
blktap: 3.55.5-6.1.xcpng8.3 -> 3.55.5-6.3.xcpng8.3
broadcom-bnxt-en: 1.10.3_232.0.155.5-1.xcpng8.3 -> 1.10.3_237.1.20.0-8.1.xcpng8.3
coreutils: 8.22-21.el7 -> 8.22-22.xcpng8.3
createrepo_c: 0.10.0-6.el7 -> 0.21.1-3.xcpng8.3
curl: 8.9.1-5.1.xcpng8.3 -> 8.9.1-5.2.xcpng8.3
gnutls: 3.3.29-9.el7_6 -> 3.3.29-10.1.xcpng8.3
gpumon: 24.1.0-71.1.xcpng8.3 -> 24.1.0-83.2.xcpng8.3
intel-i40e: 2.25.11-2.xcpng8.3 -> 2.25.11-4.xcpng8.3
intel-ixgbe: 5.18.6-1.xcpng8.3 -> 6.2.5-1.xcpng8.3
intel-microcode: 20251029-1.xcpng8.3 -> 20260115-1.xcpng8.3
ipmitool: 1.8.18-7.el7 -> 1.8.19-11.1.xcpng8.3
iputils: 20160308-10.el7 -> 20160308-10.1.xcpng8.3
krb5: 1.15.1-19.el7 -> 1.15.1-22.1.xcpng8.3
libarchive: 3.3.3-1.1.xcpng8.3 -> 3.6.1-4.1.xcpng8.3
libevent: 2.0.21-4.el7 -> 2.0.21-4.1.xcpng8.3
libssh2: 1.4.3-10.el7_2.1 -> 1.11.0-1.xcpng8.3
libtpms: 0.9.6-3.xcpng8.3 -> 0.9.6-3.1.xcpng8.3
lvm2: 7:1.02.149-18.2.1.xcpng8.3 -> 7:2.02.180-18.3.1.xcpng8.3
mdadm: 4.0-13.el7 -> 4.2-5.xcpng8.3
net-snmp: 1:5.7.2-52.1.xcpng8.3 -> 1:5.9.3-8.1.xcpng8.3
openssh: 7.4p1-23.3.3.xcpng8.3 -> 9.8p1-1.2.1.xcpng8.3
openssl: 1:1.0.2k-26.2.xcpng8.3 -> 1:3.0.9-2.0.1.3.xcpng8.3
openvswitch: 2.17.7-2.1.xcpng8.3 -> 2.17.7-4.1.xcpng8.3
python: 2.7.5-90.el7 -> 2.7.5-92.1.xcpng8.3
python3: 3.6.8-18.el7 -> 3.6.8-20.xcpng8.3
python-pycurl: 7.19.0-19.el7 -> 7.19.0-19.1.xcpng8.3
qemu: 2:4.2.1-5.2.15.2.xcpng8.3 -> 2:4.2.1-5.2.17.1.xcpng8.3
rsync: 3.4.1-1.1.xcpng8.3 -> 3.4.1-1.2.xcpng8.3
samba: 4.10.16-25.2.xcpng8.3 -> 4.10.16-25.3.xcpng8.3
sm: 3.2.12-16.1.xcpng8.3 -> 3.2.12-17.1.xcpng8.3
ssmtp: 2.64-14.el7 -> 2.64-14.1.xcpng8.3
stunnel: 5.60-4.xcpng8.3 -> 5.60-5.xcpng8.3
sudo: 1.9.15-4.1.xcpng8.3 -> 1.9.15-5.1.xcpng8.3
swtpm: 0.7.3-12.xcpng8.3 -> 0.7.3-12.1.xcpng8.3
tcpdump: 14:4.9.2-3.el7 -> 14:4.9.2-3.1.xcpng8.3
trousers: 0.3.14-2.el7 -> 0.3.15-11.1.xcpng8.3
varstored: 1.2.0-3.5.xcpng8.3 -> 1.3.1-2.1.xcpng8.3
wget: 1.14-15.el7_4.1 -> 1.21.4-1.1.xcpng8.3
xapi: 25.33.1-2.3.xcpng8.3 -> 26.1.3-1.3.xcpng8.3
xcp-featured: 1.1.8-3.xcpng8.3 -> 1.1.8-6.xcpng8.3
xcp-ng-release: 8.3.0-36 -> 8.3.0-37
xen: 4.17.5-23.2.xcpng8.3 -> 4.17.6-2.1.xcpng8.3
xo-lite: 0.17.0-1.xcpng8.3 -> 0.19.0-1.xcpng8.3

XOSTOR:

linstor: 1.33.1-1.el7_9
linstor-client: 1.27.1-1.xcpng8.3
python-linstor: 1.27.1-1.xcpng8.3
xcp-ng-linstor: 1.2-6.xcpng8.3

Optional packages:

iperf3: 3.9-13.1.xcpng8.3
ldns: 1.7.0-21.1.xcpng8.3
socat: 1.7.4.1-6.1.xcpng8.3

Test on XCP-ng 8.3

If you are using XOSTOR, please refer to our documentation for the update method.

If you are using XenOrchestra's SDN controller please apply the OpenSSL upgrade procedure.

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

XAPI tests:
- Check that the NTP servers used by hosts are set to Factory, test changing them to DHCP, or Custom.
- Check that the console limit is 0 by default, test changing it to 1, and set a timeout.
System: Check updated tools (ssh, wget, samba, mdadm...)
Normal use and anything else you want to test.

Test window before official release of the updates

~1 week

flakpyro

Installed on my usual selection of hosts. (A mixture of AMD and Intel hosts, SuperMicro, Asus, and Minisforum). No issues after a reboot, PCI Passthru, backups, etc continue to work smoothly. Also installed on a HP GL325 Gen 10 with no issues after reboot.

ph7

@rzr said:
xo-lite: 0.18.0-1.xcpng8.3

Well I was and still is on v0.19.0:
Screenshot 2026-03-04 at 12-22-48 Settings - XO Lite.png

I am running an old AMD Ryzen 5 2400GE homelab with NFS
no XOSTOR or SDN.
Tested NTP Def and dhcp, ssh, wget, cont. repl. ...
all worked fine so far.

rzr

@ph7 said:

Well I was and still is on v0.19.0:

So you're up to date ! it was a mistake, let me generate an up to date list

JeffBerntsen

@rzr Installed and seems to be working normally on my test systems.

acebmxer

@rzr

Updated my to AMD Ryzen host in my home lab. No issues with update will monitor and report back any issues.

acebmxer

@rzr

Built new Ubuntu 24.04 vm either fresh install from ISO or from cloudint i seem to be having issues. Existing vms seem to be fine. First thought was becuase vm was multiple nics add it might have caused networking issues. Powered off that vm and build new (first as fresh from iso. Second from cloudint) with 1 nic. Same issues...

Update - issues were with script that i updated prior to updating hosts. Fixed script all is working again. Didnt test script on new vm prior to updating host.

olivierlambert

I'm not sure it's the right topic for that, probably worth creating a new one

acebmxer

@olivierlambert Created new topic.

gduperrey

New update candidate for you to test!

A new update for the Xen packages is ready, which brings a significant improvement in live migration performance on AMD systems under heavy load, that we add to the previous batch of updates for a common publication.

Maintenance updates

xen: Improve migration performance on AMD systems under heavy load.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.17.6-4.1.xcpng8.3

What to test

Normal use and anything else you want to test. If you have a pool with AMD processors, we're interested in your feedback regarding live migration under heavy load.

Test window before official release of the updates

~4/5 days

Andrew

@gduperrey The new OpenSSL/SSH blocks existing/working RSA keys from older SSH clients. While you can still use a password for SSH, it will block old keys from working which will break things (not good for existing LTS installs). To maintain compatibility add PubkeyAcceptedAlgorithms +ssh-rsa to /etc/ssh/sshd_config

flakpyro

@gduperrey Tested this on the same hosts i already have running the testing updates from earlier. No issues. Mixture of AMD and Intel.

gduperrey

@Andrew I just pinged Philippe (rzr) internally to ask him to look into this

rzr

@Andrew said:

@gduperrey The new OpenSSL/SSH blocks existing/working RSA keys from older SSH clients. While you can still use a password for SSH, it will block old keys from working which will break things (not good for existing LTS installs). To maintain compatibility add PubkeyAcceptedAlgorithms +ssh-rsa to /etc/ssh/sshd_config

Hi @andrew, thank you for your feedback, the fallback option you're suggesting will work but it will downgrade the security of your system, we suggested to update clients:

"Note that older ssh-clients (with weak ciphers) will need to update, if connection is rejected."

Let me make it more explicit that older keys should be also refreshed:

  ssh-keygen # To generate new $identity_file 
  ssh-copy-id \
        -i $identity_file \
        -o HostKeyAlgorithms=+ssh-rsa \
        -o PubkeyAcceptedAlgorithms=+ssh-rsa \
        $user@$host
  ssh $user@$host

Ideally this can be done before the update, but let's us think if we have a better strategy to provide a smoother experience, meanwhile if anyone is curious please check:

https://www.openssh.org/releasenotes.html

https://www.openssh.org/txt/release-8.8

"We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519)."

https://datatracker.ietf.org/doc/html/rfc8332

As I understand, RSA is safe unless it was coupled with SHA1 hash function which was then decoupled in later versions (and then obsoleted in V_8_7_P1-4-g234475025 with https://github.com/openssh/openssh-portable/commit/2344750250247111a6c3c6a4fe84ed583a61cc11 "The use of RSA/SHA1 can be re-enabled by adding "ssh-rsa" to the
PubkeyAcceptedAlgorithms directives on the client and server.") .

Regen keys will be needed, better sooner than later, meanwhile we could support weak ~~keys~~ clients during a short (TBD) deprecation period.

Update: I think I was able to reproduce the issue @andrew reported using a RSA key generated with
OpenSSH_5.1p1 Debian-6.maemo2, OpenSSL 0.9.8e 23 Feb 2007

ssh-keygen -lf ~/.ssh/id_rsa.pub
2048 SHA256:abcde+0123456789012345678901234567890/vwxyz user@Nokia-N810-43-7 (RSA)

Used along a later client (in a debian chroot jessie amd64) :
OpenSSH_6.7p1 Debian-5+deb8u4, OpenSSL 1.0.1t 3 May 2016

While it worked as expected (in a debian chroot stretch amd64) : with:
OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019

So to conclude using rsa keys need ssh-7+ while ssh6 can be used using stronger cypher like id_ed25519 (not rsa).

PS: this post may be updated

0

djmdjm committed to openssh/openssh-portable

upstream: After years of forewarning, disable the RSA/SHA-1

signature algorithm by default. It is feasible to create colliding SHA1
hashes, so we need to deprecate its use.

RSA/SHA-256/512 remains available and will be transparently selected
instead of RSA/SHA1 for most SSH servers released in the last five+
years. There is no need to regenerate RSA keys.

The use of RSA/SHA1 can be re-enabled by adding "ssh-rsa" to the
PubkeyAcceptedAlgorithms directives on the client and server.

ok dtucker deraadt

OpenBSD-Commit-ID: 189bcc4789c7254e09e23734bdd5def8354ff1d5

stormi

Although disabling ssh-rsa is the right thing to do from a security perspective, we'll see what we can do to smoothen the transition.

psafont

@rzr said:

Hi @andrew, thank you for your feedback, the fallback option you're suggesting will work but it will downgrade the security of your system, we suggested to update clients:

If users need to take action, I would rather recommend users to do something that raises the security floor, like generating new keys with newer, future-looking ciphers, like ed25519:

ssh-keygen -t ed25519 -C "<email>"
for server in $servers do ; ssh-copy-id $server; done

gb.123

Hello,
After updating I get this:

This seems harmless as I don't have and scsi drive attached. But these messages were not there before.

Other than that, the server seems to boot fine.

Regards,

PS:
I have updated once for several updates (not one by one) so this messages may also be there in previous updates and may not be related to this particular update.

rzr

@gb.123 said:

Wrong diagnostic page; asked for 1 got 8
Failed to get diagnostic page 0x1
Failed to bind enclosure -19

be there in previous updates and may not be related to this particular update.

I think you're right because the issue you are facing is reported by kernel itself (which was not updated), Can you please share more details about the hardware you're using (is any USB device involved ? try smartmon tools too) eventually post details in other sections of forum.