XCP-ng 8.3 updates announcements and testing

[olivierlambert]

That's a weird one Ping @Team-Hypervisor-Kernel

[Pilow]

@olivierlambert shoutout to @danp that did a takeover of the incident ticket

he headed me the right way to resolution of the problem, my production pool is back up & running with its VMs.

there was indeed a diff between what was seen by "xl list"/"xenops-cli list" and what was seen by XOA in the web ui.
a couple "xl destroy pid" to destroy zombie VMs, and toolstack restarts later, all is now up.

I don't know how the hell a simple RPU did get me in this situation though...

[olivierlambert]

Oh wow. Indeed, that's strange. And big kudos to @danp then!!

[robertblissitt]

@Pilow I've been wondering lately if I should do a Rolling Pool Reboot before any Rolling Pool Update. This might allow me to identify problems in advance and I would also be installing the patches on freshly-rebooted hosts.

[Pilow]

@robertblissitt yup, afterward this seems to be a good best practice...
my hosts were up for 4 month, and because of DNS resolution problem had 77 patches to catch up (80 for one with advanced telemetry enabled)

a rolling reboot would have probably put in front the initial migration/evacuation problem (and subsequent zombies VMs)

and no patches applied, and no pool in a semi upgraded state

note to my future self, try a rolling reboot first.

[rzr]

New security and maintenance update candidates for you to test!

The whole platform has been hardened with crypto libraries updates.
We also publish other non-urgent updates which we had in the pipe for the next update release.

Important notice

️ Xen Orchestra's sdn_controller users should be aware that OpenSSL was updated to major version 3, causing XCP-ng to reject previously generated self-signed certificates for the SDN Controller: they must be updated manually, accordingly to the guide's procedure.

User feedback is valuable to all, feel free to report success or ask for clarification in the related forum thread.

What changed

OpenSSL and OpenSSH major version update

• openssl: Update to 3.0.9
  • The OpenSSL 3 upgrade is improving the security and maintainability of the system, but has impact regarding certificates generation in sdn_controller, as documented above.
  • To enable backward compatibility with older deprecated APIs, a new package, openssl-compat-10 has been introduced.
• openssh: Update to 9.8p1
  • Note that older ssh-clients (with weak ciphers) will need to update, if connection is rejected.
• libssh2: Update to 1.11.0

Maintenance updates

Virtualization & System

• xen: Update to 4.17.6
  • Xen sources updated to v4.17.6 and synchronization of previously released patches for XSA-477 and XSA-479.
• qemu: Bug fixes
  • qemu would crash when a framebuffer is relocated on a migrated HVM guest.
  • A race condition could cause events to be sent before capabilities negotiation.
• varstored: Update to 1.3.1
  • No further functional change from 1.2.0-3.5 (Fixes for XSA-478 / CVE-2025-58151 were backported).
  • Just syncing with XenServer, rebuilt with openssl-3.

Control plane

• xapi: Update to 26.1.3
  • User agents of clients are now tracked. Fetchable by using Host.get_tracked_user_agents.
  • Now it's possible to delete a VM with a snapshot that has a vTPM associated.
  • Speed up exports for mostly empty disks.
  • Now the tags of VDIs are copied when they are cloned or snapshotted done.
  • Fixed RPU scenario where pool members don't get enabled.
  • Added API for controlling NTP.
  • Fixed falling back to full backups instead of delta backups in cases where a VM was hosted in a local SR with more than 256 disks. This could also cause migrations to fail.
  • Added API to limit the number of VNC connections to a single VM.

UI

• xolite: Update to 0.19.0
  • [VM/New] Added vTPM support.
  • [VM/New] Fix wording in "Memory" section.
  • [TreeView] Scroll to current item in list view.
  • ChangeLog

Storage

• sm: Bug fixes
  • Improve Robustness FileSR GC when a host is offline.
  • Ensure LVM VDI is always active before relink.
  • Remove GC flag DB_GC_NO_SPACE when necessary to avoid errors.
  • Improve error messages when vdi_type is missing on LVM VDIs.
• blktap: Bug fix
  • Fixes a crash happening when scanning a SR with corrupt VHDs.
• lvm2: Update to 2.02.180
  • Add scini device support (Dell PowerFlex).

Network

• netsnmp: Update to 5.9.3
• openvswitch:
  • Rebuild with openssl-3 plus minor maintenance change.
• gnutls: Remove dane tool

Misc

• xcp-ng-release: UX improvement
  • The shell command history now record timestamps to improve consumer support.
• createrepo_c: Update to 0.21.1
• krb5: Synchronized with XenServer 8.4 and rebuilt for OpenSSL 3.
• ipmitool: Update to 1.8.19
• libarchive: Update to 3.6.1
• trousers: Update to 0.3.15 and rebuild for OpenSSL 3.
  • This version includes security fixes for known vulnerabilities in earlier upstream version, deemed not exploitable realistically on XCP-ng.
• wget: Update to 1.21.4

Note that libraries updates (libopenssl, notably) impacted several other packages which had to be rebuilt (some had to be patched too). Refer to the package list below.

Drivers updates (check details below)

More information about drivers and current versions is maintained on the drivers wiki page.

• broadcom-bnxt-en: Update to v1.10.3_237.1.20.0
  • No functional changes expected.
• intel-i40e : Update to 2.25.11
  • PTP-related kernel crash bugfixes for Intel i40e driver version 2.25.11.
  • ️ Google for the "intel <model-name> compatibility matrix" and make sure to update the non-volatile memory in NIC with the matching NVM version, after updating the driver.
  • This is also applicable for the intel-i40e-alt flavour of the driver package.
• intel-ixgbe: Update to 6.2.5
  • More Ethernet PCI Express 10 Gigabit Intel NIC devices are handled (E600 et E610 series).

XOSTOR

In addition to the changes in common packages, the following XOSTOR-specific packages received updates:

• drbd: Reduce the I/O load and time during resync.
• drbd-reactor: Misc improvements regarding drbd-reactor and events.
• linstor:
  • Resource delete: Fixed rare race condition where a delayed DRBD event causes "resource not found.
  • Misc changes to improve robustness LINSTOR API calls and checks.
• sm:
  • Wait for DRBD UpToDate state during LINSTOR VDI resize.
  • Improve LINSTOR error messages in the case of an excessively long VDI resize.
  • Simplify LINSTOR SR scan logic removing XAPI calls.
  • Use worker threads during LINSTOR SR's scan to improve performance.
  • Ensure a XOSTOR volume can't be destroyed if used by any process (outside of the SMAPI environment).
  • Use ss to obtain the controller IP: it's a significant improvement to avoid relying on DRBD commands or XAPI plugins.
  • Avoid issuing errors if the size of a LINSTOR volume cannot be fetched after a bad delete call.
• python-linstor: updated to version 1.27.1. LINBIT's changelog:
  • "Added api method to check the controller’s current encryption state (locked/unlocked/unset)"
• linstor-client: updated to version 1.27.1. LINBIT's changelog:
  • "Added new alias --drbd-diskless to command r td to mimic the option from r c.
  • "Added new sub-command encryption status to show the current locked-state of the controller.

Versions:

• bind: 32:9.9.4-61.el7_5.1 -> 32:9.9.4-63.1.xcpng8.3
• blktap: 3.55.5-6.1.xcpng8.3 -> 3.55.5-6.3.xcpng8.3
• broadcom-bnxt-en: 1.10.3_232.0.155.5-1.xcpng8.3 -> 1.10.3_237.1.20.0-8.1.xcpng8.3
• coreutils: 8.22-21.el7 -> 8.22-22.xcpng8.3
• createrepo_c: 0.10.0-6.el7 -> 0.21.1-3.xcpng8.3
• curl: 8.9.1-5.1.xcpng8.3 -> 8.9.1-5.2.xcpng8.3
• gnutls: 3.3.29-9.el7_6 -> 3.3.29-10.1.xcpng8.3
• gpumon: 24.1.0-71.1.xcpng8.3 -> 24.1.0-83.2.xcpng8.3
• intel-i40e: 2.25.11-2.xcpng8.3 -> 2.25.11-4.xcpng8.3
• intel-ixgbe: 5.18.6-1.xcpng8.3 -> 6.2.5-1.xcpng8.3
• intel-microcode: 20251029-1.xcpng8.3 -> 20260115-1.xcpng8.3
• ipmitool: 1.8.18-7.el7 -> 1.8.19-11.1.xcpng8.3
• iputils: 20160308-10.el7 -> 20160308-10.1.xcpng8.3
• krb5: 1.15.1-19.el7 -> 1.15.1-22.1.xcpng8.3
• libarchive: 3.3.3-1.1.xcpng8.3 -> 3.6.1-4.1.xcpng8.3
• libevent: 2.0.21-4.el7 -> 2.0.21-4.1.xcpng8.3
• libssh2: 1.4.3-10.el7_2.1 -> 1.11.0-1.xcpng8.3
• libtpms: 0.9.6-3.xcpng8.3 -> 0.9.6-3.1.xcpng8.3
• lvm2: 7:1.02.149-18.2.1.xcpng8.3 -> 7:2.02.180-18.3.1.xcpng8.3
• mdadm: 4.0-13.el7 -> 4.2-5.xcpng8.3
• net-snmp: 1:5.7.2-52.1.xcpng8.3 -> 1:5.9.3-8.1.xcpng8.3
• openssh: 7.4p1-23.3.3.xcpng8.3 -> 9.8p1-1.2.1.xcpng8.3
• openssl: 1:1.0.2k-26.2.xcpng8.3 -> 1:3.0.9-2.0.1.3.xcpng8.3
• openvswitch: 2.17.7-2.1.xcpng8.3 -> 2.17.7-4.1.xcpng8.3
• python: 2.7.5-90.el7 -> 2.7.5-92.1.xcpng8.3
• python3: 3.6.8-18.el7 -> 3.6.8-20.xcpng8.3
• python-pycurl: 7.19.0-19.el7 -> 7.19.0-19.1.xcpng8.3
• qemu: 2:4.2.1-5.2.15.2.xcpng8.3 -> 2:4.2.1-5.2.17.1.xcpng8.3
• rsync: 3.4.1-1.1.xcpng8.3 -> 3.4.1-1.2.xcpng8.3
• samba: 4.10.16-25.2.xcpng8.3 -> 4.10.16-25.3.xcpng8.3
• sm: 3.2.12-16.1.xcpng8.3 -> 3.2.12-17.1.xcpng8.3
• ssmtp: 2.64-14.el7 -> 2.64-14.1.xcpng8.3
• stunnel: 5.60-4.xcpng8.3 -> 5.60-5.xcpng8.3
• sudo: 1.9.15-4.1.xcpng8.3 -> 1.9.15-5.1.xcpng8.3
• swtpm: 0.7.3-12.xcpng8.3 -> 0.7.3-12.1.xcpng8.3
• tcpdump: 14:4.9.2-3.el7 -> 14:4.9.2-3.1.xcpng8.3
• trousers: 0.3.14-2.el7 -> 0.3.15-11.1.xcpng8.3
• varstored: 1.2.0-3.5.xcpng8.3 -> 1.3.1-2.1.xcpng8.3
• wget: 1.14-15.el7_4.1 -> 1.21.4-1.1.xcpng8.3
• xapi: 25.33.1-2.3.xcpng8.3 -> 26.1.3-1.3.xcpng8.3
• xcp-featured: 1.1.8-3.xcpng8.3 -> 1.1.8-6.xcpng8.3
• xcp-ng-release: 8.3.0-36 -> 8.3.0-37
• xen: 4.17.5-23.2.xcpng8.3 -> 4.17.6-2.1.xcpng8.3
• xo-lite: 0.17.0-1.xcpng8.3 -> 0.19.0-1.xcpng8.3

XOSTOR:

• linstor: 1.33.1-1.el7_9
• linstor-client: 1.27.1-1.xcpng8.3
• python-linstor: 1.27.1-1.xcpng8.3
• xcp-ng-linstor: 1.2-6.xcpng8.3

Optional packages:

• iperf3: 3.9-13.1.xcpng8.3
• ldns: 1.7.0-21.1.xcpng8.3
• socat: 1.7.4.1-6.1.xcpng8.3

Test on XCP-ng 8.3

If you are using XOSTOR, please refer to our documentation for the update method.

If you are using XenOrchestra's SDN controller please apply the OpenSSL upgrade procedure.

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

• XAPI tests:
  • Check that the NTP servers used by hosts are set to Factory, test changing them to DHCP, or Custom.
  • Check that the console limit is 0 by default, test changing it to 1, and set a timeout.
• System: Check updated tools (ssh, wget, samba, mdadm...)
• Normal use and anything else you want to test.

Test window before official release of the updates

~1 week

[flakpyro]

Installed on my usual selection of hosts. (A mixture of AMD and Intel hosts, SuperMicro, Asus, and Minisforum). No issues after a reboot, PCI Passthru, backups, etc continue to work smoothly. Also installed on a HP GL325 Gen 10 with no issues after reboot.

[ph7]

@rzr said:
xo-lite: 0.18.0-1.xcpng8.3

Well I was and still is on v0.19.0:

I am running an old AMD Ryzen 5 2400GE homelab with NFS
no XOSTOR or SDN.
Tested NTP Def and dhcp, ssh, wget, cont. repl. ...
all worked fine so far.

[rzr]

@ph7 said:

Well I was and still is on v0.19.0:

So you're up to date ! it was a mistake, let me generate an up to date list

[JeffBerntsen]

@rzr Installed and seems to be working normally on my test systems.

[acebmxer]

@rzr

Updated my to AMD Ryzen host in my home lab. No issues with update will monitor and report back any issues.

[acebmxer]

@rzr

Built new Ubuntu 24.04 vm either fresh install from ISO or from cloudint i seem to be having issues. Existing vms seem to be fine. First thought was becuase vm was multiple nics add it might have caused networking issues. Powered off that vm and build new (first as fresh from iso. Second from cloudint) with 1 nic. Same issues...

Update - issues were with script that i updated prior to updating hosts. Fixed script all is working again. Didnt test script on new vm prior to updating host.

[olivierlambert]

I'm not sure it's the right topic for that, probably worth creating a new one

[acebmxer]

@olivierlambert Created new topic.

[gduperrey]

New update candidate for you to test!

A new update for the Xen packages is ready, which brings a significant improvement in live migration performance on AMD systems under heavy load, that we add to the previous batch of updates for a common publication.

---

Maintenance updates

• xen: Improve migration performance on AMD systems under heavy load.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• xen: 4.17.6-4.1.xcpng8.3

What to test

Normal use and anything else you want to test. If you have a pool with AMD processors, we're interested in your feedback regarding live migration under heavy load.

Test window before official release of the updates

~4/5 days

[Andrew]

@gduperrey The new OpenSSL/SSH blocks existing/working RSA keys from older SSH clients. While you can still use a password for SSH, it will block old keys from working which will break things (not good for existing LTS installs). To maintain compatibility add PubkeyAcceptedAlgorithms +ssh-rsa to /etc/ssh/sshd_config

[flakpyro]

@gduperrey Tested this on the same hosts i already have running the testing updates from earlier. No issues. Mixture of AMD and Intel.

[gduperrey]

@Andrew I just pinged Philippe (rzr) internally to ask him to look into this

[rzr]

@Andrew said:

@gduperrey The new OpenSSL/SSH blocks existing/working RSA keys from older SSH clients. While you can still use a password for SSH, it will block old keys from working which will break things (not good for existing LTS installs). To maintain compatibility add PubkeyAcceptedAlgorithms +ssh-rsa to /etc/ssh/sshd_config

Hi @andrew, thank you for your feedback, the fallback option you're suggesting will work but it will downgrade the security of your system, we suggested to update clients:

"Note that older ssh-clients (with weak ciphers) will need to update, if connection is rejected."

Let me make it more explicit that older keys should be also refreshed:

  ssh-keygen # To generate new $identity_file 
  ssh-copy-id \
        -i $identity_file \
        -o HostKeyAlgorithms=+ssh-rsa \
        -o PubkeyAcceptedAlgorithms=+ssh-rsa \
        $user@$host
  ssh $user@$host

Ideally this can be done before the update, but let's us think if we have a better strategy to provide a smoother experience, meanwhile if anyone is curious please check:

https://www.openssh.org/releasenotes.html

https://www.openssh.org/txt/release-8.8

"We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519)."

https://datatracker.ietf.org/doc/html/rfc8332

As I understand, RSA is safe unless it was coupled with SHA1 hash function which was then decoupled in later versions (and then obsoleted in V_8_7_P1-4-g234475025 with https://github.com/openssh/openssh-portable/commit/2344750250247111a6c3c6a4fe84ed583a61cc11 "The use of RSA/SHA1 can be re-enabled by adding "ssh-rsa" to the
PubkeyAcceptedAlgorithms directives on the client and server.") .

Regen keys will be needed, better sooner than later, meanwhile we could support weak keys clients during a short (TBD) deprecation period.

Update: I think I was able to reproduce the issue @andrew reported using a RSA key generated with
OpenSSH_5.1p1 Debian-6.maemo2, OpenSSL 0.9.8e 23 Feb 2007

ssh-keygen -lf ~/.ssh/id_rsa.pub
2048 SHA256:abcde+0123456789012345678901234567890/vwxyz user@Nokia-N810-43-7 (RSA)

Used along a later client (in a debian chroot jessie amd64) :
OpenSSH_6.7p1 Debian-5+deb8u4, OpenSSL 1.0.1t 3 May 2016

While it worked as expected (in a debian chroot stretch amd64) : with:
OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019

So to conclude using rsa keys need ssh-7+ while ssh6 can be used using stronger cypher like id_ed25519 (not rsa).

PS: this post may be updated

0 djmdjm committed to openssh/openssh-portable

upstream: After years of forewarning, disable the RSA/SHA-1

signature algorithm by default. It is feasible to create colliding SHA1
hashes, so we need to deprecate its use.

RSA/SHA-256/512 remains available and will be transparently selected
instead of RSA/SHA1 for most SSH servers released in the last five+
years. There is no need to regenerate RSA keys.

The use of RSA/SHA1 can be re-enabled by adding "ssh-rsa" to the
PubkeyAcceptedAlgorithms directives on the client and server.

ok dtucker deraadt

OpenBSD-Commit-ID: 189bcc4789c7254e09e23734bdd5def8354ff1d5

[stormi]

Although disabling ssh-rsa is the right thing to do from a security perspective, we'll see what we can do to smoothen the transition.