XCP-ng 8.3 updates announcements and testing

[gb.123]

This post is deleted!

[gb.123]

This post is deleted!

[stormi]

@gb.123 I doesn't seem directly related to this thread, which is dedicated to testing update candidates. Maybe open a new thread?

[gb.123]

@stormi

Done.
Since this I checked this after a while, I wasn't sure if it was related to update or independent. I have opened the new thread as you suggested in the 'Network' section

[gduperrey]

New update candidates for you to test!

A new batch of non-urgent updates is ready for user tests before a future collective release. Below are the details about these.

• edk2: Fix "Guest has not initialized the display (yet)." error.
• intel-igc: Fix a possible update issue due to a recent package name change.
• intel-microcode:
  • Latest Intel release microcode-20250211:
    • Security updates for: INTEL-SA-01166, INTEL-SA-01213, INTEL-SA-01139, INTEL-SA-01228
    • Updates for multiple functional issues
    • Upstream update drops files for older Sapphire Rapids steppings, we kept the previous versions
• netdata:
  • Update to Netdata v1.47.5
    • Fix dmesg warnings due to setuid+capabilities on xenstat plugin
    • Improve systemd service restart with a custom script waiting for Netdata to be fully up-and-running before stopping it.
• openssl: Add security fixes from upstream: CVE-2019-1547, CVE-2019-1551, CVE-2019-1563
• qemu: Backport a security fixe (CVE-2023-3354) for QEMU VNC server vulnerability
• r8125-module: Disable some performance functionalities in the driver (TXchecksum/SG/TSO) by default to workaround bugs on Windows Server 2022 guests. These can be re-enabled using 'ethtool -K eth0 tx on tso on sg on'
• sm:
  • Fix issue where users may encounter problems with HPE Nimble arrays: unable to mount iSCSI LUNs, non-functional or imperfect multipathing.
  • Regarding Large Block driver, always enable the VG on the emulated device.
  • Prevent corruption in the LINSTOR KV-store caused by a race condition between user calls and GC.
• systemtap: No functional changes. Fix compilation for compatibility with new gcc version.
• xapi: Re-enabled nested virtualization in 8.3, with the same limitations as in 8.2.
• xcp-emu-manager: No functional changes, Fix rpm spec file for new cmake version
• xcp-ng-release: Update cipher list in .curlrc
• xcp-ng-xapi-plugins:
  • Add new service plugin to manage (start, stop, ...) XCP-ng services
  • Add a new ipmitool plugin to get information from ipmitool that:
    • Returns information about sensors
    • Returns ipmi lan information
• xen:
  • Re-enabled nested virtualization in 8.3, with the same limitations as in 8.2.
  • Fix XSA-467 / CVE-2025-1713
• xo-lite: update to version 0.8.0. For more information you can read latests posts on the Xen Orchestra blog: 0.7.1 & 0.8.0.

Regarding the nested virtualization, Xen-Orchestra is not yet updated to allow this in 8.3, even if we see the option in the advanced tab of the VM. They are working on it and this will come in a future update.

To actually enable the nested possibility, it must be done on the command line with xe.

Once your VM is created:

xe vm-param-set platform:nested-virt=true uuid=<vm-uuid>

To check this:

xe vm-param-get param-name=platform uuid=<vm-uuid>

You should have a line similar to below, with the nested-virt: true parameter visible:

nested-virt: true; timeoffset: 0; exp-nested-hvm: true; secureboot: false; device-model: qemu-upstream-compat; viridian: true; nx: true; acpi: 1; apic: true; pae:true; hpet:true

To deactivate it:

xe vm-param-remove param-name=platform param-key=nested-virt uuid=<vm-uuid>

For XOSTOR users:

• See the description for sm above.

(Reminder: XOSTOR is still in beta stage on XCP-ng 8.3)

Optional packages:

• Alternate Driver: Updated to newer version.
  • atlantic-module-alt:
    • Update vendor version of driver 2.5.12
    • Disable LRO for reliable bridging per README
    • More information about drivers and current versions is on the drivers page: (https://github.com/xcp-ng/xcp/wiki/Drivers).

Test on XCP-ng 8.3

From an up-to-date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

• edk2: 20220801-1.7.7.2.xcpng8.3
• intel-igc: 5.10.226-2.xcpng8.3
• intel-microcode: 20250127-1.xcpng8.3
• netdata: 1.47.5-4.1.xcpng8.3
• openssl: 1.0.2k-26.2.xcpng8.3
• qemu: 4.2.1-5.2.10.1.xcpng8.3
• r8125-module: 9.012.04-2.xcpng8.3
• sm: 3.2.3-1.17.xcpng8.3
• systemtap: 4.0-5.1.xcpng8.3
• xapi: 24.19.2-1.10.xcpng8.3
• xcp-emu-manager: 1.2.0-2.xcpng8.3
• xcp-ng-release: 8.3.0-29
• xcp-ng-xapi-plugins: 1.12.0-1.xcpng8.3
• xen: 4.17.5-4.2.xcpng8.3
• xo-lite: 0.8.0-1.xcpng8.3

Optional packages:

• Alternate drivers:
  • atlantic-module-alt: 2.5.12-1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~ 4/5 days

[bleader]

Home host updated successfully, no issue.

[flakpyro]

@gduperrey

installed on 2 test machines

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Both machines installed fine and all VMs came up without issue after.

[XCP-ng-JustGreat]

Latest test updates were applied to three-node home lab pool (3 x Dell OptiPlex SFF 7040 i7-6700/48GB RAM/256GB NVMe boot drive/TrueNAS Core 10Gbps-attached NFS shared storage) without incident. VMs appear to running normally, live migration between hosts works fine and no apparent problems so far.

[abudef]

The update went fine and everything is working fine.

@gduperrey said in XCP-ng 8.3 updates announcements and testing:

xapi: Re-enabled nested virtualization in 8.3, with the same limitations as in 8.2.

Since I keep bothering with nested virtualization here on the forum, I of course immediately tried the support in 8.3

Setup:

• HW XCP-ng 8.3
• • Nested XCP-ng 8.3
• • • Windows Server 2025
• • • Debian 12.9

Windows installation on the nested hypervisor went ok and the system seems to be working fine.
The problem occurred with Debian. ISO 12.9 netistall, UEFI was used. The system boots up and shows the notorious install screen: Graphic install, Install ... Regardless of the type of installation chosen, immediately after starting it, nested hypervisor XCP-ng 8.3 crashes and reboots. By the way, this problem with Debian is also on VMware - if I use nested XCP-ng 8.3 there, the Debian installation crashes it just the same.

[TeddyAstie]

@abudef Note that even with this update, nested virtualization is still not really supported in XCP-ng 8.3.
It's there, you can enable it at your own risk. It broke due to some change in XAPI (even though Xen hypervisor had "support" for it).
It never actually got removed from Xen hypervisor (it was marked experimental in Xen 4.13 used in XCP-ng 8.2, it is also the case for Xen 4.17), although nothing really changed, it still has the same issues and limitations as said.

The current state of nested virtualization in Xen is quite clumsy and there are future plans to remake it properly from ground without taking shortcuts and have proper tests to back it.

Aside that, after some experiments, it seems that mostly nested EPT is incomplete/buggy, so your L1 hypervisor should not rely on it. You should add hap=0 to nested XCP-ng Xen cmdline. Beware that it will imply a pretty large performance hit, but I had more consistent results with this.
I am quite suprised that Windows works while Linux don't, maybe it is somewhat related to PV drivers ?

[ph7]

@gduperrey

I updated my test host and all seems to work fine.
But I have 1 question:
Do I need to disable the testing repo or is it removed at the reboot?

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

[10:33 x1 ~]# yum repolist 
Inlästa insticksmoduler: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
förråds-id                                 förrådsnamn                                           status
xcp-ng-base                                XCP-ng Base Repository                                4 376
xcp-ng-updates                             XCP-ng Updates Repository                               125
repolist: 4 501

[dthenot]

@ph7 It's only enabled for the two yum command with the --enablerepo explicitly used.
It's disabled in the config otherwise.
No need to do anything

[gduperrey]

Update published: https://xcp-ng.org/blog/2025/03/12/march-2025-maintenance-update-for-xcp-ng-8-3/

Thank you for the tests!

[manilx]

@gduperrey 1 HomeLab-Pool (via cli: yum update and reboot), 3 Business-Pools (via RPU), total 7 servers updated without issues.

[bvitnik]

@TeddyAstie Is the list of "issues and limitations" of nested virtualization under Xen documented somewhere?

[stormi]

@bvitnik It's better to consider it as working by chance in some scenarios, but I think Teddy can give you details

[archw]

@gduperrey
Is a reboot required for this batch of updates?

[Andrew]

@archw Yes.

[archw]

@Andrew
Thanks!

[stormi]

@archw It's written in the blog post