XCP-ng 8.3 updates announcements and testing

[TeddyAstie]

@leroyj
We already know that our k10temp module doesn't have support beyond Zen 2, and needs to be updated. That's not incredibly complex, but still needs to be done.

I was wondering if the amd cpu temp be probed the way you did for the intel cpu.

No, AMD temperature infos are not exposed through MSR but through PCIe/MMIO (through various subsystems, like "SMU" and other ones). Like what does https://github.com/torvalds/linux/blob/master/drivers/hwmon/k10temp.c or https://github.com/ocerman/zenpower.
(regarding the AI draft, there is no documented MSR 0xc0010292 in neither APM nor public Zen3 PPM)

That doesn't require any specific Xen support aside the right Linux drivers in Dom0.

[gleh]

New maintenance update candidates for XCP-ng 8.3 LTS

This batch of updates contains performance improvements, fixes, a guest tools update, and other improvements.

What changed

Virtualization & System

• kernel: Fix potential deadlocks in the NFS subsystem.

• xen:

  • Enable Viridian enlightenments to prepare for support of Windows VMs with >64 vCPUs
  • Sync with XenServer release 4.17.7-2. It fixes XSA-511, which didn't affect XCP-ng.

• systemd: Fix detection of NVMe controllers with multiple namespaces

Control Plane

• xapi:

  • Improve RPU stability by only migrating VMs to already-updated hosts
  • Improve VDI migration performance by pipelining NBD writes

• guest-templates-json:

  • Sync with XenServer's guest-templates-json-2.0.16-1. It removes the Kylin Linux 7 Guest template.
  • Disable Viridian in the "Other install media" template by default. As a reminder, Windows VMs must use Windows templates.

Storage

• blktap:

  1. A fix for a tapdisk crash happening when tapdisk pauses a VDI when a coalesce is in progress. This issue was caused by a use-after-free bug. We took time to thoroughly audit the code around pause/commit/cancel operations, and thus robustify the pause and commit code in this release.
  2. Customers could get tapdisk with infinite stalled IO on QCOW2 VDI. A race-condition was identified that could allow to miss a notification from the front-end and therefore never check the blkif ring for new requests.
  3. On a tapdisk running on a supporter node, CBT metadata does not flush before a pause operation. During the pause operation, the master host modifies the CBT metadata, then after the unpause the old metadata are flushed and discard master's modifications. Finally the CBT logs are detected as inconsistent and discarded.
  4. Grow up the QCOW2 metadata cache to help on big disks.

• sm:

  • Robustify iSCSI calls against race condition and bad access using lock context manager.

  • Retry host key tag removal when XAPI is unreachable or restarted to prevent errors and finish action.

  • Explicitly notify a missing LINSTOR resource definition instead of displaying a human-unreadable trace.

  • Repair LINSTOR RAW volume creation. Regression caused by QCOW2 code refactoring.

  • Fix LINSTOR SR creation when a VG is missing on a host. Changes in behavior brought by an update of LINSTOR RPMs broke this feature, which prevented the creation of an SR in this situation. We're expanding the test suite to prevent this in the future.

  • To help recovering from an eventual LINSTOR database corruption, it's now backed up regularly and after every major operation, locally on the master, and on the DRBD LINSTOR database volume.

  • Fixed an issue on shared LVM-based SR (LVMoISCSI, LVMoHBA) where a failed live leaf coalesce of a QCOW2 image on a slave host caused the SRs to remain in an SR_FAILURE_1200 error state until the VM using the VDI was stopped or tapdisk was manually paused.

  • Reduce qemu-img log verbosity in SMlog.

  • Fix LVM metadata corruption on slave when CBT log deletion is triggered on VDI activation.

  • Prevent an incorrect online coalesce to be reported as successful and causing issues with the QCOW2 chain.

  • Add QCOW2 support during interrupted leaf-coalesce operations on file-based SRs.

Network

• openvswitch: The openvswitch package no longer installs the unused openvswitch-cfg-update XAPI plugin file, which has been superseded by openvswitch-config-update from xapi-core. This is only meant to avoid confusion and brings no functional change.

UI

• xo-lite:
  • [Treeview] Add VM tree actions (PR #10304)
  • [Pool/networks] Add the possibility to create new network or bonded network (PR #10145)
  • [VM] Add VDIs page with table and side panel (PR #10269)
  • [Host/Network] Add ability to rescan physical network interfaces (PIFs) (PR #10147)
  • Fix inconsistent spacing in side panel cards (PR #10279)
  • Fix missing collapse buttons on pools and hosts in the tree sidebar (added hasChildren prop) (PR #10244)
  • [Pool/networks] Add the possibility to create new internal network (PR #10235)

Drivers and Middleware

• vendor-drivers: The out of tree microsemi-aacraid driver from the RPM has proven to be unreliable. It has several issues, is missing patches from upstream and hasn't been updated for a while. So, we switch back to the in-tree driver just like XCP-ng 8.2 was doing.

• xcp-ng-pv-tools:

  • Update to XCP-ng Windows Guest Tools 9.2.385.
  • Include the XenTools-fix-9.2.351.msp hotfix tool for users running 9.2.350.

Others

• openssl: Update to 3.5.5

Versions

• blktap: 3.55.5-9.5.xcpng8.3 -> 3.55.5-11.1.xcpng8.3
• gpumon: 24.1.0-96.1.xcpng8.3 -> 24.1.0-98.1.xcpng8.3
• guest-templates-json: 2.0.15-1.1.xcpng8.3 -> 2.0.16-1.1.xcpng8.3
• kernel: 4.19.19-8.0.46.10.xcpng8.3 -> 4.19.19-8.0.46.12.xcpng8.3
• openssl: 3.0.9-2.0.1.3.xcpng8.3 -> 3.5.5-1.3.xcpng8.3
• openvswitch: 2.17.7-4.1.xcpng8.3 -> 2.17.7-4.2.xcpng8.3
• sm: 3.2.12-23.5.xcpng8.3 -> 3.2.12-25.1.xcpng8.3
• systemd: 219-57.5.xcpng8.3 -> 219-57.5.1.xcpng8.3
• vendor-drivers: 2.0.3-1.1.xcpng8.3 -> 2.0.3-1.2.xcpng8.3
• xapi: 26.1.16-1.2.xcpng8.3 -> 26.1.19-1.1.xcpng8.3
• xcp-featured: 1.2.1-4.xcpng8.3 -> 1.2.1-5.xcpng8.3
• xcp-ng-pv-tools: 8.3-18.xcpng8.3 -> 8.3-19.xcpng8.3
• xen: 4.17.6-12.3.xcpng8.3 -> 4.17.7-2.3.xcpng8.3
• xo-lite: 0.24.0-1.xcpng8.3 -> 0.25.0-1.xcpng8.3

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

As usual, normal use and anything else you want to test.

Test window before official release of the updates

4 days

We would like to thank users who shared feedback since our last call for testing:
@bufanda, @acebmxer, @MajorP93, @flakpyro, @Andrew, @mthird, @manilx, @XCP-ng-JustGreat

[flakpyro]

@gleh said:

Disable Viridian in the "Other install media" template by default. As a reminder, Windows VMs must use Windows templates.

Working on testing this update out however this change has me a bit concerned. Will existing Windows VMs using "Other install media" be affected by this? Or is this just for VMs created going forward? Back in 2024 when we migrated from VMware using V2V all Windows VMs we migrated were created with "Other Install Media" as I'm guessing that's just how the migration process worked at the time.

[dinhngtu]

@flakpyro These template updates only concern new "from scratch" VMs. Older VMs should not be impacted at all.

But if some of your VMs are missing Viridian settings, you may run into poor performance and/or odd Windows issues. You can correct them using the instructions documented here: https://docs.xcp-ng.org/vms/#enabling-viridian-extensions

[flakpyro]

@dinhngtu Thanks for clarifying. I can confirm our windows VMs that were migrated with V2V with "Other Install media" do show having this enabled.

I will deploy to my usual batch of test hosts and update this post with my results.

[dinhngtu]

@flakpyro Note that various important Viridian settings are not yet settable via Xen Orchestra, so you might want to confirm using xe instead.

[flakpyro]

@dinhngtu We did initially run into this issue a number of years ago and were able to fix it by following this kb article at the time:
https://support.citrix.com/external/article?articleUrl=CTX692813-xenserver-potential-performance-issues-on-imported-windows-vms&language=en_US

Checking s few random migrated windows VMs and it appears these flags are still set as they should be.

[acebmxer]

@gleh

Installed patches on home lab. No initial issues found. Existing Windows 11 vm no issues with it on basic vm functions. Created new Windows 11 vm with 26h2 iso no issues. Also installed guest tools in both vms.

[flakpyro]

Installed on my usual test hosts without any issues so far.

[olivierlambert]

That's a pretty beefy update! It's great to see the live storage migration performance improving by a very nice margin