XCP-ng 8.3 updates announcements and testing

[abudef]

@Greg_E Please try restarting the toolstack once more.

[Greg_E]

@abudef

Restarting the tool stack fixed it. Thanks!

[bogikornel]

@dthenot After the upgrade, it is not possible to create a new machine from a template if the VPS disk is not on the SR where the template disk is.

P.S.: Unfortunately, moving the disk from one SR to another does not work. After moving the disk it becomes corrupt, and after reset it will not boot.

[HenrikSchmidt]

Is there a nightly built iso that I could test?
my problem is that I have no network interface realtek 8125 that functions in 8.2 or 8.3.
I would love to check if there has been any changes made that are coming that would solve this.
the driver for r8125 in xcp-ng is 9.012, latest downloadable is 9.015 at realtek but I do not know how to get it compiled and installed.

[gduperrey]

@HenrikSchmidt We don't currently offer regular ISO builds. The most recent ISOs are available.

The driver you're talking about is integrated into the kernel, and to my knowledge, there are no updates for it at the moment. @Andrew is currently working on an alternate driver package, but the PR seems to be waiting for his response.

This type of driver can be loaded at the beginning of the installation with an ISO by pressing F9 when the menu prompts. You can then load an additional driver for the installation, and install it on the future system later in the procedure when prompted.

As this is not an issue with the latest updates, I suggest you create a dedicated thread. This would give you better visibility and help from the community on this issue.

[HenrikSchmidt]

@gduperrey
Ok, I havea thread about my problem but it has not gotten any solutions so far.
The realtek is distributed as source files, not a package so I cant just add it in without knowing how to make a package. I was just hoping that the 8125 package in the distibution might have been updated later then the official distros.

Thanks anyway.

[XCP-ng-JustGreat]

Latest test updates applied to 3-host Dell OptiPlex 7040 SFF (i7-6700) pool. Upgrade went smoothly and everything appears to be working normally. I did notice the flatlined stats issue mentioned previously, but the xe-toolstack-restart seemed to bring those back to life. Thanks @Greg_E and @abudef for catching and addressing that.

[abudef]

@XCP-ng-JustGreat Until this is fixed, you need to restart the toolstack after each host restart, which is not often done, so it's not really a problem.

[flakpyro]

@gduperrey

After running this for a week now i have come across an issue with EFI hosts with these testing updates. Sometimes when booting a VM or rebooting after a windows update the VM will hang at "Guest has not initialized display yet" and not boot. Doing a force shutdown and powering on again solves it. It has happened twice so far to unrelated windows VMs both in my homelab and in our test lab at work. Not sure how to provide more info or logs when it happens however.

Edit: One common thing between both these VMs is they both are using GPU Passthru, if that could at all matter.

[Greg_E]

For some reason, my hosts stop displaying the stats after about 2 days. I did a rolling pool reboot 2 days ago, then restarted the toolstack on each of 3 hosts (my pool) to restore the stats. As of right now, only 1 of the hosts has the stats working still.

The fourth identical "server" of the same hardware is just running the regular release cycle, it still has stats after many days/weeks being up, so not hardware related.

I also wanted to ask, how often updates come out for this "branch" and is there anything special we need to do to get them installed?

[flakpyro]

This happened again today when rebooting a Windows Server on my test host running these updates:

A "Force Reboot" does not solve it, i have to do a "Force Shutdown" and power on again. It will hang at that screen indefinitely. I can reproduce this on two hosts now. It seems if i reboot again immediately after the fact it will be fine but after it has sat running awhile it will always occur. Not sure if it may be GPU passthru releated. I can clone another Windows VM that has no passthru device enabled if that would help.

[stormi]

@Greg_E We announce updates here. There's no frequency set in stone. We're working on the next wave of updates, that is ready except that we found a regression so we're working on fixing that first.

If you're adventurous, there's a way to install this next wave earlier, but until it passed internal checks, I don't guarantee that your pool doesn't explode with these.

[stormi]

@flakpyro Could you export the output of xen-bugtool -y and make it available somewhere privately? This will contain all logs and information about the hardware. Let us know also of times where you rebooting a VM and got this issue.

[Greg_E]

@flakpyro

Which version of Server are you running? UEFI, vTPM, Secureboot?

I don't have an Windows Server on my lab right now, but I can give it a quick check and see. Not sure I can pass through the iGPU in my lab though, and production doesn't have a GPU (real servers with ASpeed BMC).

[flakpyro]

@Greg_E

The VM is EUFI with no vTPM or Secure Boot enabled. The CPU us a "Xeon E-2336 CPU @ 2.90GHz" Running in a Super Micro Server. We use these servers at remote sites to run a number of VMs including a "Blue Iris" server with an Nvidia T1000 GPU passed thru to it, i have one such servers as a test server as well. The second machine doing it is a Minisforum MS-01 also with a T1000 GPU Passed thru in my home lab. The OS in both cases is Windows Server 2025. Over the weekend i cloned a fresh copy without GPU Passthru to see if it occurs with no GPU. We have about 15 of these VMs running at remote locations on the stable 8.3 patch branch that are not doing this.

It should be noted though that these VMs did need to be customized to allow Blue Iris to run without BSODing the VM on Intel CPUs. Thread can be found here: https://xcp-ng.org/forum/topic/8873/windows-blue-iris-xcp-ng-8-3/35?_=1746455850378 but in the end the following needed to be applied to a VM to keep it from BSODing when running Blue Iris on new Intel CPUs:

xe vm-param-add uuid=... param-name=platform msr-relaxed=true

@stormi It does not seem to do it every time, it seems the VM must run for sometime and then be rebooted to cause it to happen. I have created a bug report file from the host as requested and will DM you a link to it! The last time i experienced this would have been at "May 2, 2025 at 4:18 PM (3 days ago)" according to our XOA appliance.

[Greg_E]

@flakpyro

I'm guessing it is something to do with the T1000. I've had so flakyness with them in our workstations. You don't have an older K620 or something do you? I'm also wondering if this might be a case for an Intel ARC GPU, they weren't out when I did my last workstation refresh or I'd probably be using them with Creative Cloud applications.

Also can you try change the Management Agent Service from auto-start to delayed auto-start? Server 2025 has been a bit odd for me for a while, and that was back running on Intel hosts (though they are old CPUs).

I'll see if I can get a VM up this afternoon, but I'm on AMD with the iGPU so not sure it even supports the acceleration that you need out of the T1000 for the camera recorders. I need to go through and configure each host to pass through the GPU, audio, and maybe some USB and then reboot. But if it works, it might make a nice Handbrake transcoding VM for me, dump files in, get back to work while it does it's thing.

[flakpyro]

@Greg_E I do not have anything older like a K620, i do have a GeForce GTX 1650 in another machine that is also doing this but i believe that's the same generation as the T1000 (Turing). I agree i think ARC could be a great replacement for this application in the future.

Since this is occurring at boot before the OS loads i'm not sure if this would be management agent related?

[Greg_E]

@flakpyro

Not going to be able to test this, my little mini-lab is having a very hard time with passthrough, to the point where the host won't boot. Tried this on my backup host (for backup DR testing) and finally got it back after the last couple of hours fooling with turning passthrough on and then back off. Sorry.

[stormi]

New update candidates for you to test!

As we move closer to making XCP-ng 8.3 the new LTS release, taking over from XCP-ng 8.2.1, a new batch of update candidates is now available for user testing ahead of a future collective release. Details are provided below.

• biosdevname: Update as a dependency for another component.
• blktap: Fix: enable NBD client only after completing handshake
• cyrus-sasl: Fix for CVE-2022-24407 (not directly affecting XCP-ng in normal use).
• gpumon: Rebuild for XAPI update.
• intel-ice: Update ice driver to v1.15.5
• kernel: improve timer handling for better compatibility with hardware.
• plymouth: Packaging update. No visible changes.
• psmisc: Update to version 23.6.
• python-urllib3: Update to version 1.26.30.
• rsync:
  • Update to version 3.4.1
  • Fixes for CVE-2024-12084, CVE-2024-12085, CVE-2024-12086, CVE-2024-12087, CVE-2024-12088, CVE-2024-12747
  • The rsyncd configuration and systemd unit files now come in a separate package named rsyncd-daemon, not installed by default
• smartmontools: Update to version 7.4
• xapi:
  • Drop FCoE support when fcoe_driver does not exists
    • FCoE support will be removed from next versions
  • No more CPU checks for halted VMs in cross-pool migration
  • Move CPU check to the target host during cross-pool migration
  • Serialize all PCI and VUSB plugs to keep them ordered
  • Fixes multiple issues in periodic scheduler
  • Fixes multiple issues in the way XAPI handles RRD metrics
  • Improve SR.scan by reducing a racing window when updating the XAPI db
  • A lot or maintenance-related changes, XAPI being a very active project.
• xcp-featured: rebuilt for XAPI
• xcp-ng-release: update copyright years and EULA
• xen:
  • Improve support for Zen 5 and Diamond Rapids CPUs
  • IOMMU logic improvements and fixes
  • add PCI quirks for problematic hardware (e.g Cisco VIC UCSX-ML-V5D200GV2)
  • fix emulation of MOVBE
• xenserver-status-report: maintenance update.
• xha:
  • Support configurable syslog printing
  • Fixes issue where sub-threads can't be scheduled enough resources.

Test on XCP-ng 8.3

From an up-to-date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

• biosdevname: 0.3.10-5.xcpng8.3
• blktap: 3.55.5-2.1.xcpng8.3
• cyrus-sasl: 2.1.26-24.el7_9
• gpumon: 24.1.0-40.1.xcpng8.3
• intel-ice: 1.15.5-2.xcpng8.3
• kernel: 4.19.19-8.0.38.1.xcpng8.3
• plymouth: 0.8.9-0.31.20140113.3.xcpng8.3
• psmisc: 23.6-2.xcpng8.3
• python-urllib3: 1.26.20-3.1.xcpng8.3
• rsync: 3.4.1-1.1.xcpng8.3
• smartmontools: 7.4-2.xcpng8.3
• xapi: 25.6.0-1.4.xcpng8.3
• xcp-featured: 1.1.8-1.xcpng8.3
• xcp-ng-release: 8.3.0-31
• xen: 4.17.5-9.1.xcpng8.3
• xenserver-status-report: 2.0.11-1.xcpng8.3
• xha: 25.0.0-1.1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

We will not be very available on this forum until Monday to help fixing issues if there are any, so don't update too fast if that's a possible problem for you.*

[flakpyro]

@stormi Updated both of my test hosts. Everything rebooted and came up fine.

No VM stats in XO / XOA i see still. I will be curious if this round of updates fixes my EFI / Windows Server reboot hangs.