Given that BIOS and everything is updated to latest version possible.
First thing I do then with these kinds of symptoms, is to disable all kinds of power management and/or C-states in BIOS.
Some combinations of OS and hardware, just doesn't work properly.
If for nothing else, it's a easy non-intrusive test to do.

Update: I see that your motherboard has an IPMI interface. If the issues happen again, after you've disabled power management/c-states. You could use the remote functionality of the impi, to hopefully get some more info from the sensors and stuff.

I haven't had any issues with NUMA node balancing, do you need this for a specific high performance application? Generally speaking it should "just work".

There are some things you can do to optimize but I'd only go down that road if you are running into issues with very wide VMs.

Also, are we talking cross socket NUMA nodes (e.g. multiple CPUs) or just nodes within an EPYC CPU?

Unfortunately, looks like my issue was ultimately a failed disk in a RAID 0 array. The errors XCP-ng was throwing were definitely misleading.

I just rebuilt the array as RAID 10 and have reinstalled XCP-ng v8.3. I should have the entire virtual infrastructure rebuilt in no time, before services this weekend.

Can you try on latest release channel and see if you still have the error? Thanks!

Enjoy XO & XCP-ng then 🙂

@olivierlambert

I resolved it, logged in the XCP-ng console and set the NTP servers to default. All good now.
Thanks

xcp-ntp2.png

I have yet to successfully reboot a host from XO. I have tools installed on three out of four VM. I try shutting down the VM without tools, then issue reboot host in XO GUI. Get warning:
Some VMs cannot be migrated without first rebooting this host. Please try force reboot.

Is this a setting perhaps?

I'm always needing to shutdown all VMs before I can reboot the host. It is a single host environment, so there will never be an available server to migrate VMs to.

Rebooting in XSCONSOLE causes the host to be unavailable in XO. I have to remove then add the host back in XO.

Can anyone shed light as to what I'm doing wrong?

Do I just need to always run the following after a reboot?
xe host-enable host=xxxx

Preferably after, since the PV drivers contain some VM-specific settings that may be wiped by Sysprep.

FWIW...having the same issue with Server 2025.

@dinhngtu said in Issues with XCP-ng Windows PV Tools 9.0.9000.0 Testsign Pre-release:

With 9.0.9000.0, you could sometimes run into this issue if you reboot too early after installing. This will also be fixed in the next release.

I just did two more tests and the boot-up failure happened both times. On the last test I waited some time (like 30 to 60 sec) after the driver installation before rebooting (using Windows restart from the start menu). Even though I waited before rebooting I still encountered the boot-up problem:

Windows is spinning on this screen and doesn't finish booting:
ef30a370-5897-41e6-a0f1-41fa3720dacb-image.png

(In regards to uninstalling XenServer / Citrix Tools)

I tested uninstalling XenServer tools v9.4.0 on Windows and it seems to uninstall cleanly. (the xcp-ng docs mention there have been problems uninstalling everything cleanly in the past)

Also since XenServer tools v9.3.1 there is a new purge function:

uninstall.exe purge - (9.3.1 and later) Resets the VM to a clean state as it was before any part of the XenServer VM Tools were installed.

Full uninstall of all XenServer VM Tools components
https://docs.xenserver.com/en-us/citrix-hypervisor/vms/windows/vm-tools.html#full-uninstall-of-all-xenserver-vm-tools-components

What does the purge option remove?
https://docs.xenserver.com/en-us/citrix-hypervisor/vms/windows/vm-tools.html#what-does-the-purge-option-remove

I wasn't aware of this plugin even existing (thanks for the discovery). I'm not sure it's really maintained, but we have a clear goal of getting one officially maintained anyway.

@olivierlambert

Which one, powershell or WAC. They are kind of one in the same when you look at how WAC makes the calls to do the work, there's a button to reveal the powershell behind the button clicks so you might be able to get both at once.

But WAC draws nice graphs and gives us a GUI to get the work done. Yes I know, real Admins don't use GUI. Whatever, AI will eat our jobs anyway, just trying to stick it out another 9 years at this point.

When I get time, I'll still probably take a swing at this task for WAC, because to me this would be valuable. Big learning curve though.

Now if MS would finally add Group Policy to WAC and fix some of the DHCP and DNS features that are still in preview...

@dinhngtu Ah thank you! Idk why I couldn't find that on my own, I searched for a while.

According to the documentation, you only need xfsprogs.

I'm not aware of such problem it's like the VM was kept in limbo. You did the right thing with your commands.

Hello,

I've tried benchmarking several Windows VMs from another client and am getting expected performance:

Read: .\diskspd.exe -c1G -d10 -Su -Sw g:\data.dat Write: .\diskspd.exe -c1G -d10 -Su -Sw -w100 g:\data.dat Windows Server 2022: Read IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 1151533056 | 17571 | 109.72 | 1755.53 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 1151533056 | 17571 | 109.72 | 1755.53 Write IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 966000640 | 14740 | 92.12 | 1473.98 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 966000640 | 14740 | 92.12 | 1473.98 Windows 11 24H2 (with Eset): Read IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 1100283904 | 16789 | 104.81 | 1677.01 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 1100283904 | 16789 | 104.81 | 1677.01 Write IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 931594240 | 14215 | 88.82 | 1421.07 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 931594240 | 14215 | 88.82 | 1421.07

Several questions:

Are you running benchmarks on a separate virtual disk? If not, could you try again using a share created from a separate virtual disk? Could you run CrystalDiskMark locally on these VMs and see the performance? You mentioned that Eset security takes up 100% CPU during the benchmark. Could you try temporarily uninstalling Eset and testing again?

Thanks for your feedback. In the meantime, I will plan a "spike" to investigate the status of USB speed on passthrough.

@andsmith windbg is also a regular win32 app and is included in any recent windows SDK. Here: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/

You just need windbg so deselect all the other crap.