I have been investigating a network hardware offload issue with the Realtek RTL8125BG 2.5G on the same XCP-ng host computer. This separate issue was resolved by shutting off TX checksum offload on the physical interface.

It made me think that maybe the mouse problem is caused by something similar on the other NIC. The second NIC in the computer is Realtek R8111EPV (1G)(using Linux r8169 driver) which is the one I have been using when I noticed the mouse issue. So I ran a capture test on the XCP-ng host:

tcpdump -i eth1 -v -nn | grep incorrect

and found bad packet checksums on some of the received packets coming in to the Windows VM:

x.x.x.x.443 > [windows.vm.ip].49715: Flags [P.], cksum 0xc656 (incorrect -> 0xb854), seq 1:2433, ack 216, win 2052, length 2432 x.x.x.x.443 > [windows.vm.ip].49717: Flags [P.], cksum 0x2e6c (incorrect -> 0xcff2), seq 1:2921, ack 218, win 501, length 2920 x.x.x.x.443 > [windows.vm.ip].49720: Flags [.], cksum 0x1b11 (incorrect -> 0xb99c), seq 1:2921, ack 213, win 16384, length 2920 ...

Running this on the XCP-ng host stopped the checksum errors:

ethtool -K eth1 gro off # testing only, not persistent

Making this setting persistent:

xe pif-param-set uuid=my_eth1_PIF_UUID other-config:ethtool-gro="off"

After making this change I will monitor the VM to see if the mouse problem returns.

@dinhngtu said in Issues with XCP-ng Windows PV Tools 9.0.9000.0 Testsign Pre-release:

With 9.0.9000.0, you could sometimes run into this issue if you reboot too early after installing. This will also be fixed in the next release.

I just did two more tests and the boot-up failure happened both times. On the last test I waited some time (like 30 to 60 sec) after the driver installation before rebooting (using Windows restart from the start menu). Even though I waited before rebooting I still encountered the boot-up problem:

Windows is spinning on this screen and doesn't finish booting:
ef30a370-5897-41e6-a0f1-41fa3720dacb-image.png

(In regards to uninstalling XenServer / Citrix Tools)

I tested uninstalling XenServer tools v9.4.0 on Windows and it seems to uninstall cleanly. (the xcp-ng docs mention there have been problems uninstalling everything cleanly in the past)

Also since XenServer tools v9.3.1 there is a new purge function:

uninstall.exe purge - (9.3.1 and later) Resets the VM to a clean state as it was before any part of the XenServer VM Tools were installed.

Full uninstall of all XenServer VM Tools components
https://docs.xenserver.com/en-us/citrix-hypervisor/vms/windows/vm-tools.html#full-uninstall-of-all-xenserver-vm-tools-components

What does the purge option remove?
https://docs.xenserver.com/en-us/citrix-hypervisor/vms/windows/vm-tools.html#what-does-the-purge-option-remove

I wasn't aware of this plugin even existing (thanks for the discovery). I'm not sure it's really maintained, but we have a clear goal of getting one officially maintained anyway.

@olivierlambert

Which one, powershell or WAC. They are kind of one in the same when you look at how WAC makes the calls to do the work, there's a button to reveal the powershell behind the button clicks so you might be able to get both at once.

But WAC draws nice graphs and gives us a GUI to get the work done. Yes I know, real Admins don't use GUI. Whatever, AI will eat our jobs anyway, just trying to stick it out another 9 years at this point.

When I get time, I'll still probably take a swing at this task for WAC, because to me this would be valuable. Big learning curve though.

Now if MS would finally add Group Policy to WAC and fix some of the DHCP and DNS features that are still in preview...

@dinhngtu Ah thank you! Idk why I couldn't find that on my own, I searched for a while.

According to the documentation, you only need xfsprogs.

I'm not aware of such problem it's like the VM was kept in limbo. You did the right thing with your commands.

Hello,

I've tried benchmarking several Windows VMs from another client and am getting expected performance:

Read: .\diskspd.exe -c1G -d10 -Su -Sw g:\data.dat Write: .\diskspd.exe -c1G -d10 -Su -Sw -w100 g:\data.dat Windows Server 2022: Read IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 1151533056 | 17571 | 109.72 | 1755.53 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 1151533056 | 17571 | 109.72 | 1755.53 Write IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 966000640 | 14740 | 92.12 | 1473.98 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 966000640 | 14740 | 92.12 | 1473.98 Windows 11 24H2 (with Eset): Read IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 1100283904 | 16789 | 104.81 | 1677.01 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 1100283904 | 16789 | 104.81 | 1677.01 Write IO thread | bytes | I/Os | MiB/s | I/O per s | file ------------------------------------------------------------------------------ 0 | 931594240 | 14215 | 88.82 | 1421.07 | g:\data.dat (1GiB) ------------------------------------------------------------------------------ total: 931594240 | 14215 | 88.82 | 1421.07

Several questions:

Are you running benchmarks on a separate virtual disk? If not, could you try again using a share created from a separate virtual disk? Could you run CrystalDiskMark locally on these VMs and see the performance? You mentioned that Eset security takes up 100% CPU during the benchmark. Could you try temporarily uninstalling Eset and testing again?

Thanks for your feedback. In the meantime, I will plan a "spike" to investigate the status of USB speed on passthrough.

@andsmith windbg is also a regular win32 app and is included in any recent windows SDK. Here: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/

You just need windbg so deselect all the other crap.

Yes, you can use XO webhooks (IIRC) to trigger something before or after a backup and send the request to a custom agent you made inside the VM.

But I'm not sure to get it, can you provide what do you want to achieve, not the technical solution (script VM backup is already a solution, not the problem you are trying to solve)

@andrewperry The release notes warn about PV guests.

@PimAarts hi!

Thx for the report - I reproduced the issue and am investigating.
Meanwhile clicking on Cancel will give you access to XOLite 🙂

2 times the charm 😆

I have no idea why it failed initially, but at least you tested the magic rollback 😎 As you can see, it's really a GREAT feature 😆

We are now live again with all the VMs - which is a relief.

The above steps from @stormi were great, the one thing I notced was the /etc/stunnel/certs directory didn't exist. I created that manually and everything else seemed happy.

I did have to mess about a bit with the fibre NICs, but I've repeated the previous renaming and that's now running fine.

Thanks very much for all the help - I appreciate it.

There is some diff, but not that much between HyperV, ESXi and Xen. However, it's big enough to require a lot of effort to get nested working correctly, one small mistake and your VM is dead.

@elvintmp75 I should have searched forum better lol.

Because it’s UEFI the file is in a different location

https://xcp-ng.org/forum/post/11939