@planedrop Thanks for the information! Back to the more general idea of the wide VMs, I think it was originally more of an efficiency issue. Our Support team noticed high CPU usage, but the pCPU and overall host usage was very low.

Turns out we had stacked multiple, heavy-utilized, wide VMs on the same hosts. After looking at the stats, there so was so much co-stop that they were wasting so much time on trying to co-schedule the vCPUs. After spreading out the wide VMs we actually saw the hosts overall consume more CPU and the performance issues went away.

With us getting fresh start on a new hypervisor, instilling a desire for right-sizing VMs and scaling out versus up will probably be the way to go.

Thanks again for all your help!

@dinhngtu I'll try this on the next VM that exhibits the behavior as to not interrupt the end users on the completed VMs.

@dinhngtu Below is what I can find on the latest VM where this occurred. I saw a Windows report error that said it might have info, but there is nothing there.

Faulting application name: xen-guest-agent.exe, version: 9.0.9137.2, time stamp: 0x68e7c2e8
Faulting module name: xen-guest-agent.exe, version: 9.0.9137.2, time stamp: 0x68e7c2e8
Exception code: 0xc0000409
Fault offset: 0x00000000000a2c39
Faulting process id: 0xa20
Faulting application start time: 0x01dc439b0e299e74
Faulting application path: C:\Program Files\XCP-ng\Windows PV Drivers\XenGuestAgent\xen-guest-agent.exe
Faulting module path: C:\Program Files\XCP-ng\Windows PV Drivers\XenGuestAgent\xen-guest-agent.exe
Report Id: 145e6fdf-9c31-408c-a704-f09b923f957d
Faulting package full name: 
Faulting package-relative application ID: 

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: xen-guest-agent.exe
P2: 9.0.9137.2
P3: 68e7c2e8
P4: xen-guest-agent.exe
P5: 9.0.9137.2
P6: 68e7c2e8
P7: 00000000000a2c39
P8: c0000409
P9: 0000000000000007
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_xen-guest-agent._3e70918689a16355b26c3269beb18dcb31a29bf_48acc870_17b92d64

Analysis symbol: 
Rechecking for solution: 0
Report Id: 145e6fdf-9c31-408c-a704-f09b923f957d
Report Status: 4
Hashed bucket: 

@dinhngtu On the Windows Event Viewer, I didn't see anything out of the ordinary. Unfortunately, Ansible posts a lot of info in there so it may be getting drowned out. If I get a chance to test with another migrated VM, I will check again.

On some of our VMs that we migrated over from ESXi, we have found that the new Xen Guest Agent (the one included in the new xcpng-winpv ISO) sometimes will not report the IP address of the Guest after a reboot.

So far this has happened on several of our Windows Server 2016 and 2019 VMs. When the reboot occurs, the IP address will display briefly after booting back up, but once the Management agent is detected, the IP will go back to saying there is no IPv4 address.

If I log into the Guest OS and restart the Xen Guest Agent (Rust-based) Windows Service, the IP address will show up immediately in XO. If I set the Xen Guest Agent Windows Service to Startup Type Automatic (Delayed Start) and reboot, the IP address will eventually be reported once the Service starts.

This isn't a major issue, since we can work around it by setting the Xen Guest Agent Service to Delayed Start in Windows, but we thought we would provide some feedback in case others are experiencing it too.

Note: We updated XO last week just before starting to install the new drivers/agents and our Pool is up to date according to the Pool Master.

@dinhngtu Awesome and thank you so much!

When the new version is built, will we need to download it and install from Github or is there another update mechanism?

Good Afternoon Everyone!

I am really excited for the release of the signed drivers and the guest agent for XCP-ng, so I installed it on one of our servers to test it. Everything went smoothly, but I noticed one thing after the install.

On the test VM, it reports the version of Windows using the underlying version as Windows 10 instead of the server version such as Windows Server 2016 Datacenter. Below is the example. While this isn't a huge deal, since we can manually tag the VM, it is nice that with the XenServer agent we don't have to tag them to make them searchable/filterable on "Windows Server nnnn" in XO. Since we also have Windows 10 and 11 VMs in our environment, the Server 2016 test VM also shows up as Windows 10 in the list.

XCP-ng Agent Windows Server 2016

Xen Server Agent Windows Server 2016

This is what the XenServer agent reports for a Windows 10 install.

As stated above, there is an easy albeit tedious workaround, so this is more feedback than an issue. I also appreciate how the XCP-ng agent does not include all the C:\ drive info that the XenServer agent does. That makes it look cleaner in my opinion.

Another thing I noticed is that if I rename the ISO in the XO UI, it will keep that name in the UI, but I can't seem to use it mount to any VMs using the Console or Disks tab. A rescan of the SR didn't seem to make a difference either.

This is still happening for us on our Community Edition instance that I updated today, but the odd thing I noticed is that it doesn't happen immediately. When I upload an iso file, it keeps the name for a while, but when I return the next day, it is renamed to the <uuid>.img file.

Maybe there's some maintenance job that runs that does the renaming?

@DustinB The vendor's documentation states official support for RHEL/Rocky/Alma/Oracle Linux 9.0+ systems.

@plaidypus said in Rocky Linux 8 (RHEL Potential for Performance Issue:

Is this still an issue, and if so, would the recommendation be to use EXT4 for both the OS and the data drive?

My apologies, I may have added too much context and information for this. I agree that this is not an issue for XO, but that was also not the original question. If I use Rocky Linux 8 and follow the XCP-ng recommendation in the documentation to use ext4, does this apply to both the root filesystem and also our secondary disk where more of the intensive IO will be?

@DustinB We currently pay for support on this product, and it supposedly supports Rocky Linux 9, but in practice it will not deploy unless we trick it into thinking it is RHEL 9 by swapping out the /etc/redhat-release file. While this workaround did work, our team was not confident in keeping the workaround in place for the automation processes.

Our team wants to get it working with Rocky 9, but upgrading the management software is out of our allowed scope for now (airgapping, InfoSec approvals) and working with their support is slow and our timeline to move these systems is short. Since we know the processes work with Rocky 8, we were instructed to go with that.

With that in mind, I think we are going to set up a new template with ext4 as the filesystem.