@Pilow Yes, we monitor observability metrics for guest OSes via other means but being able to see this info via the Xen Orchestra web UI along with the other metrics in the Stats tab definitely has is benefits, especially when performing initial troublehshooting or when dealing with groups in our org that only have access to the XO interface and not the other metrics dashboards. Don't get me wrong, I definitely appreciate all of the work that you are all doing to get these tools working and can't wait to start using them in production. I just need to make sure that they work as expected.

@dthenot Thanks for getting back to me. Yes, it seems we still have time to prepare for the robot uprising I did boot from the initrd fallback before, and ChatGPT walked me through hosing that one as well. I ran the command from that doc as verbose. [image: eNSx2yx.jpeg] I ran the exact command a 2nd time as: dracut -f --verbose /boot/initrd-4.19-xen.img 4.19-xen No change. Boot to dracut with the keyboard not working. I've tried multiple kernels.

Small warning: XenTools are still a clusterfuck. On (allmost) all machines without our main environment it keeps the ip-configuration within the Realtek emulated NIC and never transfers it to the Xen device. It's fucked for long now - but doesn't matter if all VMs are getting their IPs via DHCP. Currently struggling with: Windows Server 2016 + XenTools/Management 9.4.2. At least it didn't break the whole VM by destroying the disk device drivers this round. tl;dr: Snapshot your VMs and have a documentation at hand, when upgrading the tools/drivers.

@zpvS9 said in Error installing windows PV drivers 9.0.9137: I have this error too installing the new PV driver, after uninstalling Citrix driver, reboot, execute XenClean, reboot again, it said that an unsupported device is still present. I must uninstall the device by showing hidden device in Device manager and the Windows PV driver now install. So adding this step in XenClean would be appreciated. encountered the same problem with hidden base system that i needed manually uninstall to get the tools to work.

@rustylh We really need more information to even begin. What version of XCP-ng are you using? What hardware are you using to run XCP-ng? What is the underlying storage where the VM resides on? What OS is the VM running? How are you taking snapshots, XO? How old was the snapshot? Were the guest utilities installed into the VM? You do know snapshots are not backups, right?

Helped if i read the error message.... raise RuntimeError(f"{vm_type} {vm_uuid} is {power_state}, expected Halted; refusing to fix") Shut down vm ran the fix again, and fix applied. Edit - I did have to disable secure boot on windows vms for them to boot afterwards.

So first, it's not an XO issue, which helps to diagnose it. Then, it might be a subtle XCP-ng bug indeed.

said in Unable to enable High Availability - INTERNAL_ERROR(Not_found): said in Unable to enable High Availability - INTERNAL_ERROR(Not_found): @psafont Would designating a new pool master do the same thing? I ran the above command and its had no effect Well, I tried changing the pool master and when VMHost11 was the master I was able to enable HA. Switching back to VMHost13 as the master now so will see how that goes Everything is working as expected/hoped. So for anyone reading through this and wants a TL;DR Issue was related to the pool master setting, changing the pool master to a different host and then back to the original fixed the incorrect settings allowing HA to be enabled

@lastcmaster Hi, it's a known issue that guest agent versions are not reported after migration or suspend. Other functionalities (poweroff/reboot, suspend, network change etc.) should continue to work normally.

@flakpyro said in VM start stuck on "Guest has not initialized the display (yet).": @dinhngtu said in VM start stuck on "Guest has not initialized the display (yet).": You must run secureboot-certs clear if you're updating from 1.2.0-2.4 or 1.2.0-3.1 and have previously run secureboot-certs install with the above versions installed. Should we run this before installing the update or after 1.2.0-3.2 has been installed? You should run that preferably after updating all hosts. @lukasz_s said in VM start stuck on "Guest has not initialized the display (yet).": @dinhngtu thanks for advice i've upgraded varstored and varstored-tools: rpm -qa | grep varstored varstored-1.2.0-3.2 varstored-tools-1.2.0-3.2 than i've cleared varstore with secureboot-certs clear should taht folder contain more files ? ls /usr/share/varstored/ KEK.uth PK.auth db.auth what about dbx file ? That file is not shipped with varstored nor needed for now. We're validating the final 1.2.0-3.2 and preparing our guidance for the official update.

@dinhngtu I'll try this on the next VM that exhibits the behavior as to not interrupt the end users on the completed VMs.

I have updated the drivers for the Realtek RTL812x 2.5/5/10G cards. So far they are working correctly. There are a few minor issues that Realtek needs to fix (for the next version, they say). Also the new Realtek firmware has not been added to XCP (but it's not required). The standard included 8125 driver for XCP 8.3 is not updated. To use the new driver install the new alt version of the 8125 driver. To support the 8126 install the required 8125 alt version first and then the new 8126 driver. The 8127 driver is also available for the new 10GB chips (I just got a production PCIe card for testing). The first issue I see with this card is, it is only a PCIe x1 card, so for full performance you need PCIe 4.0... There are other 8127 chips that support x2 so they will better support PCIe 3.0. Realtek will keep releasing new versions of the chips that will require updates to the drivers to function correctly. Even current versions of Linux needs updates to support the newer chips.

@rk9268vc said in Having issues installing StartOS as a VM. Cant detect a disk for it to install to.: @TeddyAstie so can i just not run this OS on xcp-ng? is there no workaround? Would this run on proxmox? you can try adding to Linux command-line (in grub) xen_emul_unplug=never to disable PV drivers, thus making udev see ATA/NVMe drivers, maybe that's enough as long it's only the installer

@Greg_E Do you mean autoupdate? It's not planned just yet, but you can update them with Group Policy using our guide here: https://docs.xcp-ng.org/guides/winpv-update/ (I think you don't need the Autoreboot setting any more, but it's worth testing). You can also use other tools like SCCM, PDQ etc. The drivers themselves are production-ready, and should not cause the domain controller issue on Server 2025. As for getting the drivers signed, the certificate did cost some time and money, but the most difficult part is dealing with Microsoft (since we're signing drivers, we needed not just a certificate but also a Microsoft hardware vendor account).

@mlcrane You're welcome! If everything started correctly, you should see this in Xen Orchestra along with VM IP: [image: 1760383624584-9df72e83-29ba-4166-b4bd-50d75f90cc87-image.png] The previous error you had "Error 0x800b0109: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider" didn't quite make sense to me since the driver package was signed by Microsoft, perhaps you were missing an important update at the time, or your VM clock was out of sync?

@bleader IIRC I just "tried again". It failed 2 times, then I looked up the logs from other console, removed the file (which shouldn't be of any importance for our instance) and retried without reboot. I copied the whole installer-log to the usb stick before finshing the install. (Could actually be a good hint or even a menu-option for those, where the install fails and won't leave it on the harddrive - e.g. evaluating hardware) [ 128.517356] ata1.00: exception Emask 0x0 SAct 0x800000 SErr 0x0 action 0x0 [ 128.517357] ata1.00: irq_stat 0x40000008 [ 128.517359] ata1.00: failed command: READ FPDMA QUEUED [ 128.517362] ata1.00: cmd 60/80:b8:10:6c:d4/00:00:02:00:00/40 tag 23 ncq dma 65536 in res 41/40:10:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F> [ 128.517363] ata1.00: status: { DRDY ERR } [ 128.517364] ata1.00: error: { UNC } [ 128.518008] ata1.00: configured for UDMA/133 [ 128.518018] sd 0:0:0:0: [sda] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 128.518020] sd 0:0:0:0: [sda] tag#23 Sense Key : Medium Error [current] [ 128.518021] sd 0:0:0:0: [sda] tag#23 Add. Sense: Unrecovered read error - auto reallocate failed [ 128.518024] sd 0:0:0:0: [sda] tag#23 CDB: Read(10) 28 00 02 d4 6c 10 00 00 80 00 [ 128.518025] print_req_error: I/O error, dev sda, sector 47475840 [ 128.518039] ata1: EH complete [ 128.581286] ata1.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x0 [ 128.581287] ata1.00: irq_stat 0x40000008 [ 128.581288] ata1.00: failed command: READ FPDMA QUEUED [ 128.581291] ata1.00: cmd 60/08:c8:80:6c:d4/00:00:02:00:00/40 tag 25 ncq dma 4096 in res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F> [ 128.581292] ata1.00: status: { DRDY ERR } [ 128.581293] ata1.00: error: { UNC } [ 128.582111] ata1.00: configured for UDMA/133 [ 128.582117] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 128.582118] sd 0:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] [ 128.582119] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed [ 128.582121] sd 0:0:0:0: [sda] tag#25 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00 [ 128.582122] print_req_error: I/O error, dev sda, sector 47475840 [ 128.582133] ata1: EH complete [ 128.629307] ata1.00: exception Emask 0x0 SAct 0x200 SErr 0x0 action 0x0 [ 128.629309] ata1.00: irq_stat 0x40000008 [ 128.629310] ata1.00: failed command: READ FPDMA QUEUED [ 128.629313] ata1.00: cmd 60/08:48:80:6c:d4/00:00:02:00:00/40 tag 9 ncq dma 4096 in res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F> [ 128.629314] ata1.00: status: { DRDY ERR } [ 128.629315] ata1.00: error: { UNC } [ 128.630068] ata1.00: configured for UDMA/133 [ 128.630074] sd 0:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 128.630076] sd 0:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] [ 128.630077] sd 0:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed [ 128.630078] sd 0:0:0:0: [sda] tag#9 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00 [ 128.630079] print_req_error: I/O error, dev sda, sector 47475840 [ 128.630092] ata1: EH complete Indeed it looks like the SSD should be replaced. 8.3 is running stable on this (and all other hosts, I upgraded) so far. It's a system at a UAS, running various student projects for several years now, coming from XenServer originally. I voluntarily maintain it. Thx for the hint!

@dinhngtu I can't say on XCP-ng side, but it's likely linked to: August patch (and following), as Microsoft changed something to the NVMe stack. e.g. https://learn.microsoft.com/en-us/answers/questions/5536733/potential-ssd-detection-bug-in-windows-11-24h2-fol Google gives a lot about it. It seems that it most likely doesn't kill NVMes but can cause trouble. We have a few PCs becoming more unstable (BSODs) or even very slow after that upgrae.

I often wondered what's the general purpose of that option. As I only have 1 - 2 socket servers, I always choose 1 socket with x cores (mostly 2 - 8, not exeeding 1 real CPU). Also for historic reasons: Sockets have been limited, but not cores. Does it generally make any difference on Xen side/backend? VM OS might handle it different due to NUMA optimizations.

@dfrizon said in How to protect a VM and Disks from accidental exclusion: @olivierlambert The idea is to block the VM and exclusion disks even by root itself, and make it possible only via command line in the console. That's why I started the post by mentioning the command... We dream of the day when MFA authentication will be required to delete a VM... How would you prevent the root account from taking action..... that is the absolute opposite permission set of root, as if there is an account with even more permissions than root. You can use permission sets and move your team who are deleting powered off VM's that are protected from accidental deletion into a group that doesn't have the permission to delete VMs, at the same time, remove their permissions from deleting items from your SR. I think that would solve your problem, and doesn't cause any logical permission issues like above.