@JamesG, I'm the one who marked it solved, so here's what I meant. What Teddy showed in https://xcp-ng.org/forum/post/109066 is that there's one thing to do on the XCP-ng side for the B50: turn HVM PIRQ off for that VM (xe vm-param-set uuid=<vm-uuid> platform:hvm-pirq=false, then reboot the VM), as described on the docs page he linked: https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7. With that, his benchmark came out normal for the card, and the Gen1 x1 you see inside the guest is something QEMU makes up, not the real link (his earlier post: https://xcp-ng.org/forum/post/106479). It doesn't mean every problem is on the guest side though: @pandusen tried the same setting (https://xcp-ng.org/forum/post/109084) and still had driver crashes on headless distros, and I don't know why. If you try it on your B50 and post what you see, that would help tell the two cases apart.

@john.c Indeed, John, and I almost forgot that the backup network on each host was actually on a separate, isolated NIC, and not at all on the VLAN.

lspci -nn -s fra dom0 21:15 xcp-ng-test ~]# lspci -n -vvv -s 0f:00.0 0f:00.0 0300: 1002:13c0 (rev c1) (prog-if 00 [VGA controller]) Subsystem: 1002:13c0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 42 Region 0: Memory at fc10000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at fc200000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at e000 [size=256] Region 5: Memory at fc700000 (32-bit, non-prefetchable) [size=512K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 16GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=4 Masked- Vector table: BAR=5 offset=00042000 PBA: BAR=5 offset=00043000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000100, Page Request Allocation: 00000000 Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [410 v1] #26 Capabilities: [450 v1] #27 Kernel driver in use: pciback

@hrvojer My posts above will help you get out of a technical debt trap that you are currently pitfall-trap trapped in! If you do this, it will drastically help to reduce your operational costs. Maintaining an old legacy stack like this—where every component from the hypervisor to the database engine is completely End of Life (EOL)—is scandalously expensive. It sucks up valuable engineering hours in firefighting, forces you into hyper-expensive custom support agreements, and leaves you exposed to massive downtime risks—which is a nightmare scenario, especially if your other organizational costs are going up at the same time. I highly recommend framing this to your leadership using these exact terms: The Pitfall Trap: Explain that the infrastructure team is currently stuck at the bottom of a pitfall trap of compounding technical debt due to a historical lack of a system-wide overhaul. The Rescue Plan: Make it clear that trying to find deprecated, 2018-era VM tools is just a short-term sticking plaster fix that buries the team deeper. What you need is a definitive rescue plan—a clean overhaul that decouples your data layer, migrates you to a modern OS/database (or an open-source alternative), and permanently lifts you out of the trap. As well as other dependencies - systems that communicate with that one, but won’t be able to without upgrade or migration, even with configuration tweaks. The Linux Performance & Driver Advantage: If you choose to migrate your database server to Linux (running PostgreSQL or MariaDB), you gain massive performance, reliability, and security boosts. Unlike Windows, you will completely bypass the fragile legacy Citrix xenvbd.sys driver ecosystem. On Linux, the paravirtualized storage and network drivers are compiled directly into the Linux kernel. The only thing you need to install separately is the modern, official XCP-ng guest agent—which Vates has beautifully rewritten as an ultra-secure, memory-safe Rust-language agent. This gives you a lean, stable database foundation that can handle heavy memory allocations and high I/O workloads without ever breaking a sweat.

OK, marking this solved... It was the phantom reboots causing issues because the tools were not fully installed, only thing I can't understand is why only 2 out of 3 Windows servers would quit. Any way, got the Xen Tools updated and a couple reboots later, everything is working again. Also I can now report that a bug that was fixed maybe a year ago is confirmed to work in my system. The bug was backing up a Windows VM that had a lot of empty space in the drive (over 250GB free) and turning on compression. It would time out counting zeros because it thought the process wasn't doing work. Both of these VMs are now compressing with zstd, one of them is substantially smaller now, the other only shrunk about 20GB. Because of the way Windows uses the drive, these are all full backups. The differential never seemed to be small because Windows does a lot of changes on the drive (there is a post about this somewhere in here from Olivier).

@bogdantomasciuc Glad to hear. I was hoping it was just something missed or miss typed. If you can replicate the issue let me know. Its possible i might have missed that little nuance or something.

The UEFI and possibly SecureBoot requirements on a lot of stuff and going to turn a lot of old systems into garbage. Also going to be an issue when more current stuff gets upgrade from 8.3 in BIOS mode to requiring UEFI for version 9. That's the boat I'm in for production. Alma and openSuse may be a few of the last holdouts allowing v2 and BIOS boot.

So, for everyone who wants to install NUT as a service follow these steps. Modifying the systemd files can make you system unbootable! It should not happen, but there is chance since we are also modifying some "natively shipped files". So an update can break your working configuration again and that is not the fault of XCP-NG!!! As prerequisite it requires the installation and testing of this step-by-step guide https://xcp-ng.org/forum/post/104253. Make sure everything runs before proceeding to the following steps! Now: ################################# Remove "nut-driver.target" dependency ################################# Remove "nut-driver.target" dependency from "nut-server.service" by calling nano /lib/systemd/system/nut-server.service and remove nut-driver.target from routine. I prefer to duplicate and comment to keep the modified rows in the original code. It then should look like this. DO NOT COPY AND PASTE, read, compare and modify carefully! [Unit] Description=Network UPS Tools - power devices information server #After=local-fs.target network.target nut-driver.target After=local-fs.target network.target # We don't Require drivers to be successfully started! This would be # a change of behavior compared to init SysV, and could prevent from # accessing successfully started, at least to audit a system. #Wants=nut-driver.target Wants= # The `upsd` is a networked service (even if bound to a `localhost`) # so it requires that the OS has some notion of networking already. # Extending the unit does not require *this* file to be edited, you # can instead drop in an additional piece of configuration, e.g. add # a `/etc/systemd/system/nut-server.service.d/network.conf` with: # [Unit] # Requires=network-online.target # After=network-online.target Requires=network.target Before=nut-monitor.service PartOf=nut.target [Service] EnvironmentFile=-/etc/ups/nut.conf SyslogIdentifier=%N # Note: foreground mode by default skips writing a PID file (and # needs Type=simple); can use "-FF" here to create one anyway: ExecStart=/usr/sbin/upsd -F ExecReload=/usr/sbin/upsd -c reload -P $MAINPID [Install] WantedBy=nut.target ################################# Add Service for ups-driver ################################# nano /etc/systemd/system/ups-driver.service [Unit] Description=NUT UPS Driver After=network-online.target Wants=network-online.target Before=nut-server.service [Service] Type=oneshot ExecStart=/usr/sbin/upsdrvctl start ExecStop=/usr/sbin/upsdrvctl stop RemainAfterExit=yes [Install] WantedBy=multi-user.target ################################### Add Requirement for nut-server ################################# mkdir -p /etc/systemd/system/nut-server.service.d nano /etc/systemd/system/nut-server.service.d/ups-driver.conf [Unit] Requires=ups-driver.service After=ups-driver.service ################################### Add Requirement for nut monitor ################################# mkdir -p /etc/systemd/system/nut-monitor.service.d nano /etc/systemd/system/nut-monitor.service.d/nut-server.conf [Unit] Requires=nut-server.service After=nut-server.service ################################### Enable services ################################### systemctl daemon-reload systemctl enable ups-driver.service systemctl enable nut.target systemctl enable nut-server.service systemctl enable nut-monitor.service systemctl start nut-monitor.service ################################### Analyze & debugging commands ################################### systemctl list-dependencies nut-monitor.service systemd-analyze critical-chain nut-monitor.service systemctl show nut-monitor.service -p Requires -p Wants -p After systemctl show nut-server.service -p Requires -p Wants -p After

@poddingue Thank you!

@johnnezero You can use the docs directly in /rest/v0/docs. You just have to connect and fill in the uuid you need. It will work

@julienXOvates said: Just discussed this with @nathanael-h; I would strongly recommend using the load balancer plugin in XO and create some VM-to-host affinity rules, so that your CP VMs are running on different hosts ! So when the RPU comes you would have minimum impact. Yeah the Load Balancer plugin is an option, though not in my plan. I could certainly get it by using XOCE for free though. I wonder if Host Affinity might work as well...

@MathieuRA We reverted back to 6.8.2 for now. We can wait for patch, otherwise, I have opened a tunnel #38081.

@dinhngtu Thanks for fixing the note. The IP loss did happen on 9.6.0. On the first boot after I set has-vendor-device=false, the guest still reported tools 9.6.0 build 19 (xenbus 9.1.13.23, xenvif 9.1.15.8, xennet 9.1.8.6), and both static IPs were already gone: the main NIC was on DHCP and the second NIC had a 169.254 address. XenClean and the 9.2.385 install came after that.

@john.c I've put this on the feedback portal as a feature request, so it doesn't get buried in a thread that started out about pool masters: https://feedback.vates.tech/posts/121/publish-build-date-and-package-manifest-sbom-with-the-xoa-xva-download . As far as I know votes there help the team prioritise, so if you and @Forza want it, that's the place to say so. There's an open PR adding SBOM generation (https://github.com/vatesfr/xen-orchestra/pull/10368), but going by the files it touches it covers XO's own JavaScript dependencies, not the Debian packages in the appliance. I've only looked at the file list, not the code, so happy to be corrected. How often the downloadable XVA gets rebuilt, I still don't know: @Danp said back in 2024 that it isn't rebuilt for every XOA release (https://xcp-ng.org/forum/post/81220), which would fit the 2025.12 tag, but I'd rather not guess beyond that.

@bug-meister Yeah, I would say the "mechanism" is deprecated too. Getting this mechanism to work, either by using ansible-pull or by some clever scripting during the first boot, requires Ansible to be preinstalled on your VM templates. That just makes your VMs messy and bloated with additional stuff you only use once. Also, managing Ansible versions becomes more complex. If Ansible is installed from distro repositories, you will have all different versions of Ansible across different distros. If you use pip to install Ansible, it can mess up your system if Ansible dependencies are installed on a system level. You can use Python virtual environments to prevent that but requires additional managing. It's much better and much more practical to have Ansible installed and used from a single place - Control Node. That way you have one place, one OS, one Ansible version to think about. You already have a machine you are accessing your VMs from? Use that as a control node... unless it is Windows?

@dinhngtu i've manage to fixed it. Your build adds some extra entries in /var/lib/xcp/state.db that aren't in the version from testing, that broke things when i downgraded the packages. ipv4_dns VIF.ipv6_dns

@poddingue said: Thanks for posting the fix. The same thing came up in an older thread, https://xcp-ng.org/forum/topic/9957, where disabling IPv6 also made xoa check go green. @HamiltonWDS explained a likely reason in https://xcp-ng.org/forum/post/87831: Node tries the addresses with a short timeout and can end up on the IPv6 one. Your curl -6 test helps a lot here, because it shows IPv6 doesn't connect at all on that network, so it's more than a slow path. Turning it off in XOA seems reasonable if you don't use IPv6 there; if you do, I'd guess the router side is where it really needs fixing, though I could be wrong (still haven't migrated to IPv6 myself ). Curious whether it sorts out the Cloud Backup too. XOA is using our management network where ipv6 is not needed and not enabled. Looks like our cloud backup finally ran for the first time in while too.[image: cloudbackup.jpg]

Glad it's working, and thanks for posting the script, the next NUT user will want it (I hope I'll be able to experiment with NUT in the coming weeks, so I could very well be "the next NUT user"). I tried the PBD part on a lab host running 8.3: xe pbd-list never prints a storage-type line, so that grep finds nothing and step 2 quietly unplugs no PBDs. The 2>/dev/null hides it, and --force isn't an option pbd-unplug knows about. My wild guess is that ejecting the CDs is what fixed it for you. If you want the unplug to really happen, going through the SR type works on my side, though I only tested it against an ISO SR, not SMB: H=$(xe host-list hostname=$(hostname) --minimal) for t in nfs smb iso lvmoiscsi; do for sr in $(xe sr-list type=$t --minimal | tr , ' '); do for pbd in $(xe pbd-list sr-uuid=$sr host-uuid=$H currently-attached=true --minimal | tr , ' '); do xe pbd-unplug uuid=$pbd done done done Block iSCSI SRs show up as lvmoiscsi, which is why it's not iscsi in the list. @Forza's point about hard mounts may still bite if the share is already gone when the unplug runs; I don't know enough about the SMB side to say.