@john.c Indeed, John, and I almost forgot that the backup network on each host was actually on a separate, isolated NIC, and not at all on the VLAN.

lspci -nn -s fra dom0 21:15 xcp-ng-test ~]# lspci -n -vvv -s 0f:00.0 0f:00.0 0300: 1002:13c0 (rev c1) (prog-if 00 [VGA controller]) Subsystem: 1002:13c0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 42 Region 0: Memory at fc10000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at fc200000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at e000 [size=256] Region 5: Memory at fc700000 (32-bit, non-prefetchable) [size=512K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 16GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=4 Masked- Vector table: BAR=5 offset=00042000 PBA: BAR=5 offset=00043000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000100, Page Request Allocation: 00000000 Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [410 v1] #26 Capabilities: [450 v1] #27 Kernel driver in use: pciback

Wonderful !

@hrvojer My posts above will help you get out of a technical debt trap that you are currently pitfall-trap trapped in! If you do this, it will drastically help to reduce your operational costs. Maintaining an old legacy stack like this—where every component from the hypervisor to the database engine is completely End of Life (EOL)—is scandalously expensive. It sucks up valuable engineering hours in firefighting, forces you into hyper-expensive custom support agreements, and leaves you exposed to massive downtime risks—which is a nightmare scenario, especially if your other organizational costs are going up at the same time. I highly recommend framing this to your leadership using these exact terms: The Pitfall Trap: Explain that the infrastructure team is currently stuck at the bottom of a pitfall trap of compounding technical debt due to a historical lack of a system-wide overhaul. The Rescue Plan: Make it clear that trying to find deprecated, 2018-era VM tools is just a short-term sticking plaster fix that buries the team deeper. What you need is a definitive rescue plan—a clean overhaul that decouples your data layer, migrates you to a modern OS/database (or an open-source alternative), and permanently lifts you out of the trap. As well as other dependencies - systems that communicate with that one, but won’t be able to without upgrade or migration, even with configuration tweaks. The Linux Performance & Driver Advantage: If you choose to migrate your database server to Linux (running PostgreSQL or MariaDB), you gain massive performance, reliability, and security boosts. Unlike Windows, you will completely bypass the fragile legacy Citrix xenvbd.sys driver ecosystem. On Linux, the paravirtualized storage and network drivers are compiled directly into the Linux kernel. The only thing you need to install separately is the modern, official XCP-ng guest agent—which Vates has beautifully rewritten as an ultra-secure, memory-safe Rust-language agent. This gives you a lean, stable database foundation that can handle heavy memory allocations and high I/O workloads without ever breaking a sweat.

@bogdantomasciuc Glad to hear. I was hoping it was just something missed or miss typed. If you can replicate the issue let me know. Its possible i might have missed that little nuance or something.

I finally got access to a machine with Intel Arc machine. There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7). For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine. Ran a clpeak bench, and performance looks normal for this class of GPU. Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0). in-VM inxi -CGM Machine: Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026 CPU: Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB) Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594 Graphics: Device-1: driver: bochs-drm v: N/A Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48 API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits) API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandr Raw numbers Backend: Vulkan Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) Driver version : 26.2.4 API version : 1.4.354 VRAM : 16048 MB Single-precision compute float : 9.47 TFLOPS float2 : 9.48 TFLOPS float4 : 9.52 TFLOPS Half-precision compute fp16xfp16+fp16 half : 18.3 TFLOPS half2 : 18.8 TFLOPS half4 : 19.1 TFLOPS Double-precision compute double : 650 GFLOPS double2 : 647 GFLOPS double4 : 640 GFLOPS Mixed-precision compute fp16xfp16+fp32 mp : 9.31 TFLOPS mp2 : 9.32 TFLOPS mp4 : 9.42 TFLOPS BF16 compute bf16xbf16+fp32 bf16 : 9.17 TFLOPS bf16_2 : 9.30 TFLOPS bf16_4 : 9.28 TFLOPS Integer compute int32 int : 1.81 TOPS int2 : 1.88 TOPS int4 : 1.82 TOPS INT8 dot-product compute int8_dp : 28.0 TOPS int8_dp2 : 26.3 TOPS int8_dp4 : 27.1 TOPS Cooperative matrix fp16 : 67.3 TFLOPS fp16 f16acc : 68.9 TFLOPS bf16 : 68.9 TFLOPS int8 : 141 TOPS Global memory bandwidth float : 196 GB/s float2 : 206 GB/s float4 : 212 GB/s Local memory bandwidth float : 2.55 TB/s float2 : 3.65 TB/s float4 : 4.23 TB/s Image memory bandwidth float4 : 213 GB/s Transfer bandwidth h2d : 4.81 GB/s d2h : 5.48 GB/s Kernel launch latency dispatch : 48.7 us roundtrip : 113 us Real performance is much higher than the "displayed" Speed 2.5GT/s, Width x1.

Update - changes are live in the dev branch now. So figured out the update / switching branches for the proxy by connecting the proxy to my xoa free account. That triggered the available update to next version. From there I found the command that needs to be ran on the proxy to switch back and forth... [image: Screenshot_20261009_143546.png] Latest branch [image: Screenshot_20261009_144414.png] stable branch [image: Screenshot_20261008_202716.png] Backup job ran sucessful. 2026-10-09T17_48_25.447Z - backup NG.txt After the proxy has been deployed... The proxy has been registered with your Xen Orchestra instance. You can manage it from the Xen Orchestra web interface. Next: register the proxy's updater and move it to the 'latest' channel. On the pool master (IP), set a password for the proxy's 'xoa' user (choose your own) and reboot the proxy VM: xe vm-param-set uuid=UUID xenstore-data:vm-data/system-account-xoa-password='<your password>' xe vm-reboot uuid=UUID SSH to the proxy as 'xoa' (IP) and run: sudo xoa-updater register - This registers it with your vates account. sudo xoa-updater configure-channel xo-proxy-appliance-latest - set to latest or stable sudo xoa-updater upgrade - This can be ran via the UI or this command. Both need to be done twice to do the upgrade and/or downgrade. register needs a free xen-orchestra.com account; see the README. And host connected via proxy. [image: Screenshot_20261009_150011.png]

So, for everyone who wants to install NUT as a service follow these steps. Modifying the systemd files can make you system unbootable! It should not happen, but there is chance since we are also modifying some "natively shipped files". So an update can break your working configuration again and that is not the fault of XCP-NG!!! As prerequisite it requires the installation and testing of this step-by-step guide https://xcp-ng.org/forum/post/104253. Make sure everything runs before proceeding to the following steps! Now: ################################# Remove "nut-driver.target" dependency ################################# Remove "nut-driver.target" dependency from "nut-server.service" by calling nano /lib/systemd/system/nut-server.service and remove nut-driver.target from routine. I prefer to duplicate and comment to keep the modified rows in the original code. It then should look like this. DO NOT COPY AND PASTE, read, compare and modify carefully! [Unit] Description=Network UPS Tools - power devices information server #After=local-fs.target network.target nut-driver.target After=local-fs.target network.target # We don't Require drivers to be successfully started! This would be # a change of behavior compared to init SysV, and could prevent from # accessing successfully started, at least to audit a system. #Wants=nut-driver.target Wants= # The `upsd` is a networked service (even if bound to a `localhost`) # so it requires that the OS has some notion of networking already. # Extending the unit does not require *this* file to be edited, you # can instead drop in an additional piece of configuration, e.g. add # a `/etc/systemd/system/nut-server.service.d/network.conf` with: # [Unit] # Requires=network-online.target # After=network-online.target Requires=network.target Before=nut-monitor.service PartOf=nut.target [Service] EnvironmentFile=-/etc/ups/nut.conf SyslogIdentifier=%N # Note: foreground mode by default skips writing a PID file (and # needs Type=simple); can use "-FF" here to create one anyway: ExecStart=/usr/sbin/upsd -F ExecReload=/usr/sbin/upsd -c reload -P $MAINPID [Install] WantedBy=nut.target ################################# Add Service for ups-driver ################################# nano /etc/systemd/system/ups-driver.service [Unit] Description=NUT UPS Driver After=network-online.target Wants=network-online.target Before=nut-server.service [Service] Type=oneshot ExecStart=/usr/sbin/upsdrvctl start ExecStop=/usr/sbin/upsdrvctl stop RemainAfterExit=yes [Install] WantedBy=multi-user.target ################################### Add Requirement for nut-server ################################# mkdir -p /etc/systemd/system/nut-server.service.d nano /etc/systemd/system/nut-server.service.d/ups-driver.conf [Unit] Requires=ups-driver.service After=ups-driver.service ################################### Add Requirement for nut monitor ################################# mkdir -p /etc/systemd/system/nut-monitor.service.d nano /etc/systemd/system/nut-monitor.service.d/nut-server.conf [Unit] Requires=nut-server.service After=nut-server.service ################################### Enable services ################################### systemctl daemon-reload systemctl enable ups-driver.service systemctl enable nut.target systemctl enable nut-server.service systemctl enable nut-monitor.service systemctl start nut-monitor.service ################################### Analyze & debugging commands ################################### systemctl list-dependencies nut-monitor.service systemd-analyze critical-chain nut-monitor.service systemctl show nut-monitor.service -p Requires -p Wants -p After systemctl show nut-server.service -p Requires -p Wants -p After

@herhin2017 said: @john.c Your absolute wright, i do my best and train about 25 young engineers per year on linux (we use debian). I also see more and more startups or small companies building their infrastructure on linux. Best greetings from austria It may be worth getting in line and noting the mention of EFI based servers for official Vates support with XCP-ng version 9.0 and above, in government reports. That way when the school’s hardware is refreshed it can be ensured that your provided with EFI capable servers, in time for XCP-ng version 8.3 EOL. I personally are already ready for XCP-ng version 9.0 due to my servers being Dell PowerEdge R620 for XCP-ng hosts. Along with Debian version 13.6 on the VMs. While using a Dell Precision 3590 to manage those systems. The “wright” makes your above now sound like a wheel wright, and its profession instead of “right” in for when correct or agreeing with someone or something.

@johnnezero You can use the docs directly in /rest/v0/docs. You just have to connect and fill in the uuid you need. It will work

@MathieuRA We reverted back to 6.8.2 for now. We can wait for patch, otherwise, I have opened a tunnel #38081.

@john.c I've put this on the feedback portal as a feature request, so it doesn't get buried in a thread that started out about pool masters: https://feedback.vates.tech/posts/121/publish-build-date-and-package-manifest-sbom-with-the-xoa-xva-download . As far as I know votes there help the team prioritise, so if you and @Forza want it, that's the place to say so. There's an open PR adding SBOM generation (https://github.com/vatesfr/xen-orchestra/pull/10368), but going by the files it touches it covers XO's own JavaScript dependencies, not the Debian packages in the appliance. I've only looked at the file list, not the code, so happy to be corrected. How often the downloadable XVA gets rebuilt, I still don't know: @Danp said back in 2024 that it isn't rebuilt for every XOA release (https://xcp-ng.org/forum/post/81220), which would fit the 2025.12 tag, but I'd rather not guess beyond that.

@bug-meister Yeah, I would say the "mechanism" is deprecated too. Getting this mechanism to work, either by using ansible-pull or by some clever scripting during the first boot, requires Ansible to be preinstalled on your VM templates. That just makes your VMs messy and bloated with additional stuff you only use once. Also, managing Ansible versions becomes more complex. If Ansible is installed from distro repositories, you will have all different versions of Ansible across different distros. If you use pip to install Ansible, it can mess up your system if Ansible dependencies are installed on a system level. You can use Python virtual environments to prevent that but requires additional managing. It's much better and much more practical to have Ansible installed and used from a single place - Control Node. That way you have one place, one OS, one Ansible version to think about. You already have a machine you are accessing your VMs from? Use that as a control node... unless it is Windows?

@dinhngtu i've manage to fixed it. Your build adds some extra entries in /var/lib/xcp/state.db that aren't in the version from testing, that broke things when i downgraded the packages. ipv4_dns VIF.ipv6_dns

@poddingue said: Thanks for posting the fix. The same thing came up in an older thread, https://xcp-ng.org/forum/topic/9957, where disabling IPv6 also made xoa check go green. @HamiltonWDS explained a likely reason in https://xcp-ng.org/forum/post/87831: Node tries the addresses with a short timeout and can end up on the IPv6 one. Your curl -6 test helps a lot here, because it shows IPv6 doesn't connect at all on that network, so it's more than a slow path. Turning it off in XOA seems reasonable if you don't use IPv6 there; if you do, I'd guess the router side is where it really needs fixing, though I could be wrong (still haven't migrated to IPv6 myself ). Curious whether it sorts out the Cloud Backup too. XOA is using our management network where ipv6 is not needed and not enabled. Looks like our cloud backup finally ran for the first time in while too.[image: cloudbackup.jpg]

Thanks for the help; this error has been resolved following the installation. However, we have a new issue regarding CBT. We are experiencing recurring CBT failures with the following configuration: Veeam Backup & Replication: 13.1.1.18 Hypervisor: XCP-ng 8.3, three hosts Identical packages on all hosts: sm-3.2.12-25.1.xcpng8.3 blktap-3.55.5-11.1.xcpng8.3 xapi-core-26.1.19-1.1.xcpng8.3 These packages were installed on October 3, 2026, before the failures occurred. Affected VM: NMNM-V-S-L007 Source disk UUID: 2f546ede-efcb-4ccf-8c23-2ea5a5e0d01f Storage: LVM over iSCSI, SR name 70TB_HDD SR UUID: 26d9572f-01c1-9cfd-79de-ca96eab601cf The backup job finishes with a Warning. The failing operation is VDI.list_changed_blocks, which returns: SR_BACKEND_FAILURE_460 Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated] The logs show repeated failures on October 12 and 13, 2026. During cleanup of an older snapshot, CBT metadata is merged directly into the live disk despite the existence of a newer snapshot. This suggests a possible inconsistency in the CBT parent/child relationships, although the root cause has not been confirmed. The reference snapshot used on October 13 was deleted only after the CBT failures, so its subsequent absence does not explain those failures. Attached logs: cbt-old-vdi-history.txt cbt-source-disk-history.txt cbt-chain-cleanup.txt Please investigate CBT chain handling during snapshot creation and deletion, verify the installed Xen plug-in version, and advise whether a relevant hotfix or a supported procedure to re-establish CBT tracking is available. Thank you.cbt-old-vdi-history.txt cbt-chain-cleanup.txt [cbt-source-disk-history.txt.zip](Invalid file type .zip. Allowed types are: .png, .jpg, .bmp, .txt, .diff, .patch, .jpeg)

Glad the ticket sorted it, @jacob.becker, and thanks for coming back to say so. I don't know whether a patch for the stable channel is planned before 6.9 moves there. If someone else on stable lands here with the same Body Timeout Error, opening a ticket like you did seems to be the quickest route for now.

@acebmxer I would have said the logs depend on how you deployed XO. Basically it would be an error printed when you ran yarn start. But it depends if XO is started with systemd or something else. But as you found yourself the solution... this is not needed anymore for now. I'll see if we can report an helpful message in XO. Something like "Cannot reach docker.sock. As you're running a RedHat like distro, ensure SELinux rules let sshd reach it. See this link for more... Etc."