@olivierlambert said: Hello @andrija.panic ! I'm aware and I followed your work and announcements It's great! On our side, we rely on our solution since the end of September: https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md#690-2026-09-30 : "[V2V] Migrations no longer need the VDDK library". We couldn't wait much more and we had to act very fast to remove this dependency. Hi @olivierlambert, thanks for the insight on vectura! Completely understand the need to move fast and gain full vertical integration over the V2V pipeline—it’s a great move for standard migrations. My suggestion to consider a dual-path approach with an OpenVDDK fallback is less about day-to-day operations and more about strategic, ecosystem-level insurance. If we look at how the Xen Project / Xen Foundation handles governance, it provides a perfect architectural blueprint for this problem: Shared Governance vs. Vendor Control: For OpenVDDK to be a reliable fallback against Broadcom's moving goalposts, it ultimately needs a single, unified organizational structure (similar to a Linux Foundation sub-project). If Vates, Red Hat, and the CloudStack/Proxmox communities pool resources into a shared, neutrally governed utility, the burden of reverse-engineering VMware's disk-format shifts is distributed across the entire industry. Isolating the Core Stack: Just like XCP-ng handles alternative storage plugins or drivers, OpenVDDK wouldn't need to be baked into your core engine. It could live as a completely decoupled, optional legacy transport layer. If vectura hits an unforeseen edge case, a specific CBT lock, or an old ESXi topology (like the legacy versions dropped in 6.9.0), the system has an automated valve to fall back on. Focusing R&D Resources: By helping steer a shared industry standard for the low-level data extraction layer, Vates secures a massive legal and engineering buffer against VMware licensing and structural changes, allowing your internal teams to focus 100% of their energy on making Xen Orchestra the best frontend destination. It prevents Xen Orchestra from ever being caught by surprise, while still keeping your in-house binary as the primary choice.

@fred974 That error code 0xc0000225, claims that the disk either doesn't exist or doesn't contain the needed files to boot. if you can boot into safe mode, try running sfc /scannow and chkdsk C: /f /r

Thanks @jr-m4, reporting the small ones is useful too. From what I can see in the code, the VM view of the restore page has that refresh button and the Metadata view only has a link back, so I think your screenshots show a real gap and not a glitch on your side. I honestly don't know whether XO 5 still takes this kind of small UI fix now that most of the work goes into XO 6, so I'll leave that to @Team-XO-Frontend to decide.

@zorro I tried to reproduce this on a small lab host with plain xe commands on XCP-ng 8.3, not with XO and Veeam themselves. I think your read is right: both tools end up on the same CBT chain. When a newer snapshot sitting between Veeam's reference and the live disk has had its data purged and is then deleted, the changed-block query from the older snapshot fails with exactly your SR_BACKEND_FAILURE_460 ... Source and target VDI are unrelated, and if I read XO's code right, that purge-then-delete is what the "Purge snapshot data" option does on each run. When that middle snapshot is a normal one, deleting it doesn't break anything in my test. So turning purge off might well avoid it (I think XO then keeps a full snapshot on the host instead), but please treat that as a lead rather than an answer. @Team-Storage will know whether that's expected; until then @acebmxer's one-tool-per-VM advice is the safe one.

Thanks @cocoon, and I owe a correction: when I said in https://xcp-ng.org/forum/post/104865 that 6.4 would make these alerts disappear for good, I overstated it. If I'm reading the code right this time, the XO server sends a ping every 30 seconds stamped with its own clock (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/rest-api/src/events/event.class.mts), and the browser compares that stamp with its own clock and shows the banner once the gap passes 40 seconds (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/web-core/lib/packages/remote-resource/sse.store.ts). So https://github.com/vatesfr/xen-orchestra/pull/9681 widened the margin from 32 to 40 seconds, but a browser whose clock runs ahead of the XO server by more than about 10 seconds would still see it. That might be why @GregBinSD saw it on the Windows PC and not on the tablet, though that's only a guess on my part. Comparing two different clocks looks fragile to me, but @Team-XO-Frontend will know far better than I do whether there's a reason it's done that way.

@irtaza9 What do you see if you run top or xtop, or "xe task-list" when you start the process?

For some reason, we have the same issue (missing disks / snapshot-of) on our environment. (XO from source commit a5077) This is a major issue for us. We use Terraform to manage our VMs, and the missing VDIs are causing Terraform to fail. As a result, we are currently unable to manage our VMs using Terraform. ➜ dacsvm-db02 git:(main) tofu plan -var-file="dacsvm-db02.tfvars" module.vm.xenorchestra_vm.vm: Refreshing state... [id=e36fcd8a-b88c-42f7-6d61-2aa639f63b4a] Planning failed. OpenTofu encountered an error while generating this plan. ╷ │ Error: Could not find client.VDI with query: {VDIId:cad7d739-35c2-4585-9991-4c2f3fc6531c SrId: NameLabel: NameDescription: Size:0 VBDs:[] Parent: PoolId: Tags:[]} │ │ with module.vm.xenorchestra_vm.vm, │ on ../modules/vm/main.tf line 16, in resource "xenorchestra_vm" "vm": │ 16: resource "xenorchestra_vm" "vm" { │ ╵

@Velocity1192 If you can wait a little, we are atively working on the new XO6 backup job uI, that will come with some changes : one schedule per job, but with conditionnal actions You will run an incremental job, and will be able to define that the schedule of sunday is a full,and even run a healthcheck on it The ETA is on 6.11 ( next month release )

@nick.lloyd I'll look into the HA support, need to dig into that whats needed to support that. keep you posted

@acebmxer I haven't run your script, so I can't vouch for the log lines, but the systemctl status check looks like the safer of the two to me. It shows the binary the service is really running. Whichever node comes first in your shell can be a different one if more than one Node is installed. For XOA I'd go with what @Danp said above.

@blueh2o thanks for the log. In that run 16 of the 56 VMs failed this way, and each one names a different missing base, so it doesn't look like one damaged chain to me. That's past what I can work out on my own, so @Team-XO-Backend, this one has a version and a full job log attached above now if someone can take a look.

For anyone else who steps through this, I've also installed the latest xen-guest-agent with the below process. Repeat for each host in the environment, 1 at a time from your Helper VM. ### ssh into your helper ### oc debug node/fedora-cmp-1/2/3 chroot /host cd /var/tmp curl -LO https://gitlab.com/xen-project/xen-guest-agent/-/jobs/6041686376/artifacts/raw/RPMS/x86_64/xen-guest-agent-0.4.0-0.fc37.x86_64.rpm sudo rpm-ostree install /var/tmp/xen-guest-agent-0.4.0-0.fc37.x86_64.rpm mkdir /var/tmp/xga && cd /var/tmp/xga rpm2cpio ../xen-guest-agent-0.4.0-0.fc37.x86_64.rpm | cpio -idmv ### From your helper... DRAIN YOUR WORKERS ### oc adm drain fedora-cmp-1/2/3 --ignore-daemonsets --delete-emptydir-data ### log back into fedora-cmp-1/2/3... ### sudo systemctl reboot ### from helper ### oc adm uncordon fedora-cmp-1/2/3 ### login to fedora-cmp-1/2/3 ### chroot /host sudo systemctl enable --now xen-guest-agent And then from XO this is what your VMs will look like. [image: image.jpeg]

Good to hear block mode did it. If I'm reading XO's troubleshooting page right, that 5-minute wait is a setting you can raise: httpInactivityTimeout under [xapiOptions] in a file like /etc/xo-server/config.httpInactivityTimeout.toml, then a restart of xo-server (https://docs.xen-orchestra.com/backups-and-dr/backup_troubleshooting#error-http-connection-has-timed-out). That path is the XOA one, and I don't know where the ronivay installer keeps its config, so check that first. The page calls it a workaround and says the slow host is the real thing to look at, which could fit your SMR-disk theory, but I haven't tried anything close to a 3 TB qcow2 delta myself. For the close() error, unless I'm misreading the code, XO closes the source after the failed start and only logs a warning before falling back to a full, so I'd guess it's noise from the timeout and not a second bug. The stuck tapdisks look more like one for @Team-Storage, and I don't know if a keep-alive during the delta computation is planned.

Sharing some results. I enabled this on my homelab server, a Dell R630 with Intel Xeon E5-2630 v4 (Broadwell). No adverse impact noticed. The CDM results on Windows is higher than previous runs however, I do not go in depth with I/O benchmarking. (I am using the latest win-pv-drivers on Windows and the rust-based guest tools on Linux) Controller is in HBA mode. VM SR is using SMAPIv3 (ZFS, RAID-0, noop scheduler) [image: R630_XCP-ng_APICv.png] CDM results. Windows Server 2019 4 vCPU/8GB RAM/single VHD: [image: 2610_WIN2019_ZFS.png] I installed ScyllaDB on Alma Linux 10.2 (4 vCPU/8GB RAM & two VHDs, ScyllaDB uses a dedicated drive for data) with io_uring enabled and noop scheduler set on VM. Here are the IOTune results from the ScyllaDB installation: Measuring sequential write bandwidth: 826 MiB/s (deviation 13%) Measuring sequential read bandwidth: 1061 MiB/s (deviation 6%) Measuring random write IOPS: 50791 IOPS (deviation 11%) Measuring random read IOPS: 62977 IOPS (deviation 16%)

@MrQ said: @jr-m4 thank you so much for testing this updated driver. Just to make sure, I understand now that you confirm the driver is working as expected and the issues you had were not related to it? Or are there any problems still with this new driver? Many thanks again for spending the time to test and report. Hi Yes. I believe I was chasing a red herring of my own doing. I have not found anything actually problematic with this driver at all.

@poddingue Thank you!

Hello Christof, Swapping the disks between the two VMs shows the problem follows whatever runs inside, not the VM itself. Next I would run xe vm-param-list uuid=<vm-uuid> on the host once with HAOS booted and once with Debian booted, and diff the two outputs. Fields like os-version, PV-drivers-version and networks are reported back by the running OS, so they change with the guest. My guess, and it's only a guess, is that one of them comes back in a shape Commvault's parser doesn't expect: "input string has the wrong format" is what .NET says when it fails to read a number. If you find the field that differs, Commvault support would have something concrete to work with.

OK, marking this solved... It was the phantom reboots causing issues because the tools were not fully installed, only thing I can't understand is why only 2 out of 3 Windows servers would quit. Any way, got the Xen Tools updated and a couple reboots later, everything is working again. Also I can now report that a bug that was fixed maybe a year ago is confirmed to work in my system. The bug was backing up a Windows VM that had a lot of empty space in the drive (over 250GB free) and turning on compression. It would time out counting zeros because it thought the process wasn't doing work. Both of these VMs are now compressing with zstd, one of them is substantially smaller now, the other only shrunk about 20GB. Because of the way Windows uses the drive, these are all full backups. The differential never seemed to be small because Windows does a lot of changes on the drive (there is a post about this somewhere in here from Olivier).

The UEFI and possibly SecureBoot requirements on a lot of stuff and going to turn a lot of old systems into garbage. Also going to be an issue when more current stuff gets upgrade from 8.3 in BIOS mode to requiring UEFI for version 9. That's the boat I'm in for production. Alma and openSuse may be a few of the last holdouts allowing v2 and BIOS boot.