@nick.lloyd I'll look into the HA support, need to dig into that whats needed to support that. keep you posted

FYI, our approach with vectura is only temporary, we have other plans in few months to get rid of this potential cat & mouse game Broadcom could do by slightly change their protocol (without even talking about the potential legal risk with it).

@irtaza9 What do you see if you run top or xtop, or "xe task-list" when you start the process?

Adding @dinhngtu in the loop in case

@zorro I tried to reproduce this on a small lab host with plain xe commands on XCP-ng 8.3, not with XO and Veeam themselves. I think your read is right: both tools end up on the same CBT chain. When a newer snapshot sitting between Veeam's reference and the live disk has had its data purged and is then deleted, the changed-block query from the older snapshot fails with exactly your SR_BACKEND_FAILURE_460 ... Source and target VDI are unrelated, and if I read XO's code right, that purge-then-delete is what the "Purge snapshot data" option does on each run. When that middle snapshot is a normal one, deleting it doesn't break anything in my test. So turning purge off might well avoid it (I think XO then keeps a full snapshot on the host instead), but please treat that as a lead rather than an answer. @Team-Storage will know whether that's expected; until then @acebmxer's one-tool-per-VM advice is the safe one.

@acebmxer I haven't run your script, so I can't vouch for the log lines, but the systemctl status check looks like the safer of the two to me. It shows the binary the service is really running. Whichever node comes first in your shell can be a different one if more than one Node is installed. For XOA I'd go with what @Danp said above.

@blueh2o thanks for the log. In that run 16 of the 56 VMs failed this way, and each one names a different missing base, so it doesn't look like one damaged chain to me. That's past what I can work out on my own, so @Team-XO-Backend, this one has a version and a full job log attached above now if someone can take a look.

Thanks @cocoon, and I owe a correction: when I said in https://xcp-ng.org/forum/post/104865 that 6.4 would make these alerts disappear for good, I overstated it. If I'm reading the code right this time, the XO server sends a ping every 30 seconds stamped with its own clock (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/rest-api/src/events/event.class.mts), and the browser compares that stamp with its own clock and shows the banner once the gap passes 40 seconds (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/web-core/lib/packages/remote-resource/sse.store.ts). So https://github.com/vatesfr/xen-orchestra/pull/9681 widened the margin from 32 to 40 seconds, but a browser whose clock runs ahead of the XO server by more than about 10 seconds would still see it. That might be why @GregBinSD saw it on the Windows PC and not on the tablet, though that's only a guess on my part. Comparing two different clocks looks fragile to me, but @Team-XO-Frontend will know far better than I do whether there's a reason it's done that way.

@Velocity1192 If you can wait a little, we are atively working on the new XO6 backup job uI, that will come with some changes : one schedule per job, but with conditionnal actions You will run an incremental job, and will be able to define that the schedule of sunday is a full,and even run a healthcheck on it The ETA is on 6.11 ( next month release )

I'm also investigating on a solution to make XCP-ng opportunistically resize device BAR; in case the firmware didn't do it (or has no support for it).

Thanks @jr-m4, reporting the small ones is useful too. From what I can see in the code, the VM view of the restore page has that refresh button and the Metadata view only has a link back, so I think your screenshots show a real gap and not a glitch on your side. I honestly don't know whether XO 5 still takes this kind of small UI fix now that most of the work goes into XO 6, so I'll leave that to @Team-XO-Frontend to decide.

For some reason, we have the same issue (missing disks / snapshot-of) on our environment. (XO from source commit a5077) This is a major issue for us. We use Terraform to manage our VMs, and the missing VDIs are causing Terraform to fail. As a result, we are currently unable to manage our VMs using Terraform. ➜ dacsvm-db02 git:(main) tofu plan -var-file="dacsvm-db02.tfvars" module.vm.xenorchestra_vm.vm: Refreshing state... [id=e36fcd8a-b88c-42f7-6d61-2aa639f63b4a] Planning failed. OpenTofu encountered an error while generating this plan. ╷ │ Error: Could not find client.VDI with query: {VDIId:cad7d739-35c2-4585-9991-4c2f3fc6531c SrId: NameLabel: NameDescription: Size:0 VBDs:[] Parent: PoolId: Tags:[]} │ │ with module.vm.xenorchestra_vm.vm, │ on ../modules/vm/main.tf line 16, in resource "xenorchestra_vm" "vm": │ 16: resource "xenorchestra_vm" "vm" { │ ╵