@irtaza9 What do you see if you run top or xtop, or "xe task-list" when you start the process?

@nick.lloyd I'll look into the HA support, need to dig into that whats needed to support that. keep you posted

@blueh2o thanks for the log. In that run 16 of the 56 VMs failed this way, and each one names a different missing base, so it doesn't look like one damaged chain to me. That's past what I can work out on my own, so @Team-XO-Backend, this one has a version and a full job log attached above now if someone can take a look.

@zorro I would advise not backing up the same vms with both XO and veeam. One or the other. No did not see the update yet. I will look and check. Edit - thanks or pointing out the new version. I was still on 13.1.0.411. Downloading update now.

@acebmxer I haven't run your script, so I can't vouch for the log lines, but the systemctl status check looks like the safer of the two to me. It shows the binary the service is really running. Whichever node comes first in your shell can be a different one if more than one Node is installed. For XOA I'd go with what @Danp said above.

@Velocity1192 If you can wait a little, we are atively working on the new XO6 backup job uI, that will come with some changes : one schedule per job, but with conditionnal actions You will run an incremental job, and will be able to define that the schedule of sunday is a full,and even run a healthcheck on it The ETA is on 6.11 ( next month release )

For some reason, we have the same issue (missing disks / snapshot-of) on our environment. (XO from source commit a5077) This is a major issue for us. We use Terraform to manage our VMs, and the missing VDIs are causing Terraform to fail. As a result, we are currently unable to manage our VMs using Terraform. ➜ dacsvm-db02 git:(main) tofu plan -var-file="dacsvm-db02.tfvars" module.vm.xenorchestra_vm.vm: Refreshing state... [id=e36fcd8a-b88c-42f7-6d61-2aa639f63b4a] Planning failed. OpenTofu encountered an error while generating this plan. ╷ │ Error: Could not find client.VDI with query: {VDIId:cad7d739-35c2-4585-9991-4c2f3fc6531c SrId: NameLabel: NameDescription: Size:0 VBDs:[] Parent: PoolId: Tags:[]} │ │ with module.vm.xenorchestra_vm.vm, │ on ../modules/vm/main.tf line 16, in resource "xenorchestra_vm" "vm": │ 16: resource "xenorchestra_vm" "vm" { │ ╵

Good to hear block mode did it. If I'm reading XO's troubleshooting page right, that 5-minute wait is a setting you can raise: httpInactivityTimeout under [xapiOptions] in a file like /etc/xo-server/config.httpInactivityTimeout.toml, then a restart of xo-server (https://docs.xen-orchestra.com/backups-and-dr/backup_troubleshooting#error-http-connection-has-timed-out). That path is the XOA one, and I don't know where the ronivay installer keeps its config, so check that first. The page calls it a workaround and says the slow host is the real thing to look at, which could fit your SMR-disk theory, but I haven't tried anything close to a 3 TB qcow2 delta myself. For the close() error, unless I'm misreading the code, XO closes the source after the failed start and only logs a warning before falling back to a full, so I'd guess it's noise from the timeout and not a second bug. The stuck tapdisks look more like one for @Team-Storage, and I don't know if a keep-alive during the delta computation is planned.

For anyone else who steps through this, I've also installed the latest xen-guest-agent with the below process. Repeat for each host in the environment, 1 at a time from your Helper VM. ### ssh into your helper ### oc debug node/fedora-cmp-1/2/3 chroot /host cd /var/tmp curl -LO https://gitlab.com/xen-project/xen-guest-agent/-/jobs/6041686376/artifacts/raw/RPMS/x86_64/xen-guest-agent-0.4.0-0.fc37.x86_64.rpm sudo rpm-ostree install /var/tmp/xen-guest-agent-0.4.0-0.fc37.x86_64.rpm mkdir /var/tmp/xga && cd /var/tmp/xga rpm2cpio ../xen-guest-agent-0.4.0-0.fc37.x86_64.rpm | cpio -idmv ### From your helper... DRAIN YOUR WORKERS ### oc adm drain fedora-cmp-1/2/3 --ignore-daemonsets --delete-emptydir-data ### log back into fedora-cmp-1/2/3... ### sudo systemctl reboot ### from helper ### oc adm uncordon fedora-cmp-1/2/3 ### login to fedora-cmp-1/2/3 ### chroot /host sudo systemctl enable --now xen-guest-agent And then from XO this is what your VMs will look like. [image: image.jpeg]