@zorro So with purge off, Veeam stayed clean for a few nights, which fits what I saw in the lab test.
If I read the docs right, the cost is that XO's reference snapshot keeps its data on the SR again, which you'd notice more on thick-provisioned storage: https://docs.xen-orchestra.com/backups-and-dr/backup-types/incremental_backups#purge-snapshot-data-cbt.
Storage hasn't said yet whether purge breaking another tool's chain is expected, so I'd still call this a workaround rather than the answer.

If a CBT warning comes back when you add the other VMs, the Veeam task id and the time would probably help them. And keeping both makes sense to me, since Veeam doesn't do the cross-site replication you need XO for.

I had a look at the XO5 code. If I'm reading it right, the bulk action hands the list of SRs to Promise.all together with the function that should disconnect each one, but Promise.all only takes the list, so the disconnect calls never go out.
The single-SR action calls the API directly, which would explain why doing it from each SR's page works.
The bulk "Reconnect to all hosts" looks like it's written the same way, if you feel like checking it.
Until it's fixed, one SR at a time is the only workaround I know of.
Might be worth a mention to @Team-XO-Frontend, they'll know better than me if there's more to it.

Thanks @jr-m4, reporting the small ones is useful too.

From what I can see in the code, the VM view of the restore page has that refresh button and the Metadata view only has a link back, so I think your screenshots show a real gap and not a glitch on your side.
I honestly don't know whether XO 5 still takes this kind of small UI fix now that most of the work goes into XO 6, so I'll leave that to @Team-XO-Frontend to decide.

@zorro I tried to reproduce this on a small lab host with plain xe commands on XCP-ng 8.3, not with XO and Veeam themselves.
I think your read is right: both tools end up on the same CBT chain. When a newer snapshot sitting between Veeam's reference and the live disk has had its data purged and is then deleted, the changed-block query from the older snapshot fails with exactly your SR_BACKEND_FAILURE_460 ... Source and target VDI are unrelated, and if I read XO's code right, that purge-then-delete is what the "Purge snapshot data" option does on each run. When that middle snapshot is a normal one, deleting it doesn't break anything in my test. So turning purge off might well avoid it (I think XO then keeps a full snapshot on the host instead), but please treat that as a lead rather than an answer. @Team-Storage will know whether that's expected; until then @acebmxer's one-tool-per-VM advice is the safe one.

Thanks @cocoon, and I owe a correction: when I said in https://xcp-ng.org/forum/post/104865 that 6.4 would make these alerts disappear for good, I overstated it.

If I'm reading the code right this time, the XO server sends a ping every 30 seconds stamped with its own clock (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/rest-api/src/events/event.class.mts), and the browser compares that stamp with its own clock and shows the banner once the gap passes 40 seconds (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/web-core/lib/packages/remote-resource/sse.store.ts).

So https://github.com/vatesfr/xen-orchestra/pull/9681 widened the margin from 32 to 40 seconds, but a browser whose clock runs ahead of the XO server by more than about 10 seconds would still see it.

That might be why @GregBinSD saw it on the Windows PC and not on the tablet, though that's only a guess on my part. Comparing two different clocks looks fragile to me, but @Team-XO-Frontend will know far better than I do whether there's a reason it's done that way.

Hi, @Velocity1192 .

On v6 first: creating and editing backup jobs still happens in XO 5 for now, XO 6 shows the jobs but doesn't create them yet (https://docs.xen-orchestra.com/backups-and-dr/backups-in-xo6).

For the schedule, @abudef asked almost the same question earlier this year. @Pilow's answer there is clearer than anything I'd write: https://xcp-ng.org/forum/post/104108.
As I read it, you either use one daily schedule plus the "Full backup interval" setting (the full lands N runs after the chain starts), or two schedules on the same job with "Force full backup" ticked on the Sunday one.
That second way is how you get to pick the day, and he says not to mix the two. One thing to expect, from @Danp in another thread (https://xcp-ng.org/forum/post/91943) : the oldest backup is always a full and old deltas get merged into it, so for a while you'll see two fulls on the remote rather than weeks 1 and 2 being deleted. I don't think the scheduler can say "every other Sunday". I might be wrong there; someone who runs this kind of plan may know a trick.

@acebmxer I haven't run your script, so I can't vouch for the log lines, but the systemctl status check looks like the safer of the two to me. It shows the binary the service is really running. Whichever node comes first in your shell can be a different one if more than one Node is installed. For XOA I'd go with what @Danp said above.

@blueh2o thanks for the log. In that run 16 of the 56 VMs failed this way, and each one names a different missing base, so it doesn't look like one damaged chain to me.
That's past what I can work out on my own, so @Team-XO-Backend, this one has a version and a full job log attached above now if someone can take a look.

The same error came up last year (https://xcp-ng.org/forum/topic/11086/how-do-i-diagnose-a-missing-vdi-error-with-mirror-backups), and @florent's advice there was to delete the most recent backups of the failing VM on the mirror target, so they get transferred again from the source.

You already wiped the target and it came back, so I suspect yours is a different case, but I don't know the mirror code well enough to say why.
The XO version and the job log @acebmxer asked for would narrow it down, and with those it might be one for @Team-XO-Backend.

Good to hear block mode did it.

If I'm reading XO's troubleshooting page right, that 5-minute wait is a setting you can raise: httpInactivityTimeout under [xapiOptions] in a file like /etc/xo-server/config.httpInactivityTimeout.toml, then a restart of xo-server (https://docs.xen-orchestra.com/backups-and-dr/backup_troubleshooting#error-http-connection-has-timed-out).
That path is the XOA one, and I don't know where the ronivay installer keeps its config, so check that first. The page calls it a workaround and says the slow host is the real thing to look at, which could fit your SMR-disk theory, but I haven't tried anything close to a 3 TB qcow2 delta myself.

For the close() error, unless I'm misreading the code, XO closes the source after the failed start and only logs a warning before falling back to a full, so I'd guess it's noise from the timeout and not a second bug.

The stuck tapdisks look more like one for @Team-Storage, and I don't know if a keep-alive during the delta computation is planned.

Hi @Razielh,

I think that error may be the 2 TB VHD ceiling, and it should go away if the backup repository is in block mode.

The same thing came up in July on GitHub (https://github.com/vatesfr/xen-orchestra/issues/10193, a 6 TB qcow2 disk), where @Florent explained that a repository not in block mode writes each disk as one whole VHD file, which can't go past 2 TB.
The reporter made a new share on their NAS, added it as a remote with "Store backup as multiple data blocks instead of a whole VHD file" ticked, and the 6 TB VM backed up after that.
The docs say it too, though as a bullet in a list rather than as a hard limit: https://docs.xen-orchestra.com/backups-and-dr/backup-features-and-settings#remotes.

While you're in there, incremental backups of qcow2 disks also need NBD on the job, or every run falls back to a full one (https://docs.xen-orchestra.com/backups-and-dr/backup-types/incremental_backups#nbd-enabled-backups); I haven't seen your job settings, so ignore that if it's already on.

I'm not sure an existing remote can be switched to block mode in place, so a fresh share like in that issue seems the safer bet. Might also be worth a mention to @Team-XO-Backend that XO could say this up front instead of throwing a RangeError mid-transfer.

@jr-m4 : I tried it on a lab pool today. XO does catch the refusal, turn CBT off and retry, but only on the disks of the VM itself. If the VM has a normal snapshot, the snapshot's disk still has CBT on, XCP-ng checks that disk too and refuses the move, and it won't let you turn CBT off on a snapshot. So with snapshots around you have to delete them first, like you said. I had that part wrong.

@archw your setup is the other case. With "Purge snapshot data when using CBT" on and retention at 0, the backup job deletes the snapshot and keeps only a small metadata file that isn't attached to anything. In the same test that file didn't get in the way: once CBT was off on the live disk, the move went through. So for VMs backed up that way, migrating from XO should still work, with a full backup on the next run. A snapshot you take by hand would block it the same way.

Thanks for the heads-up.

If you built XO from sources, the docs say to keep Node on the latest LTS (https://docs.xen-orchestra.com/getting-started/install-from-sources#nodejs), so it's worth moving to the patched release on whichever line you run.

For XOA, I don't know how fast the appliance picks up a new Node release, but I know something's cooking.

Hello Christof,

Swapping the disks between the two VMs shows the problem follows whatever runs inside, not the VM itself.
Next I would run xe vm-param-list uuid=<vm-uuid> on the host once with HAOS booted and once with Debian booted, and diff the two outputs.
Fields like os-version, PV-drivers-version and networks are reported back by the running OS, so they change with the guest.
My guess, and it's only a guess, is that one of them comes back in a shape Commvault's parser doesn't expect: "input string has the wrong format" is what .NET says when it fails to read a number.
If you find the field that differs, Commvault support would have something concrete to work with.

From what I read in the docs and a bit of XO's code, a rolling pool update live-migrates the VMs off each host before rebooting it, and once every host is done, XO moves each VM back to the host it started on (https://docs.xen-orchestra.com/manage-your-infrastructure/manage_infrastructure#rolling-pool-updates-rpu).

So the control planes should keep running rather than be shut down. The load balancer is suspended for the whole RPU, though, which means anti-affinity (https://docs.xen-orchestra.com/manage-your-infrastructure/load_balancing#vm-anti-affinity) keeps the CPs apart day to day but won't steer them while the update runs.

Whether etcd copes with the short freeze at the end of each live migration, I don't know. I've never run OKD myself.

How many hosts are in that pool? With two, one host always carries two of the three CPs during the update, and that might be where quorum gets tight.

If you do the storage migration from XO, you shouldn't have to do anything.
XAPI does refuse to move a disk while CBT is on, as @jr-m4 said, but reading XO's code it catches that refusal, turns CBT off on the VM's disks and retries.
The price to pay is that the next backup of that VM is a full one, and the backup job switches CBT back on.

The other trade-offs are in the docs (https://docs.xen-orchestra.com/backups-and-dr/backup-types/incremental_backups#purge-snapshot-data-cbt) : NBD has to be enabled, snapshot retention has to be 0, and since the purged snapshot only keeps metadata you can't roll back to it.

@JamesG, I'm the one who marked it solved, so here's what I meant.

What Teddy showed in https://xcp-ng.org/forum/post/109066 is that there's one thing to do on the XCP-ng side for the B50: turn HVM PIRQ off for that VM (xe vm-param-set uuid=<vm-uuid> platform:hvm-pirq=false, then reboot the VM), as described on the docs page he linked: https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7.

With that, his benchmark came out normal for the card, and the Gen1 x1 you see inside the guest is something QEMU makes up, not the real link (his earlier post: https://xcp-ng.org/forum/post/106479). It doesn't mean every problem is on the guest side though: @pandusen tried the same setting (https://xcp-ng.org/forum/post/109084) and still had driver crashes on headless distros, and I don't know why. If you try it on your B50 and post what you see, that would help tell the two cases apart.

Glad it's working, and thanks for posting the script, the next NUT user will want it (I hope I'll be able to experiment with NUT in the coming weeks, so I could very well be "the next NUT user").

I tried the PBD part on a lab host running 8.3: xe pbd-list never prints a storage-type line, so that grep finds nothing and step 2 quietly unplugs no PBDs. The 2>/dev/null hides it, and --force isn't an option pbd-unplug knows about. My wild guess is that ejecting the CDs is what fixed it for you.
If you want the unplug to really happen, going through the SR type works on my side, though I only tested it against an ISO SR, not SMB:

H=$(xe host-list hostname=$(hostname) --minimal)
for t in nfs smb iso lvmoiscsi; do
  for sr in $(xe sr-list type=$t --minimal | tr , ' '); do
    for pbd in $(xe pbd-list sr-uuid=$sr host-uuid=$H currently-attached=true --minimal | tr , ' '); do
      xe pbd-unplug uuid=$pbd
    done
  done
done

Block iSCSI SRs show up as lvmoiscsi, which is why it's not iscsi in the list. @Forza's point about hard mounts may still bite if the share is already gone when the unplug runs; I don't know enough about the SMB side to say.

@jr-m4 since my last post I ran the whole thing through XO to be sure. Two VMs, each with nothing but an empty 2 MiB disk, same delta job: the UEFI one passed its health check in a few seconds, and the BIOS one failed after the 10 minute timeout waiting for the guest tools, which is what the docs promise.

So that sentence only holds for BIOS VMs, and your case of a rarely used VM with a bad disk is where that matters. I've passed both results to the XO team, though I don't know yet whether they'll change the check or the docs. Until then, the xo-backup-health-check-xenstore tag I mentioned earlier is the way to make the check depend on something actually running inside the VM.

@dknight-bg the Feeder post you linked asks for two things: sending audit events to syslog, which is what this PR does, and recording successful logins, which as far as I can tell the PR doesn't touch.

If others reading this need either one, a vote on https://feedback.vates.tech/posts/88/audit-log-forward-events-to-remote-syslog-record-successful-logins is the easiest way to show it. The PR is still waiting for a reviewer, as far as I can see.