@MajorP93: Thanks for expressing yourself regarding that, and I'll be transparent about the thinking behind it.

Filing those on GitHub isn't me asking anyone to stop using the forum, quite the opposite in fact.
The forum is where real conversations happen, and that's valuable in a way a GitHub issue never quite is. But forum threads scroll, get buried, and developers can't easily maintain a stable backlog out of them. GitHub gives the team a place where things don't disappear or get buried.

Think of it as belt and suspenders (which I need now that I'm getting old ). The discussion lives here, the tracking lives there. My goal as community manager is to be the relay between the two, so you don't have to worry about it.
File things here, talk about them here, and I'll make sure what matters makes it into the right repo.
Or at least, that's the plan. Mine.

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

Following up since the situation changed: QCOW2 went GA in XO 6.5 (released 2026-05-28), so the old ~2TB VHD ceiling is gone.
A disk at exactly 2TB, and well beyond it, is fine now without shrinking to 1.99TB first. When acebmxer and john.c replied, it was still a release candidate; it's the stable story now. I haven't migrated a disk quite that size myself, so I won't promise it's totally painless, but the format limit that was blocking you isn't there anymore.
The release blog has the details if you want to read up before the migration: https://xen-orchestra.com/blog/xen-orchestra-6-5/

The Kubernetes CSI driver for Xen Orchestra just hit v0.4.0, and we want it on more real clusters before it reaches a stable release candidate. If you run Kubernetes on XCP-ng VMs, this is a good time to give it a proper workout.

What's new in v0.4.0:

• Local-storage support
• Automatic pool-discovery fallback
• Kubernetes metadata now lives in Xen Orchestra VDI tags instead of the deprecated other_config. That change also drops the old requirement for Xen Orchestra 6.4 or newer, so the driver runs on more deployments now.

️ Read this before you upgrade. v0.4.0 is a breaking change. The Kubernetes metadata moved from other_config to VDI tags, so you must migrate before upgrading from v0.3.0. Do not upgrade in place: follow the v0.3.0 to v0.4.0 migration guide in the release notes, then move to v0.4.0.

What helps us most is hearing how it behaves on your own setup: what works, what breaks, which storage backend you use, and which flavour of Kubernetes you run (k3s, full k8s, or something else). Edge cases on real clusters are the ones we don't see in our own testing.

Where to report: start right here in this thread. It keeps everything visible to the community and lets others on the same setup jump in. If something turns out to be a reproducible bug, we'll move it to a GitHub issue on the repo so the team can track it to a fix.

Release notes and migration guide: https://github.com/vatesfr/xenorchestra-csi-driver/releases/tag/v0.4.0

Glad it's working, and thanks for posting the script, the next NUT user will want it (I hope I'll be able to experiment with NUT in the coming weeks, so I could very well be "the next NUT user").

I tried the PBD part on a lab host running 8.3: xe pbd-list never prints a storage-type line, so that grep finds nothing and step 2 quietly unplugs no PBDs. The 2>/dev/null hides it, and --force isn't an option pbd-unplug knows about. My wild guess is that ejecting the CDs is what fixed it for you.
If you want the unplug to really happen, going through the SR type works on my side, though I only tested it against an ISO SR, not SMB:

H=$(xe host-list hostname=$(hostname) --minimal)
for t in nfs smb iso lvmoiscsi; do
  for sr in $(xe sr-list type=$t --minimal | tr , ' '); do
    for pbd in $(xe pbd-list sr-uuid=$sr host-uuid=$H currently-attached=true --minimal | tr , ' '); do
      xe pbd-unplug uuid=$pbd
    done
  done
done

Block iSCSI SRs show up as lvmoiscsi, which is why it's not iscsi in the list. @Forza's point about hard mounts may still bite if the share is already gone when the unplug runs; I don't know enough about the SMB side to say.

The docs have a section for this symptom, Windows Server 2025 hanging at 0% CPU: https://docs.xcp-ng.org/troubleshooting/windows-pv-tools#windows-server-2025-hangs-randomly-with-0-cpu .

Besides the Viridian flags you've already set, it says to revert useplatformclock and useplatformtick with bcdedit if they were ever set in the guest.

Your earlier thread showed a VM created from the "Other install media" template, so it's worth checking which template this one came from, since the Windows templates bring their own Viridian settings.
I don't think the QEMU SIGTERM is the cause, though: when I force-shut down a VM on a lab host, qemu-dm logged that exact terminating on signal 15 line, sent by xenopsd-xc (ps -o comm -p 1526469 will tell you on yours), so yours most likely just matches your forced shutdown around 10:20 on 5 Oct.
If it freezes again after that, the memory dump procedure further down the same page (https://docs.xcp-ng.org/troubleshooting/windows-pv-tools#how-to-gather-kernel-memory-dumps-for-in-depth-troubleshooting) is likely what people who know Windows guests better than me would want to see.

Good to hear Restart got maintenance mode going.

I agree it shouldn't depend on spotting the one VM left on best-effort, and the Feeder item you found is the right home for that idea, so a vote plus a line about your production plans will do more there than anything I can add.

About 7146: its description says it only changes the evacuation when HA tolerates zero host failures, which is basically the two-host case.
I had that wrong in my first reply.
If your 3-host pool is set to tolerate one failure (my guess, I haven't seen your config), it wouldn't change anything for you.

It was too tempting not to test, so I went and tested the other half.
On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it.
The clone came up at 800x600.
A control clone of the same original, with nothing set, came up at 1024x768.
So your workaround holds, the resolution really does ride along into VMs built from the template.

For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record.

One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024.
So you may not need to raise videoram for the common ones.

Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it.
At least, that's my understanding.

Thanks for coming back with the actual cause instead of leaving it hanging.
To me, a permissions reset on /etc/rc.d/rc.local would explain it completely, and it fails in the worst way, since nothing complains at boot and the VMs just never start.
It also looks worth documenting, because our own troubleshooting page tells people to append a clocksource workaround to /etc/rc.local without mentioning the executable bit either, so anyone following that could end up with a workaround that quietly does nothing.
I haven't tested whether the 8.2 to 8.3 upgrade resets that bit every time or just happened to in your case, so that half stays your finding, not mine.

I'm curious what you turn up on the colon versus equals behaviour.

Hey, thanks for writing this up with the actual commands and the screenshot, that really helps.

Not sure why rc.local + vApps stopped working after your 8.2 to 8.3 jump. rc.local is more your own dom0-level script than something XCP-ng promises to keep stable across releases, so something underneath may well have shifted, but I can't point to what.

On the delay question, there does seem to be a per-VM "start-delay" parameter. I ran into it replying to a pretty similar issue a couple days ago, if you want to compare notes: https://xcp-ng.org/forum/topic/12388. There are also the "appliance" CLI commands, the vApp equivalent, though from another recent thread it looked like nobody's fully sure how start order and delay actually interact there: https://xcp-ng.org/forum/topic/12149. So I think the capability is there, just not pulled together in one place.

The colon-vs-equals thing is interesting. I honestly don't know xe's map-param parsing well enough to say if that's expected or a rough edge. Maybe someone closer to the CLI internals can say whether it should just reject that instead of creating duplicate entries. Given how scattered this all is, might be worth flagging to @Team-Documentation-Knowledge-Management.

Hope some of that's useful, others probably know xe's internals better than me though.

Two PRs out of the lowest priority bug ever, that made my morning.

From what I understand, when XCP-ng reverts to a snapshot it restores the full VM state from that point (metadata included, not just the disk contents) so the creation date field would get rolled back along with everything else; that might be why it now matches the snapshot timestamp rather than
the original. I might be wrong about the internals though.

It's a bit confusing if you were relying on that field to track VM history, and I don't think https://docs.xen-orchestra.com/xo5/manage_infrastructure#snapshot-management covers this explicitly.
Might be worth a mention to @Team-Documentation-Knowledge-Management; it's the kind of thing that catches people off guard because nothing warns you upfront that metadata rolls back too.
My $0.02.

@Jonathon if I read your xen02 output right, from before the resync it already had 0 out of sync towards both other nodes, so the big number only ever lived between xen01 and xen05.

After invalidate-remote both of them are copying fully from xen02 (that's the SyncSource and Inconsistent lines in your last post).

I don't know whether that secondary-to-secondary figure comes back on its own once the resync is done.
If it does, the date it reappears and roughly what that k8s worker was doing at the time are probably what @Team-Storage would want to see. I'm curious too, so please do post how it looks after a few days.

Thanks @jr-m4, reporting the small ones is useful too.

From what I can see in the code, the VM view of the restore page has that refresh button and the Metadata view only has a link back, so I think your screenshots show a real gap and not a glitch on your side.
I honestly don't know whether XO 5 still takes this kind of small UI fix now that most of the work goes into XO 6, so I'll leave that to @Team-XO-Frontend to decide.

Hi, @Velocity1192 .

On v6 first: creating and editing backup jobs still happens in XO 5 for now, XO 6 shows the jobs but doesn't create them yet (https://docs.xen-orchestra.com/backups-and-dr/backups-in-xo6).

For the schedule, @abudef asked almost the same question earlier this year. @Pilow's answer there is clearer than anything I'd write: https://xcp-ng.org/forum/post/104108.
As I read it, you either use one daily schedule plus the "Full backup interval" setting (the full lands N runs after the chain starts), or two schedules on the same job with "Force full backup" ticked on the Sunday one.
That second way is how you get to pick the day, and he says not to mix the two. One thing to expect, from @Danp in another thread (https://xcp-ng.org/forum/post/91943) : the oldest backup is always a full and old deltas get merged into it, so for a while you'll see two fulls on the remote rather than weeks 1 and 2 being deleted. I don't think the scheduler can say "every other Sunday". I might be wrong there; someone who runs this kind of plan may know a trick.

@jr-m4 : I tried it on a lab pool today. XO does catch the refusal, turn CBT off and retry, but only on the disks of the VM itself. If the VM has a normal snapshot, the snapshot's disk still has CBT on, XCP-ng checks that disk too and refuses the move, and it won't let you turn CBT off on a snapshot. So with snapshots around you have to delete them first, like you said. I had that part wrong.

@archw your setup is the other case. With "Purge snapshot data when using CBT" on and retention at 0, the backup job deletes the snapshot and keeps only a small metadata file that isn't attached to anything. In the same test that file didn't get in the way: once CBT was off on the live disk, the move went through. So for VMs backed up that way, migrating from XO should still work, with a full backup on the next run. A snapshot you take by hand would block it the same way.

@jr-m4 since my last post I ran the whole thing through XO to be sure. Two VMs, each with nothing but an empty 2 MiB disk, same delta job: the UEFI one passed its health check in a few seconds, and the BIOS one failed after the 10 minute timeout waiting for the guest tools, which is what the docs promise.

So that sentence only holds for BIOS VMs, and your case of a rarely used VM with a bad disk is where that matters. I've passed both results to the XO team, though I don't know yet whether they'll change the check or the docs. Until then, the xo-backup-health-check-xenstore tag I mentioned earlier is the way to make the check depend on something actually running inside the VM.

I tried this on a lab host this morning. XO's health check waits for XAPI's PV-drivers-detected flag on the restored VM.

A test VM with nothing but an empty 2 MiB disk reported that flag as true within 3 seconds when started with UEFI firmware, while the same VM under BIOS just halted and stayed false.

So if your two VMs are UEFI, that would explain the pass, though I'm only guessing at why the firmware sets it. Are they?

The docs line you quoted doesn't hold in that case, and https://xcp-ng.org/forum/topic/11066 ran into a cousin of this with a FreeBSD guest. If you need the check to prove the OS really came up, the xo-backup-health-check-xenstore tag makes it wait for a script inside the VM instead: https://docs.xen-orchestra.com/backups-and-dr/backup-features-and-settings#backup-health-check.

I think it's worth a look from @Team-XO-Backend too, since a passing check on a UEFI VM may say less than people assume.

Thanks for the feedback so far, folks!

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.