@MajorP93: Thanks for expressing yourself regarding that, and I'll be transparent about the thinking behind it.

Filing those on GitHub isn't me asking anyone to stop using the forum, quite the opposite in fact.
The forum is where real conversations happen, and that's valuable in a way a GitHub issue never quite is. But forum threads scroll, get buried, and developers can't easily maintain a stable backlog out of them. GitHub gives the team a place where things don't disappear or get buried.

Think of it as belt and suspenders (which I need now that I'm getting old ). The discussion lives here, the tracking lives there. My goal as community manager is to be the relay between the two, so you don't have to worry about it.
File things here, talk about them here, and I'll make sure what matters makes it into the right repo.
Or at least, that's the plan. Mine.

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

The Kubernetes CSI driver for Xen Orchestra just hit v0.4.0, and we want it on more real clusters before it reaches a stable release candidate. If you run Kubernetes on XCP-ng VMs, this is a good time to give it a proper workout.

What's new in v0.4.0:

• Local-storage support
• Automatic pool-discovery fallback
• Kubernetes metadata now lives in Xen Orchestra VDI tags instead of the deprecated other_config. That change also drops the old requirement for Xen Orchestra 6.4 or newer, so the driver runs on more deployments now.

️ Read this before you upgrade. v0.4.0 is a breaking change. The Kubernetes metadata moved from other_config to VDI tags, so you must migrate before upgrading from v0.3.0. Do not upgrade in place: follow the v0.3.0 to v0.4.0 migration guide in the release notes, then move to v0.4.0.

What helps us most is hearing how it behaves on your own setup: what works, what breaks, which storage backend you use, and which flavour of Kubernetes you run (k3s, full k8s, or something else). Edge cases on real clusters are the ones we don't see in our own testing.

Where to report: start right here in this thread. It keeps everything visible to the community and lets others on the same setup jump in. If something turns out to be a reproducible bug, we'll move it to a GitHub issue on the repo so the team can track it to a fix.

Release notes and migration guide: https://github.com/vatesfr/xenorchestra-csi-driver/releases/tag/v0.4.0

It was too tempting not to test, so I went and tested the other half.
On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it.
The clone came up at 800x600.
A control clone of the same original, with nothing set, came up at 1024x768.
So your workaround holds, the resolution really does ride along into VMs built from the template.

For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record.

One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024.
So you may not need to raise videoram for the common ones.

Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it.
At least, that's my understanding.

Thanks for coming back with the actual cause instead of leaving it hanging.
To me, a permissions reset on /etc/rc.d/rc.local would explain it completely, and it fails in the worst way, since nothing complains at boot and the VMs just never start.
It also looks worth documenting, because our own troubleshooting page tells people to append a clocksource workaround to /etc/rc.local without mentioning the executable bit either, so anyone following that could end up with a workaround that quietly does nothing.
I haven't tested whether the 8.2 to 8.3 upgrade resets that bit every time or just happened to in your case, so that half stays your finding, not mine.

I'm curious what you turn up on the colon versus equals behaviour.

Hey, thanks for writing this up with the actual commands and the screenshot, that really helps.

Not sure why rc.local + vApps stopped working after your 8.2 to 8.3 jump. rc.local is more your own dom0-level script than something XCP-ng promises to keep stable across releases, so something underneath may well have shifted, but I can't point to what.

On the delay question, there does seem to be a per-VM "start-delay" parameter. I ran into it replying to a pretty similar issue a couple days ago, if you want to compare notes: https://xcp-ng.org/forum/topic/12388. There are also the "appliance" CLI commands, the vApp equivalent, though from another recent thread it looked like nobody's fully sure how start order and delay actually interact there: https://xcp-ng.org/forum/topic/12149. So I think the capability is there, just not pulled together in one place.

The colon-vs-equals thing is interesting. I honestly don't know xe's map-param parsing well enough to say if that's expected or a rough edge. Maybe someone closer to the CLI internals can say whether it should just reject that instead of creating duplicate entries. Given how scattered this all is, might be worth flagging to @Team-Documentation-Knowledge-Management.

Hope some of that's useful, others probably know xe's internals better than me though.

Two PRs out of the lowest priority bug ever, that made my morning.

From what I understand, when XCP-ng reverts to a snapshot it restores the full VM state from that point (metadata included, not just the disk contents) so the creation date field would get rolled back along with everything else; that might be why it now matches the snapshot timestamp rather than
the original. I might be wrong about the internals though.

It's a bit confusing if you were relying on that field to track VM history, and I don't think https://docs.xen-orchestra.com/xo5/manage_infrastructure#snapshot-management covers this explicitly.
Might be worth a mention to @Team-Documentation-Knowledge-Management; it's the kind of thing that catches people off guard because nothing warns you upfront that metadata rolls back too.
My $0.02.

Thanks for the feedback so far, folks!

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.

That's excellent news, thanks a lot for the feedback!

Reviving this one because it never really got a clean ending. The fix for the Body Timeout Error on full backups went into xapi-project/xen-api#6786 and shipped with the March 2026 8.3 maintenance updates.

@nikade, back in March, you mentioned still hitting it on both Linux and Windows VMs even after updating. Is that still the case for you on current versions? And anyone else who landed here: are you still seeing it, or did the update sort it out?

Mostly trying to work out whether there's still something to chase or whether we can call this resolved. I'm not deep enough in the backup internals to say for sure, so your real-world results would tell us far more than anything I could guess. Thanks!

Thanks for following up and opening the issue; that's exactly the kind of report the team needs. I've subscribed to it, so I'll see when it moves. The second-run failure pattern is a useful clue; hopefully they can reproduce it from there.

Hi!
I think what you're seeing may be stale entries in the XAPI database, ghost records that can survive upgrades or host reconfigurations.
From what I've read, xe vm-destroy uuid=<vm-uuid> removes the record without touching any storage, which seems like what you need here; the xe CLI reference confirms storage is left intact.
I think you can get the UUID first with xe vm-list name-label="Before Ubuntu Update" (replacing the name with whichever one you're after).
I'm not entirely sure why xsconsole would show them but XO wouldn't, so if the VMs don't turn up in xe vm-list, it might be worth a mention to Team-XAPI-Network, they'll know the right way to dig into XAPI state.

Following up since the situation changed: QCOW2 went GA in XO 6.5 (released 2026-05-28), so the old ~2TB VHD ceiling is gone.
A disk at exactly 2TB, and well beyond it, is fine now without shrinking to 1.99TB first. When acebmxer and john.c replied, it was still a release candidate; it's the stable story now. I haven't migrated a disk quite that size myself, so I won't promise it's totally painless, but the format limit that was blocking you isn't there anymore.
The release blog has the details if you want to read up before the migration: https://xen-orchestra.com/blog/xen-orchestra-6-5/

This is great to see, thank you for taking the time to rescue this; and thanks to @john.c for the recovery work and to @tjkreidl for writing it in the first place.
I went looking, and there is a small XCP-ng-specific piece on this in the official docs under NUMA affinity (https://docs.xcp-ng.org/compute#numa-affinity), but it's nothing like the depth of the Tale of Two Servers series, so having the originals archived is genuinely useful.
I won't pretend to judge how much of the 2019 BIOS and GPU-scheduler guidance still maps cleanly onto current hardware and XCP-ng versions; others here will know where it's aged and where it hasn't.
I'll make sure this is on our radar on the docs side, because it keeps coming up. Really appreciate you keeping this from disappearing.

Good news, this already shipped in XO 6.5.0 (released 2026-05-28), PR #9835.
If your XOA is on the latest channel, you've got PATCH /vms/{id} now; on the stable channel, it should land when 6.5.0 gets promoted (stable is on 6.4.1 at the moment). name_description looks like the field you'd write that "what's running here" summary to for your n8n flow, though I haven't tried the new endpoint myself yet. Let us know how it goes.

Coming back to this with what the distros actually ship, because "the fix is upstream" turned out not to mean much on its own.

The commit is f24df84cbe05. I checked each distro by grepping kernel/time/jiffies.c at the version they ship, rather than comparing version numbers, since some of them cherry-pick.

Already fixed, nothing to do:

• Fedora 44: 7.1.8
• Alpine 3.22: linux-virt 6.12.103. Alpine 3.23 and edge: 6.18.44

Not fixed in what you get today:

• Debian 13: trixie ships 6.12.94, which doesn't have it. 6.12.100 and 6.12.101 do, and they're in trixie-proposed-updates, so it should land with the next point release.
• Ubuntu 26.04: -updates moved to 7.0.0-30 this morning and that doesn't have it either. 7.0.0-31 does, sitting in resolute-proposed, expected early September.

Rocky 10 I couldn't settle. I don't see it in the CentOS Stream 10 kernel changelog, which does list per-commit subjects, and Stream is at 6.12.0-260 while @henri9813 is on the 10.2 branch at -211. So probably not yet. But I might just be failing to find it, so if someone can check properly I'd rather be corrected.

@acebmxer @MajorP93 on Debian, and anyone on Ubuntu: keep tsc_mode=2 and nomigrate for now. One thing that won't help you there, the console=ttyS0 removal from the other thread is an Ubuntu cloud image thing. Debian's cloud image recipe only sets a serial console for Azure, EC2 arm64 and ppc64el, so I don't think the generic amd64 image carries it. Worth a look at /proc/cmdline on yours though.

I did check that the fix works rather than assuming it: same VM, Ubuntu 7.0.0-30 against 7.0.0-31, and the sched_clock correction went from about -59s to -0.4s on 6 vCPUs.

Good news, I think this one's finally sorted: the cleanVm: incorrect backup size in metadata warning was fixed in XO 6.4.0 (released end of April) via PR #9637, which makes incremental remote backups record the actual VHD size instead of the slightly-off metadata value.
If you're on 6.4.0 or later, it should be gone, so worth confirming your version and upgrading if you're behind. @manilx, you were waiting on this one.

Take it with a grain of salt, but I think bonds are usually managed as a whole rather than edited port by port in the UI. As far as I can tell, the supported route is from the network section in Xen Orchestra (the bonding part of the infrastructure docs is here: https://docs.xen-orchestra.com/xo5/manage_infrastructure#network-bonding), and on the CLI side, the bond commands are documented at https://docs.xcp-ng.org/appendix/cli_reference#bond-create (there's a matching bond-destroy command alongside it).
My honest guess is you may end up destroying and recreating the bond with the four ports you want to keep, since I'm not sure removing a single member in place is exposed anywhere, but I could easily be wrong.
If there's a cleaner way that avoids the recreate, someone will let us know.