poddingue

poddingue

@MajorP93: Thanks for expressing yourself regarding that, and I'll be transparent about the thinking behind it.

Filing those on GitHub isn't me asking anyone to stop using the forum, quite the opposite in fact.
The forum is where real conversations happen, and that's valuable in a way a GitHub issue never quite is. But forum threads scroll, get buried, and developers can't easily maintain a stable backlog out of them. GitHub gives the team a place where things don't disappear or get buried.

Think of it as belt and suspenders (which I need now that I'm getting old ). The discussion lives here, the tracking lives there. My goal as community manager is to be the relay between the two, so you don't have to worry about it.
File things here, talk about them here, and I'll make sure what matters makes it into the right repo.
Or at least, that's the plan. Mine.

poddingue

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

poddingue

The Kubernetes CSI driver for Xen Orchestra just hit v0.4.0, and we want it on more real clusters before it reaches a stable release candidate. If you run Kubernetes on XCP-ng VMs, this is a good time to give it a proper workout.

What's new in v0.4.0:

Local-storage support
Automatic pool-discovery fallback
Kubernetes metadata now lives in Xen Orchestra VDI tags instead of the deprecated other_config. That change also drops the old requirement for Xen Orchestra 6.4 or newer, so the driver runs on more deployments now.

️ Read this before you upgrade. v0.4.0 is a breaking change. The Kubernetes metadata moved from other_config to VDI tags, so you must migrate before upgrading from v0.3.0. Do not upgrade in place: follow the v0.3.0 to v0.4.0 migration guide in the release notes, then move to v0.4.0.

What helps us most is hearing how it behaves on your own setup: what works, what breaks, which storage backend you use, and which flavour of Kubernetes you run (k3s, full k8s, or something else). Edge cases on real clusters are the ones we don't see in our own testing.

Where to report: start right here in this thread. It keeps everything visible to the community and lets others on the same setup jump in. If something turns out to be a reproducible bug, we'll move it to a GitHub issue on the repo so the team can track it to a fix.

Release notes and migration guide: https://github.com/vatesfr/xenorchestra-csi-driver/releases/tag/v0.4.0

poddingue

From what I understand, when XCP-ng reverts to a snapshot it restores the full VM state from that point (metadata included, not just the disk contents) so the creation date field would get rolled back along with everything else; that might be why it now matches the snapshot timestamp rather than
the original. I might be wrong about the internals though.

It's a bit confusing if you were relying on that field to track VM history, and I don't think https://docs.xen-orchestra.com/xo5/manage_infrastructure#snapshot-management covers this explicitly.
Might be worth a mention to @Team-Documentation-Knowledge-Management; it's the kind of thing that catches people off guard because nothing warns you upfront that metadata rolls back too.
My $0.02.

poddingue

That's excellent news, thanks a lot for the feedback!

poddingue

Reviving this one because it never really got a clean ending. The fix for the Body Timeout Error on full backups went into xapi-project/xen-api#6786 and shipped with the March 2026 8.3 maintenance updates.

@nikade, back in March, you mentioned still hitting it on both Linux and Windows VMs even after updating. Is that still the case for you on current versions? And anyone else who landed here: are you still seeing it, or did the update sort it out?

Mostly trying to work out whether there's still something to chase or whether we can call this resolved. I'm not deep enough in the backup internals to say for sure, so your real-world results would tell us far more than anything I could guess. Thanks!

poddingue

Thanks for following up and opening the issue; that's exactly the kind of report the team needs. I've subscribed to it, so I'll see when it moves. The second-run failure pattern is a useful clue; hopefully they can reproduce it from there.

poddingue

Following up since the situation changed: QCOW2 went GA in XO 6.5 (released 2026-05-28), so the old ~2TB VHD ceiling is gone.
A disk at exactly 2TB, and well beyond it, is fine now without shrinking to 1.99TB first. When acebmxer and john.c replied, it was still a release candidate; it's the stable story now. I haven't migrated a disk quite that size myself, so I won't promise it's totally painless, but the format limit that was blocking you isn't there anymore.
The release blog has the details if you want to read up before the migration: https://xen-orchestra.com/blog/xen-orchestra-6-5/

poddingue

This is great to see, thank you for taking the time to rescue this; and thanks to @john.c for the recovery work and to @tjkreidl for writing it in the first place.
I went looking, and there is a small XCP-ng-specific piece on this in the official docs under NUMA affinity (https://docs.xcp-ng.org/compute#numa-affinity), but it's nothing like the depth of the Tale of Two Servers series, so having the originals archived is genuinely useful.
I won't pretend to judge how much of the 2019 BIOS and GPU-scheduler guidance still maps cleanly onto current hardware and XCP-ng versions; others here will know where it's aged and where it hasn't.
I'll make sure this is on our radar on the docs side, because it keeps coming up. Really appreciate you keeping this from disappearing.

poddingue

Good news, this already shipped in XO 6.5.0 (released 2026-05-28), PR #9835.
If your XOA is on the latest channel, you've got PATCH /vms/{id} now; on the stable channel, it should land when 6.5.0 gets promoted (stable is on 6.4.1 at the moment). name_description looks like the field you'd write that "what's running here" summary to for your n8n flow, though I haven't tried the new endpoint myself yet. Let us know how it goes.

poddingue

steff22, the answer you want is one post above yours.
Yann confirmed at post #4 that AMD iGPU passthrough doesn't work on XCP-ng today and that he's working on it, so your Invalid PCI ROM header signature looks like the known symptom rather than something specific to your board.
He also linked https://xcp-ng.org/forum/topic/8909 , same error string, if you want the longer history. I don't know enough about the ROM BAR side to add anything useful to what he wrote, and I'd rather not guess in front of him.
Worth following this thread, he said he'd keep at it.

poddingue

That timeline is the clearest version of this yet. xen04 going down the same way as xen05 makes the race look real to me rather than coincidental, unless I am reading the ordering wrong.
The fault lands inside drbd_bm_count_bits, so in the DRBD module itself, which makes me think this is an upstream LINBIT question as much as an XCP-ng one. I do not know enough about how that module gets packaged here to say who owns the fix though. Might be worth another mention to @Team-Storage so someone can open your bundles with your post #10 next to them.
On the padding idea I am out of my depth: no idea whether LINSTOR can be told to pre-round a volume at creation, but asking LINBIT directly seems worth a try since it would sidestep drbd_bm_resize entirely.

poddingue

There is a public alliance, announced back in April: Vates joined Everpure's Technology Alliance Program, and Everpure came in as an associate member of the Vates Alliance Network (https://vates.tech/blog/vates-joins-the-everpure-technology-alliance-program/).
What that post actually commits to is validated reference architectures and compatibility documentation, with backup, disaster recovery and automation workflows named as planned expansion, so the area you're asking about is at least in scope.
Whether any of that becomes an equivalent of the VMware pure-plugin, with per-VM restore from SafeMode snapshots, I genuinely don't know, and I'd rather not guess while you're deciding whether to spend weeks on it.
Before you go far it might be worth a mention to @Team-Storage, since they'd know whether the array-side work is heading that way.
Selfishly I'd rather you asked first than quietly built something that ends up overlapping. I'm not close enough to the storage roadmap to give you a straight yes or no.

poddingue

On the bundle, the docs warn it can carry sensitive details about your setup, so better not to attach it publicly: https://docs.xcp-ng.org/troubleshooting/log-files#produce-a-status-report. As far as I can tell there is no self-serve upload spot, it is usually someone from Vates dropping you a private Nextcloud link when you ask, so flagging here that you have the bundle ready is probably the quickest route.

Since this was a kernel panic, is there anything in /var/crash on the host that went down, and which host was it this time? The earlier ones were xen04 and xen03.

poddingue

@john.c your PR landed on the 15th and the live page now reads "XCP-ng 8.3 LTS", so the table matches the advice sitting right above it again: https://github.com/vatesfr/xen-orchestra/pull/10092.
You spotted it and then wrote the fix yourself, which is the part that saved the next person the trouble.

poddingue

Thanks, that complicates the large-VM theory in a good way.
If these are long-established VMs that backed up fine for years, "too much free space on a new VM" probably isn't the whole story here.
Your deltas run from a completely different XO instance, on a different host, to different remotes, at non-overlapping times, so "deltas never fail, fulls do" might not be purely about job type, it could be tangled up with which instance or remote is doing the work.
If you ever get the chance to run a full from the XO instance that normally handles your deltas, that would help tell whether the timeout follows the job type or the instance.
I could easily be wrong, though, but that split feels worth isolating before we lean too hard on the free-space angle in https://github.com/vatesfr/xen-orchestra/issues/9181 .

poddingue

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes.
Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 926 log lines around the panic suggests dom0 was under real pressure right then.
If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side.
I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful.
Fingers crossed it stays boring from here.

poddingue

I converted the topic to a question, then marked it solved.
Thanks!

poddingue

Quick one to close this off: florent's fix for the encrypted-remote mirror alignment landed in a numbered release now, XO 6.6.2 (2026-07-09).
The PR was https://github.com/vatesfr/xen-orchestra/pull/10061, so it's out of master and into something you can just update to.
Did your retest come good on it?
I'm also curious whether you still need the minPartSize=100000000 setting with the fix in place, or whether that's redundant now, since it'd help us work out whether it's worth writing into the B2 docs.

poddingue

The timing evidence here looks strong to me. Your second Grafana window is doing a lot of work: linstor-satellite.service jumps to roughly 8,700 log lines in the three minutes around the panic, well above linstor-controller at 926 and xapi at 460, and the priority chart goes red at the same moment. A backup that fans out volume creates and deletes across three replicas, landing on a DRBD race, fits what you are seeing.

Throttling Velero should tell you a lot. If the crashes stop with clientQPS: 3 and itemBlockWorkerCount: 1, that narrows it to concurrency rather than anything about those particular volumes.

For the logs question, a xen-bugtool --yestoall bundle from a host that has panicked is usually the thing people ask for first (https://docs.xcp-ng.org/troubleshooting/log-files), since it sweeps up the kernel side alongside the storage logs you already have in Loki. I don't know DRBD internals well enough to say which trace matters most, so it might be worth a mention to @Team-Storage. They can say what they actually need rather than have you guess, and a panic that reproduces on a schedule is a good deal easier for them to chase than most.

poddingue

@poddingue