poddingue

@MajorP93: Thanks for expressing yourself regarding that, and I'll be transparent about the thinking behind it.

Filing those on GitHub isn't me asking anyone to stop using the forum, quite the opposite in fact.
The forum is where real conversations happen, and that's valuable in a way a GitHub issue never quite is. But forum threads scroll, get buried, and developers can't easily maintain a stable backlog out of them. GitHub gives the team a place where things don't disappear or get buried.

Think of it as belt and suspenders (which I need now that I'm getting old ). The discussion lives here, the tracking lives there. My goal as community manager is to be the relay between the two, so you don't have to worry about it.
File things here, talk about them here, and I'll make sure what matters makes it into the right repo.
Or at least, that's the plan. Mine.

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

The Kubernetes CSI driver for Xen Orchestra just hit v0.4.0, and we want it on more real clusters before it reaches a stable release candidate. If you run Kubernetes on XCP-ng VMs, this is a good time to give it a proper workout.

What's new in v0.4.0:

• Local-storage support
• Automatic pool-discovery fallback
• Kubernetes metadata now lives in Xen Orchestra VDI tags instead of the deprecated other_config. That change also drops the old requirement for Xen Orchestra 6.4 or newer, so the driver runs on more deployments now.

️ Read this before you upgrade. v0.4.0 is a breaking change. The Kubernetes metadata moved from other_config to VDI tags, so you must migrate before upgrading from v0.3.0. Do not upgrade in place: follow the v0.3.0 to v0.4.0 migration guide in the release notes, then move to v0.4.0.

What helps us most is hearing how it behaves on your own setup: what works, what breaks, which storage backend you use, and which flavour of Kubernetes you run (k3s, full k8s, or something else). Edge cases on real clusters are the ones we don't see in our own testing.

Where to report: start right here in this thread. It keeps everything visible to the community and lets others on the same setup jump in. If something turns out to be a reproducible bug, we'll move it to a GitHub issue on the repo so the team can track it to a fix.

Release notes and migration guide: https://github.com/vatesfr/xenorchestra-csi-driver/releases/tag/v0.4.0

It was too tempting not to test, so I went and tested the other half.
On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it.
The clone came up at 800x600.
A control clone of the same original, with nothing set, came up at 1024x768.
So your workaround holds, the resolution really does ride along into VMs built from the template.

For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record.

One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024.
So you may not need to raise videoram for the common ones.

Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it.
At least, that's my understanding.

Thanks for coming back with the actual cause instead of leaving it hanging.
To me, a permissions reset on /etc/rc.d/rc.local would explain it completely, and it fails in the worst way, since nothing complains at boot and the VMs just never start.
It also looks worth documenting, because our own troubleshooting page tells people to append a clocksource workaround to /etc/rc.local without mentioning the executable bit either, so anyone following that could end up with a workaround that quietly does nothing.
I haven't tested whether the 8.2 to 8.3 upgrade resets that bit every time or just happened to in your case, so that half stays your finding, not mine.

I'm curious what you turn up on the colon versus equals behaviour.

Hey, thanks for writing this up with the actual commands and the screenshot, that really helps.

Not sure why rc.local + vApps stopped working after your 8.2 to 8.3 jump. rc.local is more your own dom0-level script than something XCP-ng promises to keep stable across releases, so something underneath may well have shifted, but I can't point to what.

On the delay question, there does seem to be a per-VM "start-delay" parameter. I ran into it replying to a pretty similar issue a couple days ago, if you want to compare notes: https://xcp-ng.org/forum/topic/12388. There are also the "appliance" CLI commands, the vApp equivalent, though from another recent thread it looked like nobody's fully sure how start order and delay actually interact there: https://xcp-ng.org/forum/topic/12149. So I think the capability is there, just not pulled together in one place.

The colon-vs-equals thing is interesting. I honestly don't know xe's map-param parsing well enough to say if that's expected or a rough edge. Maybe someone closer to the CLI internals can say whether it should just reject that instead of creating duplicate entries. Given how scattered this all is, might be worth flagging to @Team-Documentation-Knowledge-Management.

Hope some of that's useful, others probably know xe's internals better than me though.

Two PRs out of the lowest priority bug ever, that made my morning.

From what I understand, when XCP-ng reverts to a snapshot it restores the full VM state from that point (metadata included, not just the disk contents) so the creation date field would get rolled back along with everything else; that might be why it now matches the snapshot timestamp rather than
the original. I might be wrong about the internals though.

It's a bit confusing if you were relying on that field to track VM history, and I don't think https://docs.xen-orchestra.com/xo5/manage_infrastructure#snapshot-management covers this explicitly.
Might be worth a mention to @Team-Documentation-Knowledge-Management; it's the kind of thing that catches people off guard because nothing warns you upfront that metadata rolls back too.
My $0.02.

Thanks for the feedback so far, folks!

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.