poddingue

poddingue

@MajorP93: Thanks for expressing yourself regarding that, and I'll be transparent about the thinking behind it.

Filing those on GitHub isn't me asking anyone to stop using the forum, quite the opposite in fact.
The forum is where real conversations happen, and that's valuable in a way a GitHub issue never quite is. But forum threads scroll, get buried, and developers can't easily maintain a stable backlog out of them. GitHub gives the team a place where things don't disappear or get buried.

Think of it as belt and suspenders (which I need now that I'm getting old ). The discussion lives here, the tracking lives there. My goal as community manager is to be the relay between the two, so you don't have to worry about it.
File things here, talk about them here, and I'll make sure what matters makes it into the right repo.
Or at least, that's the plan. Mine.

poddingue

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

poddingue

The Kubernetes CSI driver for Xen Orchestra just hit v0.4.0, and we want it on more real clusters before it reaches a stable release candidate. If you run Kubernetes on XCP-ng VMs, this is a good time to give it a proper workout.

What's new in v0.4.0:

Local-storage support
Automatic pool-discovery fallback
Kubernetes metadata now lives in Xen Orchestra VDI tags instead of the deprecated other_config. That change also drops the old requirement for Xen Orchestra 6.4 or newer, so the driver runs on more deployments now.

️ Read this before you upgrade. v0.4.0 is a breaking change. The Kubernetes metadata moved from other_config to VDI tags, so you must migrate before upgrading from v0.3.0. Do not upgrade in place: follow the v0.3.0 to v0.4.0 migration guide in the release notes, then move to v0.4.0.

What helps us most is hearing how it behaves on your own setup: what works, what breaks, which storage backend you use, and which flavour of Kubernetes you run (k3s, full k8s, or something else). Edge cases on real clusters are the ones we don't see in our own testing.

Where to report: start right here in this thread. It keeps everything visible to the community and lets others on the same setup jump in. If something turns out to be a reproducible bug, we'll move it to a GitHub issue on the repo so the team can track it to a fix.

Release notes and migration guide: https://github.com/vatesfr/xenorchestra-csi-driver/releases/tag/v0.4.0

poddingue

It was too tempting not to test, so I went and tested the other half.
On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it.
The clone came up at 800x600.
A control clone of the same original, with nothing set, came up at 1024x768.
So your workaround holds, the resolution really does ride along into VMs built from the template.

For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record.

One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024.
So you may not need to raise videoram for the common ones.

Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it.
At least, that's my understanding.

poddingue

Thanks for coming back with the actual cause instead of leaving it hanging.
To me, a permissions reset on /etc/rc.d/rc.local would explain it completely, and it fails in the worst way, since nothing complains at boot and the VMs just never start.
It also looks worth documenting, because our own troubleshooting page tells people to append a clocksource workaround to /etc/rc.local without mentioning the executable bit either, so anyone following that could end up with a workaround that quietly does nothing.
I haven't tested whether the 8.2 to 8.3 upgrade resets that bit every time or just happened to in your case, so that half stays your finding, not mine.

I'm curious what you turn up on the colon versus equals behaviour.

poddingue

Hey, thanks for writing this up with the actual commands and the screenshot, that really helps.

Not sure why rc.local + vApps stopped working after your 8.2 to 8.3 jump. rc.local is more your own dom0-level script than something XCP-ng promises to keep stable across releases, so something underneath may well have shifted, but I can't point to what.

On the delay question, there does seem to be a per-VM "start-delay" parameter. I ran into it replying to a pretty similar issue a couple days ago, if you want to compare notes: https://xcp-ng.org/forum/topic/12388. There are also the "appliance" CLI commands, the vApp equivalent, though from another recent thread it looked like nobody's fully sure how start order and delay actually interact there: https://xcp-ng.org/forum/topic/12149. So I think the capability is there, just not pulled together in one place.

The colon-vs-equals thing is interesting. I honestly don't know xe's map-param parsing well enough to say if that's expected or a rough edge. Maybe someone closer to the CLI internals can say whether it should just reject that instead of creating duplicate entries. Given how scattered this all is, might be worth flagging to @Team-Documentation-Knowledge-Management.

Hope some of that's useful, others probably know xe's internals better than me though.

poddingue

Two PRs out of the lowest priority bug ever, that made my morning.

poddingue

From what I understand, when XCP-ng reverts to a snapshot it restores the full VM state from that point (metadata included, not just the disk contents) so the creation date field would get rolled back along with everything else; that might be why it now matches the snapshot timestamp rather than
the original. I might be wrong about the internals though.

It's a bit confusing if you were relying on that field to track VM history, and I don't think https://docs.xen-orchestra.com/xo5/manage_infrastructure#snapshot-management covers this explicitly.
Might be worth a mention to @Team-Documentation-Knowledge-Management; it's the kind of thing that catches people off guard because nothing warns you upfront that metadata rolls back too.
My $0.02.

poddingue

Thanks for the feedback so far, folks!

poddingue

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.

poddingue

Nice, thanks for the feeder entry and the explanation, @CAPS!

poddingue

Thanks!

poddingue

@gduperrey pointed you here, so paste the two screenshots and the description into this thread too, otherwise whoever picks it up lands on a bare link.

In your tasks screenshot the Rolling pool update row on PRD is dated 2026-08-10, 27 minutes, progress bar never filled, which looks more like an older failed run than the one you'd just finished.
Was that the entry you deleted, or was there a second one from today as well?

I don't know how XO decides when a failed task leaves that list, so I could easily be reading your screenshot wrong.

Might be worth a mention to Team-XO-Backend once the details are here, and your XOA version would help them too.

poddingue

Thanks for chasing this with them, and for offering to post back after the call.

Your Alma 10 read looks right (the 9.0 demonstrator is Xen 4.19 with a dom0 loosely based on AlmaLinux 10 and a 6.12 kernel), but @yann said in July they're out of prototyping and rebuilding the foundations component by component, so I wouldn't treat that kernel as the one their driver ends up targeting.
On the packaging side, the kernel module policy page is the nearest thing I've found to an answer for what you'd need to pull together: https://docs.xcp-ng.org/project/development-process/kernel-module-policy

For VROC, my understanding is that it sits on VMD, though I could have that backwards. VMD stalled boot on the demonstrator kernel back in January until yann said he'd include a patch for it, so that stretch of the thread is worth a read before your call: https://xcp-ng.org/forum/topic/11698

I don't know enough about driver packaging to say what Vates would actually need from GRAID, so it might be worth pulling in @Team-Hypervisor-Kernel, since drivers are their scope.
If the call gives you a list of what they need from our side, that's probably the most useful thing to bring back here.

poddingue

The same assumption was already fixed once on the dashboard side: https://github.com/vatesfr/xen-orchestra/pull/9997 counts sequence schedules in the backup computation, merged back in June.
So the home filter looks like that same fix never made it across, which is a better argument than "it would be nice".

On your second point, XO 6 may already have part of it: there's a Backed up VMs tab that lists from what's actually on the remotes rather than from the job config, which is the "like the restore page does" bit you were after, and https://github.com/vatesfr/xen-orchestra/pull/10277 is an open fix for a caching bug in it.

I haven't used that tab myself, and since it's built around VMs that do have backups, I don't know if it gives you the negative view you want. If it doesn't, https://feedback.vates.tech is where that gets counted.

poddingue

It was too tempting not to test, so I went and tested the other half.
On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it.
The clone came up at 800x600.
A control clone of the same original, with nothing set, came up at 1024x768.
So your workaround holds, the resolution really does ride along into VMs built from the template.

For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record.

One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024.
So you may not need to raise videoram for the common ones.

Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it.
At least, that's my understanding.

poddingue

I think I may have found why, though someone on the XO side should check me on this.

The filter builds its list from backup jobs that have at least one enabled schedule of their own, right?
A sequence works the other way round: it gets its own timer and runs the member schedules from there.

So once those member schedules are switched off, their jobs drop out of the filter and every VM they cover reads as not backed up.
If that's right, it's a wrong answer rather than a missing feature, which matters because it changes who needs to look at it.

I got there by reading the code with bad glasses, not by reproducing it on a real install, so treat that last step as my wild guess.

Might be worth a mention to @Team-XO-Backend.
Your dashboard idea could well be the better home for it anyway.

poddingue

To me, the template idea does hold up, at least the part I could check.

On an 8.3 host I cloned a UEFI VM, then turned that clone into a template and cloned it again.
The NVRAM field came back byte-identical both times, so whatever sits in a VM's UEFI variables really does travel into VMs built from it.

What I didn't test is the preferred-resolution setting itself, so I can't promise the resolution survives and applies, only that the place it lives in does.
So it's half an answer.

The "would Vates do this" part isn't mine to answer, and https://feedback.vates.tech is where that gets counted rather than a forum thread.

Worth saying there that https://docs.xcp-ng.org/vms/#manage-screen-resolution only covers the guest-OS side, so the firmware route you wrote up isn't documented anywhere official.

You can say so by clicking at the bottom of the page on :

Was this page helpful?
Yes/No

Or even click on:

Edit this page

poddingue

Thanks a lot for this feedback, @pedro!

poddingue

On the XO Lite one, there's an issue open that sounds like what you're describing, https://github.com/vatesfr/xen-orchestra/issues/9801, filed off @Greg_E's report in this thread.
It says UEFI VMs created through XO Lite don't boot since the LTS release, with Debian 13 and Windows Server 2022 both failing, while VMs built under 8.2 with UEFI still work.

Your timing may not line up, since you're saying last year and that issue is scoped to since the LTS release, so I'd rather ask than assume.
If it does look like yours, a comment on it would carry more than anything I can add, because it hasn't had one since May and I don't think anyone has picked it up.

poddingue

@poddingue