From the log you attached, the failure is on session.login_with_password, and XO tries ::1:443 and then 127.0.0.1:443.
So it is XO trying to reach a host, not your remote, and whatever address it holds for that host is resolving to loopback.
You rebuilt one of the three nodes and you moved XO into Docker, so my guess is the host came back into the pool with a hostname instead of an IP, and that name does not resolve inside the container the way it did for the script install.
It would fit the rest too: migration goes host to host over the management address, copying does not, which is why one works and the other doesn't.

Could you paste the output of xe host-list params=uuid,name-label,address? I am guessing here, and @Team-XAPI-Network will know much better than me whether resetting that address is the right move.

I don't think "Protect from accidental shutdown" is what you are hitting here.
If any VM on that host has a host-bound device attached (PCI passthrough, vGPU, or an SR-IOV VIF), XAPI refuses to suspend it with VM_HAS_PCI_ATTACHED, and Smart Reboot stops there. xe host-get-vms-which-prevent-evacuation uuid=<host-uuid> should name them.
For the Rolling Pool Update half, XO 6.7.0 (released 30 July) added a shutdownPinnedVms option that shuts those VMs down and starts them again afterwards instead of aborting the run, so upgrading might get you past it.
I could easily be wrong about which of the two you are actually hitting, though, so I would start with that command and see what comes back.

Hi RN0, and thanks for photographing the panic on your first post here, because that is what made it findable.
The last line reads acpi_device_hid, which looks like the same crash someone else hit on an MS-A2 last year: https://xcp-ng.org/forum/topic/10972/xcp-ng-8-3-lts-install-on-minisforum-ms-a2-7945hx .

What got them past the installer was a BIOS change rather than a boot flag: disabling Global C-State Control under Advanced > AMD CBS > CPU Common Options worked for one of them, and another got through by turning off PSS Support instead.
That might be why nomodeset and safe mode made no difference for you, since neither touches ACPI.
There is a kernel fix for it now too, and once installed and updated they put the BIOS setting back to Auto and it kept working. So I do not think this is an HCL or driver problem, though I am not deep enough in ACPI to promise that, and someone who knows the kernel side better will correct me if I have misread your screenshot.

The PoD crash half is known: https://xcp-ng.org/forum/topic/10179/rockylinux-vm-s-random-reboots hit the same p2m_pod_demand_populate line, and the answer there was closing the gap between static-max and dynamic-max, with the background at https://xenproject.org/blog/ballooning-rebooting-and-the-feature-youve-never-heard-of/ .
The guest tools angle I have not seen reported before, and the 13-second repro makes it hard to wave off. What gives me pause is https://xcp-ng.org/forum/topic/11955/memory-ballooning-dmc-broken-since-xcp-ng-8.3-january-2026-patches, where the Vates Rust guest agent stopped re-arming ballooning after live migration and the Citrix utilities were the ones behaving.
Different guest OS and probably a different mechanism, so it may well be unrelated, but it makes me wary of reading your result as "Citrix tools do not reclaim" rather than the narrower "on Windows under PoD pressure".
I could not find an existing issue for either of your XO asks, so both look worth filing at vatesfr/xen-orchestra, and they stand on their own regardless of where the tools question lands.

Whether the XO 6 creation form already treats the memory side differently, I do not know, and I would rather say that than guess.

That is clearer than I expected, thank you.
The part that lands for me is that the cap sits on the agent rather than on the token, so a delete is not refused after the fact, it simply never appears in the catalogue the model can see.
I had assumed most of the work would live in a policy engine, and it sounds like the bulk of it is classifying the 267 tools once and shipping that classification with the description itself.
Treating a refused call as a signal rather than a near miss is the bit I want to borrow if/when we talk about this, because it turns the log from an audit trail into something closer to a smoke alarm.
I am not deep enough in MCP to have a useful opinion on where this model breaks, so I will read the endpoint matrix properly before asking anything else.
Passing it on internally as promised.

That is a useful pair of findings, thanks for going back and testing both.
So the appliance was never the blocker, it just needs the Windows console next to it to add the NFS repository and connect XCP-ng, which fits what @AtaxyaNetwork saw too.
The part I would not have guessed is that upgrading an existing console pulls the plug-in in by itself, while running the standalone plugin installer against a fresh 13.1 offers to uninstall it instead, so anyone coming from 12.x probably wants to look at the console component list rather than hunt for a separate download.

Your screenshot showing Veeam Plug-In for Xen UI Extension 13.1.0.295 sitting next to the Nutanix, oVirt and Proxmox extensions is a good confirmation it is a proper component now and not a side channel.

I have not run 13.1 myself so I am going entirely on your screenshots here. If you get as far as a first successful backup run, it would be great to hear how that went.

Merci pour les précisions, elles écartent ma piste. Si vous reproduisez en BIOS comme en UEFI, le ticket #151 ne colle pas, d'autant qu'il parle de l'installeur Windows et pas d'Anaconda : je l'avais résumé un peu vite.
La différence intéressante est peut-être RHEL 9 contre RHEL 10, puisque @AtaxyaNetwork a un build qui fonctionne en 10 avec cd_files et sans DHCP, et son test en 9 ce week-end devrait trancher.
Sur cd_label = "OEMDRV", je n'ai trouvé ni issue ni PR dans le dépôt du plugin, donc ça n'existe visiblement pas encore : si ça vous va, ça ferait une demande propre à ouvrir sur https://github.com/vatesfr/packer-plugin-xenserver, en reprenant l'argument air-gapped que vous détaillez plus haut, qui est plus convaincant que ce que j'écrirais moi.

Je ne connais pas assez le plugin pour dire si c'est simple à ajouter, donc je préfère ne rien promettre.
Merci en tout cas de documenter tout précisément, ça servira sans doute bien au-delà de votre POC.

Trying to pull the last few posts together, because there is a pattern I keep staring at.
Every failed run in @JB's screenshots lasts exactly 5 minutes while the successful ones finish in seconds, and @abudef's history going back to 23 June shows the same 5-minute wall.
That reads to me like a fixed timeout firing rather than a merely slow transfer, though I could be wrong about which one.
@christopher-petzel's syslog looks like a different animal, since assert(!this.paused) in undici's Parser.finish is xo-server crashing outright at job start while 13 other instances backed up fine the same minute. The only knob I can find written down is httpInactivityTimeout at https://docs.xen-orchestra.com/xo5/backup_troubleshooting#error-http-connection-has-timed-out, but that one sits on the XAPI GET path, and I do not know whether it touches pool metadata jobs at all. @florent, does the 5-minute signature change anything on your side?

My earlier wording was the problem here, not your reading of it.

There are two official descriptions and they do not say the same thing. The CLI reference calls start-delay "the delay to wait before a call to start up the VM returns", which is the one I quoted. The XenAPI field description calls it "the delay to wait before proceeding to the next order in the startup sequence". That second one is exactly what you saw: the delay lands on the next VM to start, not on the one you set it on. I quoted the confusing one and then said it explained your behaviour, which it does not really.

What I still cannot explain is why a start-delay would stop a VM autostarting altogether. Neither wording predicts that, so I would rather say I do not know than invent a reason.

On auto_poweron_delay, I went looking and could not find it anywhere in the XCP-ng or XO docs, so I think your instinct about that search result was right.

I have updated the docs PR I had open so it leads with the XenAPI wording and cites both, since the CLI reference phrasing is what sent me wrong in the first place.

Bonjour, et bienvenue dans la communauté.
Je suis content (même si je n'y suis pour rien) que les tutos de @bvivi57 et @ataxyanetwork vous aient fait gagner du temps.

A priori ce n'est ni normal ni voulu : le support de cd_files a été ajouté par la PR #144 (https://github.com/vatesfr/packer-plugin-xenserver/pull/144), mergée fin juin 2025, donc présent depuis la v0.8.1 du plugin, et la dernière version publiée est la v0.11.4. Ce qui me met plutôt sur une autre piste, d'autant que floppy_files échoue aussi chez vous : le ticket #151 (https://github.com/vatesfr/packer-plugin-xenserver/issues/151) décrit exactement ce symptôme, une image floppy que l'installeur ne voit pas en UEFI alors qu'elle passe sans souci en BIOS legacy.

Vos builds RHEL démarrent-ils en UEFI, et avec quelle version du plugin ? Je ne connais pas assez le plugin pour en être certain, donc corrigez-moi si je me trompe.

Thanks for coming back with the actual cause instead of leaving it hanging.
To me, a permissions reset on /etc/rc.d/rc.local would explain it completely, and it fails in the worst way, since nothing complains at boot and the VMs just never start.
It also looks worth documenting, because our own troubleshooting page tells people to append a clocksource workaround to /etc/rc.local without mentioning the executable bit either, so anyone following that could end up with a workaround that quietly does nothing.
I haven't tested whether the 8.2 to 8.3 upgrade resets that bit every time or just happened to in your case, so that half stays your finding, not mine.

I'm curious what you turn up on the colon versus equals behaviour.

Nested RDP eating the arrow keys is not something I'd have guessed, and @AtaxyaNetwork's post probably explains why I went the wrong way too.
If the default selection is reinstall rather than install, that fits your log better than my disk-naming guess did, since the thing that aborted was ReinstallLayoutGenerator looking for an existing SYSTEM volume a fresh disk wouldn't have.
So the LVM error was probably the installer doing the right thing in the wrong mode, and my xvda theory was, at best, misleading.
I'm less sure about the other part, that the appliance doesn't support XCP-ng while B&R on Windows does, since AtaxyaNetwork seems to have the appliance running.
Where did that one come from, the Veeam docs or your support case?

Worth pinning down before it gets repeated.

That's a big jump, XAPI 24.19.2 to 26.1.11.
I think the guard is deliberate rather than a bug: a metadata restore overwrites the whole XAPI database, so XAPI refuses when the builds don't line up.
The docs at https://docs.xen-orchestra.com/xo5/metadata_backup#performing-a-restore warn about the overwrite but say nothing about version compatibility, which feels like a gap on our side.
I don't know if there's a supported way to replay an older metadata backup onto a newer build, and I wouldn't want to guess at hand-editing it.
Might be worth a mention to @Team-XAPI-Network, they can say whether a staged path exists, something like installing the matching older build, restoring, then updating.

Correct me if someone here has actually pulled this off.

Thanks for the screenshot, that's a lot more useful than just telling us it failed. Reading it, I'm not sure it's failing to boot at all: anaconda starts fine, then Veeam's own pre-installation disk layout script gives up with "Could not find data volume group in the system LVM spec" and never writes the /tmp/pre-storage.cfg it then tries to read.
That would at least explain why Generic UEFI and Rocky 9 behave the same way. Your screen says there's a shell on TTY2, and if you run lsblk there I'd be curious whether the disks show up as xvda and xvdb rather than sda and sdb, which is how they usually turn up under XCP-ng.
That's a wild guess and I don't know their installer at all, but it might be worth putting in the Veeam ticket either way.

Congrats on getting this out so fast after 6.7, and thanks for the writeup, genuinely interesting approach. The declarative angle instead of wrapper code per endpoint makes sense to me, especially given how often the REST surface grows.

One thing I keep wondering about: you framed the auth/audit/fail-closed checks as sitting in front of the gateway. What happens when the model actually asks for something destructive, a VDI delete, a host disable, that kind of thing? Does the gateway block it outright, or does a human need to sign off somewhere in there?

I'm asking because that's close to the exact boundary question our own Insights piece touched on this month. I'm community manager here, not a developer, so I'd rather ask than guess at how the security model actually behaves. Want to understand it properly before I say more.

Passing this along internally too. No promises on timing, but it's the kind of thing worth a closer look.

This looks like it might be the snapshot_of corruption XCP-ng docs describe: https://docs.xcp-ng.org/troubleshooting/xapi/snapshot-of. Interrupted operations can apparently leave that field pointing at the wrong thing, which is what makes a real disk get read as a snapshot. The docs say a fix is being worked on across several storage backends but don't mention XOSTOR by name, so I can't promise this is the same bug rather than something specific to the LINSTOR side.

There's a snapshot-fixer.py script mentioned there with a dry-run mode, it shouldn't change anything, just tell you if it sees the same corruption. Worth trying that first before anything else.

I'm not deep enough in XAPI or LINSTOR internals to know if XOSTOR is covered by that fix or not. If dry-run comes back clean and this really does look XOSTOR-specific, might be worth a mention to @Team-Storage, since production VMs are affected here.

I think this is fixed in 6.7: when a delta backup falls back to a full, the task now shows the actual reason instead of just contradicting itself the way you and Andrew both saw (PR #10148).

Worth checking after you upgrade.

Both things reported here look fixed in 6.7: the retry succeeding but the job still showing as failed (PR #10129), and the EEXIST: file already exists error, which happened when a job got recreated after a crash (PR #10139).

Curious if upgrading clears it for you all too.

The progress bar issue you and Florent were chasing looks fixed in 6.7 (PR #10133). Worth upgrading and seeing if it moves now instead of sitting still.

Separately, since this thread is specifically about QCOW2 backups: 6.7 also fixes an offset bug that could leave incremental backups mostly empty after the first block when NBD isn't enabled. Worth a look at the release notes if that's your setup too.

Hey, thanks for writing this up with the actual commands and the screenshot, that really helps.

Not sure why rc.local + vApps stopped working after your 8.2 to 8.3 jump. rc.local is more your own dom0-level script than something XCP-ng promises to keep stable across releases, so something underneath may well have shifted, but I can't point to what.

On the delay question, there does seem to be a per-VM "start-delay" parameter. I ran into it replying to a pretty similar issue a couple days ago, if you want to compare notes: https://xcp-ng.org/forum/topic/12388. There are also the "appliance" CLI commands, the vApp equivalent, though from another recent thread it looked like nobody's fully sure how start order and delay actually interact there: https://xcp-ng.org/forum/topic/12149. So I think the capability is there, just not pulled together in one place.

The colon-vs-equals thing is interesting. I honestly don't know xe's map-param parsing well enough to say if that's expected or a rough edge. Maybe someone closer to the CLI internals can say whether it should just reject that instead of creating duplicate entries. Given how scattered this all is, might be worth flagging to @Team-Documentation-Knowledge-Management.

Hope some of that's useful, others probably know xe's internals better than me though.