@john.c I've put this on the feedback portal as a feature request, so it doesn't get buried in a thread that started out about pool masters: https://feedback.vates.tech/posts/121/publish-build-date-and-package-manifest-sbom-with-the-xoa-xva-download .

As far as I know votes there help the team prioritise, so if you and @Forza want it, that's the place to say so.

There's an open PR adding SBOM generation (https://github.com/vatesfr/xen-orchestra/pull/10368), but going by the files it touches it covers XO's own JavaScript dependencies, not the Debian packages in the appliance.

I've only looked at the file list, not the code, so happy to be corrected. How often the downloadable XVA gets rebuilt, I still don't know: @Danp said back in 2024 that it isn't rebuilt for every XOA release (https://xcp-ng.org/forum/post/81220), which would fit the 2025.12 tag, but I'd rather not guess beyond that.

Glad the ticket sorted it, @jacob.becker, and thanks for coming back to say so.

I don't know whether a patch for the stable channel is planned before 6.9 moves there. If someone else on stable lands here with the same Body Timeout Error, opening a ticket like you did seems to be the quickest route for now.

Thanks for opening https://github.com/xcp-ng/xcp/issues/851.

I checked an 8.3 lab host with every update applied and it's the same there: tzdata-2024a-1.el7, and nothing newer in the updates or testing repositories. zdump -v America/Winnipeg on that host still has Manitoba falling back to CST on November 1st, so a host set to that zone would show local times an hour off from then on (UTC itself isn't affected).

I don't know how the team usually handles tzdata on the CentOS 7 based dom0, so a mention to @Team-OS-Platform-Release might help get #851 looked at before the 31st.

Thanks for coming back with that. Commit 8c2f3 is PR #10518 (https://github.com/vatesfr/xen-orchestra/pull/10518), and the changelog lists it under XO 6.9.1 from 5 October, so XOA users get it with 6.9.1, on the latest channel for now if I read the release channels right.

The 6.9 release post has it as "Fixed: BodyTimeoutError during long transfers" (https://xen-orchestra.com/blog/xen-orchestra-6-9/), thanks @john.c.
It's in the changelog too, I just didn't recognise it: 6.9.0 lists PR #10282 as the fix for assert(!this.paused), and that PR is the undici update behind the BodyTimeoutError fix.

@jacob.becker it's only on the latest channel for now, and I'd go straight to 6.9.1, which fixes servers stuck in Connecting behind an HTTP/2 proxy (#10507), a side effect of that same update. The post talks about backup jobs in general and I'm not sure it covers the metadata job the same way, so it'd be great to hear whether it still times out after the update.

I tried this on my lab (with XO 6.9.1), moving the pool master to the other host with xe pool-designate-new-master and then back. XO stayed connected both times and picked up the new master, even though Settings > Servers still showed the old host's address; on the way back it took a minute or two to catch up.

I didn't test 6.2.2, and I used the CLI rather than the XO 5 Advanced tab, so my guess that your older version is the difference is only a guess.

For the XVA, I'm not aware of release notes for the appliance image itself: it updates itself after import, and the notes that exist are per XO version, in the monthly blog post and the changelog, both linked from https://docs.xen-orchestra.com/getting-started/releases#release-cadence-and-channels .

How often the downloadable XVA gets rebuilt I don't know, so the 2025.12 tag may just be the date of the base image, but I'd rather ask than guess.

The docs have a section for this symptom, Windows Server 2025 hanging at 0% CPU: https://docs.xcp-ng.org/troubleshooting/windows-pv-tools#windows-server-2025-hangs-randomly-with-0-cpu .

Besides the Viridian flags you've already set, it says to revert useplatformclock and useplatformtick with bcdedit if they were ever set in the guest.

Your earlier thread showed a VM created from the "Other install media" template, so it's worth checking which template this one came from, since the Windows templates bring their own Viridian settings.
I don't think the QEMU SIGTERM is the cause, though: when I force-shut down a VM on a lab host, qemu-dm logged that exact terminating on signal 15 line, sent by xenopsd-xc (ps -o comm -p 1526469 will tell you on yours), so yours most likely just matches your forced shutdown around 10:20 on 5 Oct.
If it freezes again after that, the memory dump procedure further down the same page (https://docs.xcp-ng.org/troubleshooting/windows-pv-tools#how-to-gather-kernel-memory-dumps-for-in-depth-troubleshooting) is likely what people who know Windows guests better than me would want to see.

Welcome to the forum, and nice to hear the DL380 G6/G7s are still earning their keep in a classroom.

I'm not the expert here (far from that), but @teddyastie from the team answered something close to this last year: XCP-ng 9 will need an x86_64-v2 CPU, which he summed up as Intel Nehalem or newer (https://xcp-ng.org/forum/post/98416).

If I've got the generations right (and that's a real if ), the G6 and G7 carry Xeon 5500 and 5600 chips, Nehalem and Westmere, so the CPU level shouldn't be what stops you, which fits john.c's point about Alma having a v2 build.

Like @JeffBerntsen says, boot mode is the bigger question: Teddy's post says UEFI will be required, "or at least the only officially supported option", and Jeff runs G7s himself and confirms they're BIOS-only. So the honest answer is that I don't know yet whether legacy boot will still work on 9, even unofficially. Either way, 8.3 is the LTS release and it's supported until 2028-11-30 (https://docs.xcp-ng.org/releases/release-8-3), so there's no rush for your lab.

There's a similar report on 6.8 in https://xcp-ng.org/forum/topic/12453, where @flakpyro said it got fixed through a support ticket and that @florent would know the exact fix.

I read through the 6.9.0 changelog, but I couldn't find an entry about metadata backups or Body Timeout Error, so I can't confirm that moving to the latest channel fixes it, though it may have gone in without a changelog line.

If you have a support contract, a ticket that points at that thread might be the quickest way to the patch Florent made.

This came up in https://xcp-ng.org/forum/topic/11728 last winter, with the same kind of setup, an ISO SR served by a VM on the host.

Olivier's answer there was to eject the CDs from the VMs, unplug the ISO SR's PBD, and only then shut the host down, using xe vm-cd-eject --multiple and xe pbd-unplug uuid=<PBD UUID> for the first two steps (the PBD one is in the CLI reference: https://docs.xcp-ng.org/appendix/cli_reference#pbd-unplug).

@mickwilli then added the unplug to his UPS shutdown script, and his host shut down cleanly after that. I haven't tried it with an SMB share myself, so I can't promise it behaves like his NFS one, but your NUT script on the Pi looks like the natural place for those two steps, just before the shutdown command.

XO is supposed to follow a master change by itself, at least on paper.

When it opens a session on a host that's no longer the master, XAPI answers HOST_IS_SLAVE with the new master's address and XO switches to it, and since 6.7.0 it also tries the other pool members when the stored host stops answering (https://github.com/vatesfr/xen-orchestra/pull/10016).
It doesn't rewrite the address you saved in Settings > Servers, though, so that entry keeps the old host, which is what your second screenshot shows. So I'd have expected a reconnect rather than a lost connection, but I haven't tried it myself.

Which XOA version are you on? If it's 6.7 or later and it still drops, that's useful to know, and I could well be missing a case.

I didn't know, so I went and looked at the code. XO prunes that list in a cleanup step: backup runs are kept for 31 days, and other tasks are kept by count, the newest 1000 (https://github.com/vatesfr/xen-orchestra/blob/b59c8f3d20954aa1d72e92c1e8f2db635d74fcba/@xen-orchestra/mixins/Tasks.mjs#L119-L127).

You can change both with tasks.gc.keep and tasks.gc.backupKeepDuration in the xo-server config.

As far as I can tell the cleanup runs when xo-server starts, not on a timer, so a long-running XO can collect quite a few before the next restart.

My guess for the Cloud Backup entries @acebmxer still sees after 30 days is that they're kept by count rather than by age.

I tried it on a lab host (XCP-ng 8.3, xapi 26.1.16): calling VM.pool_migrate on a halted VM gives me your error, VM_BAD_POWER_STATE with running, halted, so the two values are the state XAPI wanted and the one it found.

On the XO side, a migration inside the same pool with no SR or network mapping goes straight to VM.pool_migrate without checking the power state first: https://github.com/vatesfr/xen-orchestra/blob/b59c8f3d20954aa1d72e92c1e8f2db635d74fcba/packages/xo-server/src/xapi/index.mjs#L942-L981.
So a stopped VM in that batch would give this, which fits what you remember.

I haven't tried the batch migrate from the UI itself though, so I don't know if the UI is supposed to filter halted VMs out before that. It's a good concrete example for the feedback post anyway, XO could just say "this VM is halted" instead of passing XAPI's raw code through.

Thanks for posting the fix.

The same thing came up in an older thread, https://xcp-ng.org/forum/topic/9957, where disabling IPv6 also made xoa check go green.

@HamiltonWDS explained a likely reason in https://xcp-ng.org/forum/post/87831: Node tries the addresses with a short timeout and can end up on the IPv6 one.
Your curl -6 test helps a lot here, because it shows IPv6 doesn't connect at all on that network, so it's more than a slow path.

Turning it off in XOA seems reasonable if you don't use IPv6 there; if you do, I'd guess the router side is where it really needs fixing, though I could be wrong (still haven't migrated to IPv6 myself ).

Curious whether it sorts out the Cloud Backup too.

The update candidates posted yesterday on the 8.3 testing thread might help here: https://xcp-ng.org/forum/post/108803.

One of the sm fixes in that batch reads "a failed live leaf coalesce of a QCOW2 image on a slave host caused the SRs to remain in an SR_FAILURE_1200 error state until the VM using the VDI was stopped or tapdisk was manually paused", and to me (could be wrong, though), that looks a lot like your second case.
You're also on the versions it replaces (sm 23.5 and blktap 9.5, going to 25.1 and 11.1).

I'm less sure about the qcow2_commit: The device is not writable error from the first VM. The blktap notes mention reworking the pause/commit code, but I can't tell if that's the same path. If you have a test pool, trying the candidates there would tell a lot.

A short label on that step would've saved you the ghost hunt, yes.

Since it's a UI change more than a bug, I think https://feedback.vates.tech is where it'd get picked up, ideally with your two screenshots showing "All (8)" against seven VMs.

From your screenshot they all finished green, so this doesn't look like the tasks stuck at 97% in https://xcp-ng.org/forum/topic/10857 a while back.

I don't know how long XO is meant to keep finished backup tasks in that list, or whether they're supposed to clear on their own.

Do they go away after an xo-server restart, or do they come back?

I went looking and couldn't find the answer either.

The 9.2.385 changelog puts that fix under "Changes since 9.2.350", so the fix isn't in 9.2.350, but as far as I can see that says nothing about 9.1.x either way.

The driver changes between the two tags are submodule bumps, and I can't read those well enough to tell where the bug came from.

XenServer VM Tools are a separate build from Citrix as far as I know, so I wouldn't assume anything for your 9.6.0 VM.

This needs someone who worked on the fix, so maybe @Team-Hypervisor-Kernel can say which versions had the bug and whether older backups are worth re-checking.

Both of those are XAPI's raw error codes, and neither one tells you what to actually do about it. Annoying, I get it.

I looked up how XAPI defines them: OPERATION_BLOCKED means the operation is listed in the VM's blocked_operations field, and the true is just the value stored there, so I think it's the Prevent migration toggle from your screenshot (described here: https://docs.xen-orchestra.com/manage-your-infrastructure/manage_infrastructure#preventing-vm-migration).

For VM_BAD_POWER_STATE the parameters are the VM, the state XAPI expected, then the state it found, so running, halted would mean it wanted a running VM and got a halted one.
That reads like a live migration attempted on a stopped VM, though I haven't reproduced it.

Turning those into "this VM has Prevent migration on" or "this VM is halted" sounds like a good one for https://feedback.vates.tech, if you feel like posting it there with these two examples.

Thanks for letting us know, @acebmxer.

@Team-XO-Backend