@pkgw Thanks for the feedback. The good news is that there is already an effort under way rework and enhance the RPU process. Let me ping @Team-XAPI-Network and @Team-XO-Backend in case they want to request additional information from you.
Vates
People working at Vates
Posts
-
RE: Rolling Pool Update fails with HOST_NOT_ENOUGH_FREE_MEMORY, when it really ought to be fine
-
RE: Potential bug with Windows VM backup: "Body Timeout Error"
I'm picking this one up alongside the other Body Timeout thread (9002), since they look (to me, at least) like the same underlying issue.
Pierre from the XO team is on it and has tested metadata backups on the XO side without hitting it, so the current thinking is it may be coming from the XCP-ng side rather than XO.
What would move it along fastest is a/var/log/xensource.logfrom the pool master covering one failed run's window, so if anyone here can reproduce and grab that slice, it'd really help line the timeout up against what xapi was doing.
I know it's especially frustrating when the thing that breaks is your backups, thanks for bearing with us while we chase it.
A mention to @Team-Storage might help route the storage-layer angle. -
RE: Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"
Thanks @bogikornel, that version comparison really helps.
Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out".
On the log Pierre asked for: the useful one is
/var/log/xensource.logfrom the pool master, covering one failed run's window.
You already have the Job and Run IDs and the 00:05 to 01:22 timestamps from this morning's run, so that exact span is perfect.
If you can pull it from the master and attach it here (or just the slice around the timeout), that's what lets them line the failure up against what xapi was doing.
It's looking like it may be coming from the XCP-ng side rather than XO, so a mention to @Team-Storage might help too. I could be wrong about where it actually lands, but the log is the thing that'll tell us. -
RE: Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"
@bogikornel @jb @gregbinsd @majorp93
We tested metadata backups every 5 min and it works in XO. It seems it might be coming from XCP-NG, can you provide us a xensource.log from the problematic timeline? -
RE: Backups failing back to Full Backups
I think you've hit the known purge-snapshot-data issue. The XO incremental backup docs have a "Known issues" note (https://docs.xen-orchestra.com/xo5/incremental_backups#known-issues) about this: with purge snapshot data on, backups can transfer a full on every run, and the two suggested fixes are to disable purge (which you tried) or migrate the disk to another storage to reset the disk state.
Since disabling purge didn't clear it, the disk migration is probably the one that resets the CBT chain.
There's also an open issue matching your "it came back" symptom (https://github.com/vatesfr/xen-orchestra/issues/8713) where leftover invalid CBT snapshots keep forcing fulls until the chain is reset, so it's already tracked.
I could be wrong about which step actually clears it, so if migrating the disk fixes it for you, dropping a note on that issue would help the next person. -
RE: Rolling Pool Update fails with HOST_NOT_ENOUGH_FREE_MEMORY, when it really ought to be fine
It's possible that you hit this known issue.
