@poddingue said: Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out". The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

@Danp No, it's not quite that. We don't use HA anywhere. But reading the source code I did notice exactly what's mentioned in that bug: in some HA cases, the xen-api code will throw a NOT_ENOUGH_FREE_MEMORY that is just absolutely misleading. At the macro scale, I think that you could say that the issue I'm running into is that the RPU migrations aren't planned out across the entire process of the RPU. If you have VMs that take up a significant fraction of some hosts' memory, failure to plan the whole process at once can lead the system to get itself "wedged" where it's unable to evacuate one of the hosts. More narrowly, you could say that the issue is that during an RPU the host evacuation code gives up more easily than it could. Say that you're trying to evacuate host A with a 64 GB VM, and you have two other hosts, B and C, with 60 GB free. It will give up, even if there are VMs that you could move from B to C (or vice versa) that would open up one of them to having 64 GB free. If the evacuation during the RPU was more resilient, then the macro-scale issue would be less of a problem. (Although with a more resilient evacuation command but no large-scale planning, you might end up with an RPU that succeeds but is wildly inefficient.)

I'm picking this one up alongside the other Body Timeout thread (9002), since they look (to me, at least) like the same underlying issue. Pierre from the XO team is on it and has tested metadata backups on the XO side without hitting it, so the current thinking is it may be coming from the XCP-ng side rather than XO. What would move it along fastest is a /var/log/xensource.log from the pool master covering one failed run's window, so if anyone here can reproduce and grab that slice, it'd really help line the timeout up against what xapi was doing. I know it's especially frustrating when the thing that breaks is your backups, thanks for bearing with us while we chase it. A mention to @Team-Storage might help route the storage-layer angle.

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue. EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

I think you've hit the known purge-snapshot-data issue. The XO incremental backup docs have a "Known issues" note (https://docs.xen-orchestra.com/xo5/incremental_backups#known-issues) about this: with purge snapshot data on, backups can transfer a full on every run, and the two suggested fixes are to disable purge (which you tried) or migrate the disk to another storage to reset the disk state. Since disabling purge didn't clear it, the disk migration is probably the one that resets the CBT chain. There's also an open issue matching your "it came back" symptom (https://github.com/vatesfr/xen-orchestra/issues/8713) where leftover invalid CBT snapshots keep forcing fulls until the chain is reset, so it's already tracked. I could be wrong about which step actually clears it, so if migrating the disk fixes it for you, dropping a note on that issue would help the next person. rtjdamen created this issue in vatesfr/xen-orchestra open CBT backup fails repeatedly due to leftover invalid CBT snapshots #8713

Merci beaucoup pour vos retours ! @nathanael-h : C'est l'étape suivante !!! J'ai découvert cluster API le mois dernier et j'ai déja testé sur d'autres plateforme ! ça déchire !!! Je ne savais pas qu'il y avait un support de XCP-ng ! Je finis ma série d'article pour une installation classique et je bascule sur cluster API ! @ataxyanetwork Merci pour ton repo. Je vais regarder ça