@ravenet well, applied those updates and still the same results unfortunately... I am definitely seeing load on the GPU, so it is able to start the process, lemonade server just sits waiting logging 2026-07-08 20:06:06.477 [Debug] (WrappedServer) Still waiting for llama-server... while the GPUs are sitting there with the model in mem (or at least the correct amount of mem for the model size being consumed). looks like the rocm backend is using version b1292 which does note the correct gfx1201 model in the release notes.

@poddingue Just ran one more backup to confirm working, and it fell back to full backup again.... What logs or other information should I look into to find the root cause or do you still think it fall under the known issue? [image: 1783507806440-screenshot_20260708_064922.png]

@MajorP93 I'l do that, didn't know about it.

@poddingue said: Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out". The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

@Danp No, it's not quite that. We don't use HA anywhere. But reading the source code I did notice exactly what's mentioned in that bug: in some HA cases, the xen-api code will throw a NOT_ENOUGH_FREE_MEMORY that is just absolutely misleading. At the macro scale, I think that you could say that the issue I'm running into is that the RPU migrations aren't planned out across the entire process of the RPU. If you have VMs that take up a significant fraction of some hosts' memory, failure to plan the whole process at once can lead the system to get itself "wedged" where it's unable to evacuate one of the hosts. More narrowly, you could say that the issue is that during an RPU the host evacuation code gives up more easily than it could. Say that you're trying to evacuate host A with a 64 GB VM, and you have two other hosts, B and C, with 60 GB free. It will give up, even if there are VMs that you could move from B to C (or vice versa) that would open up one of them to having 64 GB free. If the evacuation during the RPU was more resilient, then the macro-scale issue would be less of a problem. (Although with a more resilient evacuation command but no large-scale planning, you might end up with an RPU that succeeds but is wildly inefficient.)

Merci beaucoup pour vos retours ! @nathanael-h : C'est l'étape suivante !!! J'ai découvert cluster API le mois dernier et j'ai déja testé sur d'autres plateforme ! ça déchire !!! Je ne savais pas qu'il y avait un support de XCP-ng ! Je finis ma série d'article pour une installation classique et je bascule sur cluster API ! @ataxyanetwork Merci pour ton repo. Je vais regarder ça