In some third-party backup solutions (for example Vinchin and similar enterprise platforms), proxy pool management and job distribution are fully automated, which helps when managing hundreds or thousands of VMs across sites.

Hi @olivierlambert, Finally got round to following up on this. I found a better solution using DEFAULT_CHUNK_SIZE, which is normally set to 4MB, with MAX_PART_NUMBER being 1000, this limits the largest object backed up to 3rd party S3 implementations as 4GB. Increasing the DEFAULT_CHUNK_SIZE in the S3 code, or having a setting in configuration, either file or web GUI would allow backups of VMs larger than 4GB, without the memory required to track several thousand object chunks. Regards, Mark

Update 4: Hello, After a few tries, i reattempted to re-add the host to the pool, this time capturing the logs in another terminal and this time it seems to have worked. I think i followed the same steps the previous time, but maybe i did something different, like using pif-scan to introduce the reordered interfaces. This is how I've managed to get it to work: All hosts on version 8.2.1 all up to date. On the new host disconnected the management, remove pif, reorder the interfaces, reboot, add interfaces manually (used pif-introduce, not pif-scan), reboot, re-enable management, join pool. ifconfig eth2 down ifconfig eth3 down xe pif-list device=eth2 xe pif-list device=eth3 xe pif-forget uuid=9104e54c-6c82-5d83-b9fc-1d2b73d5d6f1 xe pif-forget uuid=27abec81-6861-796e-9abe-3b2653444c8f interface-rename --update eth6=14:23:f2:24:5a:80 eth7=14:23:f2:24:5a:81 reboot now interface-rename --list ifconfig eth6 up ifconfig eth7 up xe host-list xe pif-introduce device=eth6 host-uuid=edfaf68e-2c28-4486-8939-723bf2c72820 mac=14:23:f2:24:5a:80 xe pif-introduce device=eth7 host-uuid=edfaf68e-2c28-4486-8939-723bf2c72820 mac=14:23:f2:24:5a:81 reboot now I'm still baffled why it worked now, I've tried a few time with all the host up to date, and was planning to source a dual nic pcie card, to solve this issue. However on this host the the bond id doesnt match the pool ... on poll: [image: 1772217597892-f68e687e-5896-459f-9463-8806ffe4bb63-image.png] on the new host: [image: 1772217689154-b58bac6f-f7fa-484e-be30-de2c4d4304bc-image.png] xensource_error_log.txt xensource.log also presents lots of errors, so I'm going to leave it running over the weekend, and will try to add the necessary data for our SR on monday. what is the best way to send the full log file?

Doh. I just posted a feature request on the github repo for XOA for this exact thing. Can you make a guess on a timeline?

@benapetr you seem gifted with app development. do you know RV Tools ? https://www.dell.com/support/kbdoc/en-us/000325532/rvtools-4-7-1-installer this tool is pretty handy when auditing VMWARE infrastructures, it can connect to vcenter or directly to ESXi and full dump in csv/xlsx the infrastructure configuration (all aspects of the config, be it VM, hosts, networks, datastores, files, ...) I could see a real production use of same tool, but for XCP Pools/XCP hosts would be a great add to XenAdmin !

@MajorP93 we have almost same size 30 backup jobs (mixed delta/cr/repl/dr) for 95+ VMs [image: 1772211113722-42570f88-2792-4954-8315-b9fa92aa9047-image.png] but these are not operated by XOA but dispatched in 4 XO Proxies and still XOA get OOM killed

@florent root@xo-server:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm root@xo-server:~# dpkg -l | grep gnutls ii libcurl3-gnutls:amd64 7.88.1-10+deb12u12 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour) ii libgnutls30:amd64 3.7.9-2 amd64 GNU TLS library - main runtime library ii libsrt1.5-gnutls:amd64 1.5.1-1 amd64 Secure Reliable Transport UDP streaming library (GnuTLS flavour)

@bleader thanks for the tip. You see my knowledge in redhat based systems is almost non existant. Think I'm almost 99% debian and 1% ubuntu (and that not by choice) So I will take care of redeploying the script as soon as I upgrade, should be easily noticeable since monitoring will loose access and yell at me.

@florent Awesome, i have patched a second location's XOA and am not experiencing the issue anymore there either.

@olivierlambert said in Rust-based guest-tools... How are things going?: The thing is that it works very well and doesn't need fixes… I know, not usual right? It was so stable that we left it like that waiting for people coming for bugs and nothing happened… We'll cut a 1.0 in the next weeks/months. I have indeed seen the stability of it myself. But the lack of updates made me worry anyway. But I'm really looking forward to the promotion to 1.0 as well. Thanks!

@rniesen Do you mean occasional hangs? Yes, Server 2025 is known to hang if these options are not present.

Hello, I have hardware issues, specially on the disks. Unfortunately I don't have fully access to the machine to be able to confirm if is effectively disks( more likely ) or back-plane. Interestingly some of those VMs does work, I can boot them and work with them but cannot copy the disks. Since those are Windows server VM I have also tried the Windows backup tool and this failed as well to perform to finish the backup. So I'm closing this investigation as hardware issue.

Just go to Settings > Servers tab in XO and remove the entry associated with this pool.

@mjr99 I bypassed it weeks ago. Didn't have to downgrade netbox. At the time. I upgraded to the latest Xo from sources. Then netbox worked. I shall be upgrading xo soon again Thank you all.

Instruction published: https://docs.xen-orchestra.com/sdn_controller#openssl-3--sdn-upgrade-path Any volunteer to test this along upcoming XCP-ng update ?

@Andrew This was the exact problem I was facing. The affected VMs still had the guest-tools.iso mounted in the virtual drive. Once I ejected the ISO, I was able to clear the errors and the backup jobs resumed normal operation. For anyone else hitting this "stuck delta" or VDI_IN_USE state, here is the exact workflow I followed to resolve it: Shut down the VM (Essential to release all storage locks). Eject the ISO/CD from the VM console or CLI. Delete all snapshots associated with the VM (including any old or "ghost" backup snapshots). Delete the VM's backup folder from the remote SR/Backup target to ensure a clean metadata start. Monitor the Coalesce process (via SMlog) and wait for it to finish flattening the VHD chain. Power the VM back on. Manually trigger a Backup Job. Following these steps, the first run was (as expected) a Full backup, but every subsequent run has been a successful Delta with no "hanging" snapshots left behind.