@florent Great work, thanks for the fixes

@olivierlambert Thanks!

@olivierlambert I’ve got a few VM’s that lock up from time to time. There is no rhyme nor reason (different hosts, different storage, etc). As a result, I just have a job that restarts them at 5:30 am every day. I’d like to have the software tool software restart them but, in the occasion wher the VM is locked up tight as Fort Knox, I’d like the hard reset to happen. Make sense ?

@TS79 That makes sense - thanks!

@olivierlambert Very cool...thanks for the explanation (but I'll admit I had to google "Node streams")!

Update: It’s weird: • There are three VM’s on this host. The backup works with two but not with the third. It does with the “Body Timeout Error” error. • Two of the VM’s are almost identical (same drive sizes). The only difference is that one was set up as “Other install media” (came over from esxi) and the one that fails was set up using the “Windows Server 2022” template. • I normally backup to multiple NFS servers so I changed to try one at a time; both failed. • After watching it do the backup too many times to thing, I found that, at about the 95% stage, the snapshot stops writing to the NFS share. • About that time, the file /var/log/xensource.log records this information: o Feb 26 09:43:33 HOST1 xapi: [error||19977 HTTPS 192.168.1.5->:::80|[XO] VM export R:c14f4c4c1c4c|xapi_compression] nice failed to compress: exit code 70 o Feb 26 09:43:33 HOST1 xapi: [ warn||19977 HTTPS 192.168.1.5->:::80|[XO] VM export R:c14f4c4c1c4c|pervasiveext] finally: Error while running cleanup after failure of main function: Failure("nice failed to compress: exit code 70") o Feb 26 09:43:33 HOST1 xapi: [debug||922 |xapi events D:d21ea5c4dd9a|xenops] Event on VM fbcbd709-a9d9-4cc7-80de-90185a74eba4; resident_here = true o Feb 26 09:43:33 HOST1 xapi: [debug||922 |xapi events D:d21ea5c4dd9a|dummytaskhelper] task timeboxed_rpc D:a08ebc674b6d created by task D:d21ea5c4dd9a o Feb 26 09:43:33 HOST1 xapi: [debug||922 |timeboxed_rpc D:a08ebc674b6d|xmlrpc_client] stunnel pid: 339060 (cached) connected to 192.168.1.6:443 o Feb 26 09:43:33 HOST1 xapi: [debug||19977 HTTPS 192.168.1.5->:::80|[XO] VM export R:c14f4c4c1c4c|xmlrpc_client] stunnel pid: 296483 (cached) connected to 192.168.1.6:443 o Feb 26 09:43:33 HOST1 xapi: [error||19977 :::80|VM.export D:c62fe7a2b4f2|backtrace] [XO] VM export R:c14f4c4c1c4c failed with exception Server_error(CLIENT_ERROR, [ INTERNAL_ERROR: [ Unix.Unix_error(Unix.EPIPE, "write", "") ] ]) o Feb 26 09:43:33 HOST1 xapi: [error||19977 :::80|VM.export D:c62fe7a2b4f2|backtrace] Raised Server_error(CLIENT_ERROR, [ INTERNAL_ERROR: [ Unix.Unix_error(Unix.EPIPE, "write", "") ] ]) • I have no idea if it means anything but the “failed to compress” made me try something. I change “compression” from “Zstd” to “disabled” and that time it worked. Here are my results: Regular backup to TrueNas, “compression” set to “Zstd”: backup fails. Regular backup to TrueNas, “compression” set to “disabled”: backup is successful. Regular backup to vanilla Ubuntu test VM, “compression” set to “Zstd”: backup is successful. Delta backup to TrueNas: backup is successful. Sooooo…the $64,000 question is why doesn’t it work on that one VM when compression is on and it a TrueNas box?

This happens to me on 8.2 with one specific VM. It doesn't always happen but about once a week (it feels like) the backup will get stuck in Starting and it's always, or almost always, the same VM it's stuck on. I usually reboot the VM running XO and that makes the backup task change to Interrupted and then the next scheduled backup usually works. Would be nice to know if there's something that's causing this that I can try to fix. Not a world-ending problem but it's a nuisance since it stops all future backups for that Job until it's resolved. I'm happy that all the other VMs in that job get backed up at least.

@archw Ignore that entire post...user error. I have two networks that are spelled very close to the same thing and I was using the wrong one.

Can you write exactly what you would expect to be written as a "pre written" text?

@archw Scratch that one.....I should have googled first. I saw that someone suggested deleting the cache.json.gz file. I did and ran it again and it worked.

@Bambos try restarting your hosts.

Coalesce isn't instant, and yes, as soon you remove a snapshot, there's some work to do from the storage perspective

Yes, so for that to be done automatically, you can set up "offline backup" option, that will do exactly that

@archw said in Backup fails with "all targets have failed, step: writer.beforeBackup()": @florent Where would I find that log? I assume its in /var/log but which file? For a xoa it's in journalctl -u xo-server from source, it depends how you did setup XO

If this specific VM is causing issue, you can fallback on Clonezilla or something similar