Delta Backups deleting .vhd files during merge operation

NeilGrevitt

Hi,

I am running xo-server 5.105.0 and xo-web 5.106.0 on Ubuntu 20.04.5 LTS, backing up a Xenserver, which has been running without trouble for many months.

I recently updated the server from a previous version, and since then the scheduled delta backups have been being deleted from the remote during the merge process. This is almost identical to another issue here

I am using TrueNas 13.0 SMB share as the remote. I'm running the old .vhd style of backup and have no syslog configured (afaik)

Output from journalctl is:

Nov 18 08:40:53 xoce xo-server[2223]: 2022-11-18T08:40:53.324Z xo:backups:mergeWorker INFO merge in progress {
Nov 18 08:40:53 xoce xo-server[2223]:   done: 1489,
Nov 18 08:40:53 xoce xo-server[2223]:   parent: '/xo-vm-backups/5371b36f-f3d2-bb48-6525-79a9def027da/vdis/890c2c65-027d-4d9d-ae9d-8580c4c1a802/a69eba8f-2f1f-4ca4-a5ad-7629f059abd3/20221117T060008Z.vhd',
Nov 18 08:40:53 xoce xo-server[2223]:   progress: 100,
Nov 18 08:40:53 xoce xo-server[2223]:   total: 1491
Nov 18 08:40:53 xoce xo-server[2223]: }
Nov 18 08:40:58 xoce xo-server[2223]: 2022-11-18T08:40:58.189Z xo:backups:mergeWorker WARN failure handling task {
Nov 18 08:40:58 xoce xo-server[2223]:   error: [Error: EACCES: permission denied, rename '/run/xo-server/mounts/9e36d44a-9bad-4df6-8b1a-0f6181ac1bfb/xo-vm-backups/5371b36f-f3d2-bb48-6525-79a9def027da/vdis/890c2c65-027d-4d9d-ae9d-8580c4c1a802/a69eba8f-2f                                                                                                                       1f-4ca4-a5ad-7629f059abd3/20221117T060008Z.vhd' -> '/run/xo-server/mounts/9e36d44a-9bad-4df6-8b1a-0f6181ac1bfb/xo-vm-backups/5371b36f-f3d2-bb48-6525-79a9def027da/vdis/890c2c65-027d-4d9d-ae9d-8580c4c1a802/a69eba8f-2f1f-4ca4-a5ad-7629f059abd3/20221118T083                                                                                                                       549Z.vhd'] {
Nov 18 08:40:58 xoce xo-server[2223]:     errno: -13,
Nov 18 08:40:58 xoce xo-server[2223]:     code: 'EACCES',
Nov 18 08:40:58 xoce xo-server[2223]:     syscall: 'rename',
Nov 18 08:40:58 xoce xo-server[2223]:     path: '/run/xo-server/mounts/9e36d44a-9bad-4df6-8b1a-0f6181ac1bfb/xo-vm-backups/5371b36f-f3d2-bb48-6525-79a9def027da/vdis/890c2c65-027d-4d9d-ae9d-8580c4c1a802/a69eba8f-2f1f-4ca4-a5ad-7629f059abd3/20221117T060008

As part of troubleshooting, I created a new Ubuntu VM and installed XO from scratch, and updated to the latest version xo-server 5.106.1 and xo-web 5.107.0, which gives the same results.

Many thanks
Neil

olivierlambert

I heard various stories on SMB issues on TrueNAS. Could you try to setup an NFS share to see if you still have the issue?

NeilGrevitt

@olivierlambert I've seen the mentions of it here also. It's strange as it's been working for the last 18months without issue!

I shall try to setup NFS. One thing that prevents me doing, (although I would imagine massively unsupported), is making copies of the larger VHD files from the SMB share on a Windows machines, as you can mount them to get at the files. Restoring large amounts of files through the browser doesn't work for me. Perhaps NFS would resolve that also! Thanks

NeilGrevitt

I may not have NFS correctly setup, as now I get a failure during the transfer stage:

Nov 18 16:45:33 xoce-new xo-server[10801]: 2022-11-18T16:45:33.150Z xo:backups:VmBackup WARN writer step failed {
Nov 18 16:45:33 xoce-new xo-server[10801]:   error: AssertionError [ERR_ASSERTION]: footer1 !== footer2
Nov 18 16:45:33 xoce-new xo-server[10801]:       at VhdFile.readHeaderAndFooter (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:192:7)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async VhdFile.open (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:96:5)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async openVhd (/opt/xen-orchestra/packages/vhd-lib/openVhd.js:10:12)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async Promise.all (index 0)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async checkVhd (/opt/xen-orchestra/@xen-orchestra/backups/writers/_checkVhd.js:7:3)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async NfsHandler._outputStream (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:562:9)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async NfsHandler.outputStream (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:186:5)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async RemoteAdapter.outputStream (/opt/xen-orchestra/@xen-orchestra/backups/RemoteAdapter.js:682:5)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async RemoteAdapter.writeVhd (/opt/xen-orchestra/@xen-orchestra/backups/RemoteAdapter.js:677:7)
Nov 18 16:45:33 xoce-new xo-server[10801]:       at async /opt/xen-orchestra/@xen-orchestra/backups/writers/DeltaBackupWriter.js:220:11 {
Nov 18 16:45:33 xoce-new xo-server[10801]:     generatedMessage: false,
Nov 18 16:45:33 xoce-new xo-server[10801]:     code: 'ERR_ASSERTION',
Nov 18 16:45:33 xoce-new xo-server[10801]:     actual: false,
Nov 18 16:45:33 xoce-new xo-server[10801]:     expected: true,
Nov 18 16:45:33 xoce-new xo-server[10801]:     operator: '=='
Nov 18 16:45:33 xoce-new xo-server[10801]:   },

tjkreidl

@NeilGrevitt I generally do a "showmount" against the storage device from each host to see if can talk to the storage device. I would hope that an upgrade would not affect connectivity, but sometimes drivers might get updated and change connectivity.

Gheppy

I'm using TrueNAS-12.0-U8.1, because as far as I can tell, 13.0+ has more bugs.
I tried TrueNAS SCALE version 22.02.4 and it is not ideal for use in production, I have a lot of failures with such as iSCSI, SMB connections.
Now I'm waiting for a version 13+ but U+

lawrencesystems

@Gheppy

The current version of TrueNAS-13.0-U3.1 has been stable for me, but this issue has persisted when doing Delta backups with SMB
& TrueNAS, even older versions.
https://xcp-ng.org/forum/topic/6148/delta-backup-showing-success-no-delta-saved

My resolution to the issue was to move the backups over to NFS and they work well. Also, since I am using the newer "Store backup as multiple data blocks instead of a whole VHD file" option NFS is generally a bit faster with handling small files.

sluflyer06

@lawrencesystems
What's the advantage of the newer block based backups?

olivierlambert

Almost no merge time (just renaming blocks around), and compatible with compression, encryption and in the future, dedup, tiering and such.

NeilGrevitt

Thanks all for your comments. I've managed to overcome the NFS errors listed, by using the new "multiple data blocks" setting! So for me, looks like backing up to .vhd won't work on either SMB or NFS.