Delta Backup Fails for a Single VM - ENOTDIR (.vhd/chunk-filters.json)
-
We're running Xen Orchestra from the sources and our current version is xo-server 5.103.1, xo-web 5.104.0, and commit c964a. The server itself is running Ubuntu 20.04.5 with 16 vCPU's and 32GB of RAM. In our case we're running Citrix Hypervisor 8.2.
We currently have a daily delta backup job with 22 virtual machines set to a 7-day retention period. Data is stored on a separate NFS server. Lately we have a single virtual machine where the delta backup is unable to complete towards the end of the process.
Some details about the virtual machine in question below:
- VM has 2 VHD's (100GB and 1TB). Issue appears to be with the 1TB VHD
- A full backup completes successfully every time
- Ubuntu 20.04.5 with XenTools installed
Here's the error we keep running in to...
"errno": -20, "code": "ENOTDIR", "syscall": "open",Error: ENOTDIR: not a directory, open '/nfs/remote/xo-vm-backups/6f8c2e66-85c1-cea6-0d75-2a974bab2408/vdis/2584632c-2f85-4f11-a3b0-47096b057c02/fc2c54ca-3fbb-4b20-ab04-dce8cd401d15/.20221015T050036Z.vhd/chunk-filters.json"Error at LocalHandler.addSyncStackTrace [as _addSyncStackTrace] (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:28:26) at LocalHandler._readFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:184:23) at LocalHandler.readFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:326:29) at LocalHandler.readFile (/opt/xen-orchestra/node_modules/limit-concurrency-decorator/dist/index.js:107:24) at VhdDirectory.#readChunkFilters (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdDirectory.js:312:46) at VhdDirectory.readHeaderAndFooter (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdDirectory.js:186:33) at Function.open (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdDirectory.js:113:15) at openVhd (/opt/xen-orchestra/packages/vhd-lib/openVhd.js:16:31) at async Promise.all (index 0) at async checkVhd (/opt/xen-orchestra/@xen-orchestra/backups/writers/_checkVhd.js:7:3)",I have tried moving all existing backup data stored on the remote to a different location and clearing the old backups to give Xen Orchestra the opportunity to start fresh and that didn't help. As mentioned above, full backup tasks (and everything else) work with the VM in question, we only run into an issue with the delta backup method. Updating/rebuilding the Xen Orchestra server hasn't helped as of yet.
We have VMs with the same guest OS/Hypervisor/XenTools combination that are working fine with the same backup task. My efforts of searching for any details regarding this error have come up short as of now. Any details you could provide would be amazing.
I'll update to the latest commit as soon as possible however I currently have a full backup task running for the specific VM to ensure we're covered for the time being.
-
Hi,
Could you test if you have the same behavior in a XOA on
latestrelease channel?In the meantime, I'm pinging @florent
-
hi @Can-eh-dian11
Is there a way to give me access to the VHD ? given its size, I would prefer a remote access to the serverregards
-
That's why a XOA will be helpful, so a simple support tunnel will do the trick
