We're running Xen Orchestra from the sources and our current version is xo-server 5.103.1, xo-web 5.104.0, and commit c964a. The server itself is running Ubuntu 20.04.5 with 16 vCPU's and 32GB of RAM. In our case we're running Citrix Hypervisor 8.2.
We currently have a daily delta backup job with 22 virtual machines set to a 7-day retention period. Data is stored on a separate NFS server. Lately we have a single virtual machine where the delta backup is unable to complete towards the end of the process.
Some details about the virtual machine in question below:
- VM has 2 VHD's (100GB and 1TB). Issue appears to be with the 1TB VHD
- A full backup completes successfully every time
- Ubuntu 20.04.5 with XenTools installed
Here's the error we keep running in to...
"errno": -20, "code": "ENOTDIR", "syscall": "open",
Error: ENOTDIR: not a directory, open '/nfs/remote/xo-vm-backups/6f8c2e66-85c1-cea6-0d75-2a974bab2408/vdis/2584632c-2f85-4f11-a3b0-47096b057c02/fc2c54ca-3fbb-4b20-ab04-dce8cd401d15/.20221015T050036Z.vhd/chunk-filters.json
"Error at LocalHandler.addSyncStackTrace [as _addSyncStackTrace] (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:28:26) at LocalHandler._readFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:184:23) at LocalHandler.readFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:326:29) at LocalHandler.readFile (/opt/xen-orchestra/node_modules/limit-concurrency-decorator/dist/index.js:107:24) at VhdDirectory.#readChunkFilters (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdDirectory.js:312:46) at VhdDirectory.readHeaderAndFooter (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdDirectory.js:186:33) at Function.open (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdDirectory.js:113:15) at openVhd (/opt/xen-orchestra/packages/vhd-lib/openVhd.js:16:31) at async Promise.all (index 0) at async checkVhd (/opt/xen-orchestra/@xen-orchestra/backups/writers/_checkVhd.js:7:3)",
I have tried moving all existing backup data stored on the remote to a different location and clearing the old backups to give Xen Orchestra the opportunity to start fresh and that didn't help. As mentioned above, full backup tasks (and everything else) work with the VM in question, we only run into an issue with the delta backup method. Updating/rebuilding the Xen Orchestra server hasn't helped as of yet.
We have VMs with the same guest OS/Hypervisor/XenTools combination that are working fine with the same backup task. My efforts of searching for any details regarding this error have come up short as of now. Any details you could provide would be amazing.
I'll update to the latest commit as soon as possible however I currently have a full backup task running for the specific VM to ensure we're covered for the time being.