XO error/warning: Clean VM directory. unhandled error while checking alias.

[MajorP93]

@pierrebrunet Thanks for fixing this critical issue!

Do you have any plans regarding internal automated tests that would ensure that critical issues like these (deletion / corruption of backups etc.) do not get shipped?

Best regards

[Andrew]

@pierrebrunet XO updated to master daefcab, same issue (delta backup S3, problem at merge).

I tried running backup several times to see if the error would clear (it did not). I tried deleting the backup and starting with a new full, error happened at the merge again.

Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.037Z xo:backups:MixinBackupWriter INFO Disk chain needs merging { count: 1 }
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.039Z xo:backups:MixinBackupWriter INFO merging disk chain {
Jun 01 10:07:24 xo2 xo-server[9341]:   chain: [
Jun 01 10:07:24 xo2 xo-server[9341]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134029Z.alias.vhd',
Jun 01 10:07:24 xo2 xo-server[9341]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134306Z.alias.vhd'
Jun 01 10:07:24 xo2 xo-server[9341]:   ]
Jun 01 10:07:24 xo2 xo-server[9341]: }
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.318Z xo:backups:MixinBackupWriter WARN unhandled error while checking alias {
Jun 01 10:07:24 xo2 xo-server[9341]:   alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134029Z.alias.vhd',
Jun 01 10:07:24 xo2 xo-server[9341]:   err: NotFound: UnknownError
Jun 01 10:07:24 xo2 xo-server[9341]:       at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1850:27)
Jun 01 10:07:24 xo2 xo-server[9341]:       at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
Jun 01 10:07:24 xo2 xo-server[9341]:       at async S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:424:13)
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:27:24
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:387:20
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:64:28
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:91:20
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26
Jun 01 10:07:24 xo2 xo-server[9341]:       at async S3Handler._getSize (/opt/xo/xo-builds/xen-orchestra-202606010843/@xen-orchestra/fs/dist/s3.js:304:20) {
Jun 01 10:07:24 xo2 xo-server[9341]:     '$fault': 'client',
Jun 01 10:07:24 xo2 xo-server[9341]:     '$retryable': undefined,
Jun 01 10:07:24 xo2 xo-server[9341]:     '$metadata': {
Jun 01 10:07:24 xo2 xo-server[9341]:       httpStatusCode: 404,
Jun 01 10:07:24 xo2 xo-server[9341]:       requestId: '18B4FA31C10C3A3E',
Jun 01 10:07:24 xo2 xo-server[9341]:       extendedRequestId: 'a18350611414fa9366e109699727e695e309120afc9803fcaff78a199f9bc4d6',
Jun 01 10:07:24 xo2 xo-server[9341]:       cfId: undefined,
Jun 01 10:07:24 xo2 xo-server[9341]:       attempts: 1,
Jun 01 10:07:24 xo2 xo-server[9341]:       totalRetryDelay: 0
Jun 01 10:07:24 xo2 xo-server[9341]:     }
Jun 01 10:07:24 xo2 xo-server[9341]:   }
Jun 01 10:07:24 xo2 xo-server[9341]: }
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.490Z xo:backups:worker INFO backup has ended

[pierrebrunet]

@MajorP93 We are building an automated QA that we keep updating and running regularly (preferably nightly) to add any event possible. We missed this one because we did not took enough failures into account, so we added a regression test for this too.

[pierrebrunet]

@Andrew I think I found the issue, I am working on it

[pierrebrunet]

@Andrew
Your error happens after a successful merge in s3 which removes old alias not found (expected). However, the error given by s3 is not generic so it was not handled by alias checking. This issue was always there but we decided to show the warnings during backup and fix them with this new refactoring.

Can you try this branch please?
https://github.com/vatesfr/xen-orchestra/tree/pbt_fix_warning_backup

[Andrew]

@pierrebrunet The error happens using Wasabi and MinIO. The pbt_fix_warning_backup still has the error (but no 404 error):

Clean VM directory 
 missing target of alias

Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.571Z xo:backups:MixinBackupWriter INFO Disk chain needs merging { count: 1 }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.571Z xo:backups:MixinBackupWriter INFO merging disk chain {
Jun 01 11:23:22 xo2 xo-server[4216]:   chain: [
Jun 01 11:23:22 xo2 xo-server[4216]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152042Z.alias.vhd',
Jun 01 11:23:22 xo2 xo-server[4216]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152256Z.alias.vhd'
Jun 01 11:23:22 xo2 xo-server[4216]:   ]
Jun 01 11:23:22 xo2 xo-server[4216]: }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.741Z xo:backups:MixinBackupWriter WARN missing target of alias {
Jun 01 11:23:22 xo2 xo-server[4216]:   alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152042Z.alias.vhd'
Jun 01 11:23:22 xo2 xo-server[4216]: }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.741Z xo:backups:MixinBackupWriter INFO removing alias with missing target {
Jun 01 11:23:22 xo2 xo-server[4216]:   alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152042Z.alias.vhd'
Jun 01 11:23:22 xo2 xo-server[4216]: }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.866Z xo:backups:worker INFO backup has ended

[pierrebrunet]

@Andrew You mean it fails the whole job for this warning? Or this is just a warning inside a successful job?

[Andrew]

@pierrebrunet It's a warning. Backup job is successful.

[pierrebrunet]

@Andrew ok, thank you. This one indeed is not supposed to be triggered indeed.

[FagnerMoraes]

Hello again,

When running the backup routine to the NAS, it is not saving the file anymore.

Please check the image showing:

the backup routine,
the path it points to on the NAS,
and the contents of the folder.

The .XVA file was not saved.

[FagnerMoraes]

On the S3 storage, the backup is being saved correctly and the old files are not being deleted unexpectedly.

Using the same backup configuration, the directory cleanup process is not executed twice there.