@florent Finally getting back to this post, I know it's been months, just haven't had a lot of time, sorry! I think I am still a bit confused, but I will do some additional testing to see if I can confirm my suspicions. My confusion is that, you can't merge the deltas into the key if the key is locked behind Object Lock, the file isn't writable so you can't do the merge operation, right? So that being said, it sounds to me like maybe the retention of the object lock/immutability needs to be set to be less than the retention period in XOA, right? This way the original key is not immutable and can be written to when the merge happens? Or does XOA just "wait" until the key isn't locked and then do the merge operation?

@florent Looking forward to the docs. Just this one thing (Guess most would be interested): On my example: keeping 7 daily backups, I presume it keeps the last backup from each day (as there are 2 schedules I presume it would keep the backup from 22:00)

@florent What are your thoughts on my last three questions please.

@DustinB thanks all, the agent based backups seem to work the way we need, we removed the older alike backup appliances so fully on xoa backup now. I saw alike is going opensource but almost no activity on the github account, i expect that the product will discontinue as it is allready pretty outdated, nice chance for XOA to fill the gap.

@stephane-m-dev that happens again for 1 vm. { "data": { "type": "VM", "id": "316e7303-c9c9-9bb6-04ef-83948ee1b19e", "name_label": "name" }, "id": "1732299284886", "message": "backup VM", "start": 1732299284886, "status": "failure", "tasks": [ { "id": "1732299284997", "message": "clean-vm", "start": 1732299284997, "status": "failure", "warnings": [ { "data": { "path": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241101T181520Z.alias.vhd", "error": { "generatedMessage": true, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==" } }, "message": "VHD check error" }, { "data": { "alias": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241101T181520Z.alias.vhd" }, "message": "missing target of alias" } ], "end": 1732299341663, "result": { "code": "ERR_INVALID_ARG_TYPE", "message": "The \"paths[1]\" argument must be of type string. Received undefined", "name": "TypeError", "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"paths[1]\" argument must be of type string. Received undefined\n at resolve (node:path:1169:7)\n at normalize (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/path.js:21:27)\n at NfsHandler.__unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/abstract.js:412:32)\n at NfsHandler.unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/node_modules/limit-concurrency-decorator/index.js:97:24)\n at checkAliases (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:132:25)\n at async Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:284:5)\n at async Promise.all (index 1)\n at async RemoteAdapter.cleanVm (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:283:3)" } }, { "id": "1732299285125", "message": "clean-vm", "start": 1732299285125, "status": "failure", "warnings": [ { "data": { "path": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241101T181520Z.alias.vhd", "error": { "generatedMessage": true, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==" } }, "message": "VHD check error" }, { "data": { "alias": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241101T181520Z.alias.vhd" }, "message": "missing target of alias" } ], "end": 1732299343111, "result": { "code": "ERR_INVALID_ARG_TYPE", "message": "The \"paths[1]\" argument must be of type string. Received undefined", "name": "TypeError", "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"paths[1]\" argument must be of type string. Received undefined\n at resolve (node:path:1169:7)\n at normalize (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/path.js:21:27)\n at NfsHandler.__unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/abstract.js:412:32)\n at NfsHandler.unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/node_modules/limit-concurrency-decorator/index.js:97:24)\n at checkAliases (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:132:25)\n at async Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:284:5)\n at async Promise.all (index 3)\n at async RemoteAdapter.cleanVm (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:283:3)" } }, { "id": "1732299343953", "message": "snapshot", "start": 1732299343953, "status": "success", "end": 1732299346495, "result": "ee646d05-83b2-31d8-e54b-0d3b0cf7df1d" }, { "data": { "id": "4b6d24a3-0b1e-48d5-aac2-a06e3a8ee485", "isFull": false, "type": "remote" }, "id": "1732299346495:0", "message": "export", "start": 1732299346495, "status": "success", "tasks": [ { "id": "1732299353253", "message": "transfer", "start": 1732299353253, "status": "success", "end": 1732299450434, "result": { "size": 9674571776 } }, { "id": "1732299501828:0", "message": "clean-vm", "start": 1732299501828, "status": "success", "warnings": [ { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241101T181520Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241102T180758Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241102T180758Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241103T180648Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241103T180648Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241104T180802Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241104T180802Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241105T181019Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241104T180802Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241104T180802Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241102T180758Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241102T180758Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241103T180648Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241103T180648Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241105T181019Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241105T181019Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" } ], "end": 1732299518747, "result": { "merge": false } } ], "end": 1732299518760 }, { "data": { "id": "8da40b08-636f-450d-af15-3264b9692e1f", "isFull": false, "type": "remote" }, "id": "1732299346496", "message": "export", "start": 1732299346496, "status": "success", "tasks": [ { "id": "1732299353244", "message": "transfer", "start": 1732299353244, "status": "success", "end": 1732299450546, "result": { "size": 9674571776 } }, { "id": "1732299451765", "message": "clean-vm", "start": 1732299451765, "status": "success", "warnings": [ { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241101T181520Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241102T180758Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241102T180758Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241103T180648Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241103T180648Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241104T180802Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "parent": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241104T180802Z.alias.vhd", "child": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241105T181019Z.alias.vhd" }, "message": "parent VHD is missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241103T180648Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241103T180648Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241104T180802Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241104T180802Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241102T180758Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241102T180758Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" }, { "data": { "backup": "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/20241105T181019Z.json", "missingVhds": [ "/xo-vm-backups/316e7303-c9c9-9bb6-04ef-83948ee1b19e/vdis/90d0b5ca-9364-4011-adc4-b8c74a534da9/53843891-126f-4f0c-b645-8e8aa0a41b36/20241105T181019Z.alias.vhd" ] }, "message": "some VHDs linked to the backup are missing" } ], "end": 1732299501791, "result": { "merge": false } } ], "end": 1732299501828 } ], "infos": [ { "message": "Transfer data using NBD" } ], "end": 1732299518760 } ], "end": 1732299518761 }

@Andrew Got it. As we only have a single production server there is no shared storage so I guess the idea of a pool is mute.

@olivierlambert Thanks! Seems obvious now that I clicked on it but I didn't know I could get to them through there.

Ping @florent

@olivierlambert If the issue turns up again.... It was a weird one! As I deleted the previous backup jobs I can't repeat. Just had to move on. Perhaps someone else stumbles upon this and finds this thread...... I will report if it happens again (will try it wit the new backup)

@McHenry Look at this post: VM limits - CPU Limits

@msupport we use the AnywhereUSB for this reason. Haven't tested the performance with USB disks, but it works good for all our license keys.

@Andrew Got it. So the DR host may have fewer resources than the production host, which would work for a failure of one or two VMs but not for a total failure of the production host whereby all VMs need to run on the DR host. Is it only RAM that is a consideration between hosts? Meaning if my production host has 16 vCPUs and my DR host only has 8 vCPUs can I still set the VM to use 16 vCPUs and it will work on both hosts?

@CodeMercenary i have seen this as well, one of our backup repos was almost out of space, this decreased the backup speed, we thought of issues on the xoa side but it was just related to the repo itself…. So it is good to check both. The 1 gig part makes sense.

Excellent Happy to see our fix solved your problem and enhanced our compatibility with older XenServer version. Don't forget to upgrade, the path to get to XCP-ng is relatively straightforward

Is it PV, HVM or a PVHVM VM? Health Check can't work on HVM because there are no tools.

@cirosantos0 Indeed, our detection is wrong in this case, I will put this in our backlog, but don't expect it to be fixed soon. PRs are welcome though

An update, if anyone ever comes across this via search engine. Turns out it was my container's timezone. The image was set to pure UTC, no timezone, by default, so I believe when it was writing files to my network storage it introduced a discrepancy. My network share was recording the file metadata accurately to real-time, so I assume when it came time to do another backup, the file time XO expected was different, making it think it was "stale" or still being "held". Have now run both scheduled metadata and VM backups without any errors . In summary: make sure your time, date and timezone are set correctly!

@florent It was working before, this is likely something that has been broken when the handling of snapshots has been moved from VMs to VDIs. We need to fix this.

@probain Noted in backlog

Hi, That's because our doc isn't fully up to date with the new naming (even XO in the app itself). See this recap: [image: Schema-new-wording-backup-2.png] More details in here: https://xen-orchestra.com/blog/xen-orchestra-5-83/#backup-nomenclature