Continuous replication is using export/import mechanism, so I think that's the reason for this task

I just had a backup fail with a similar error. Details: XO Community Xen Orchestra, commit 749f0 Master, commit 749f0 Merge backups synchronously was off on the backup job, going to enable it. [image: 1740267400062-88c76052-e8f1-43dc-a218-5794d46ebaad-image.png] { "data": { "type": "VM", "id": "4f715b32-ddfb-5818-c7bd-aaaa2a77ce70", "name_label": "PROD_SophosXG" }, "id": "1740263738986", "message": "backup VM", "start": 1740263738986, "status": "failure", "warnings": [ { "message": "the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution." } ], "end": 1740263738993, "result": { "code": "ELOCKED", "file": "/run/xo-server/mounts/f992fff1-e245-48f7-8eb3-25987ecbfbd4/xo-vm-backups/4f715b32-ddfb-5818-c7bd-aaaa2a77ce70", "message": "Lock file is already being held", "name": "Error", "stack": "Error: Lock file is already being held\n at /opt/xo/xo-builds/xen-orchestra-202502212211/node_modules/proper-lockfile/lib/lockfile.js:68:47\n at callback (/opt/xo/xo-builds/xen-orchestra-202502212211/node_modules/graceful-fs/polyfills.js:306:20)\n at FSReqCallback.oncomplete (node:fs:199:5)\n at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)\nFrom:\n at NfsHandler.addSyncStackTrace (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/local.js:21:26)\n at NfsHandler._lock (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/local.js:135:48)\n at NfsHandler.lock (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/abstract.js:234:27)\n at IncrementalRemoteWriter.beforeBackup (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:54:34)\n at async IncrementalRemoteWriter.beforeBackup (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:68:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:343:7\n at async callWriter (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:33:9)\n at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:342:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38" } }, Email report of failure VM Backup report Global status : failure 🚨 • Job ID: bfadfecc-b651-4fd6-b104-2f015100db29 • Run ID: 1740263706121 • Mode: delta • Start time: Saturday, February 22nd 2025, 5:35:06 pm • End time: Saturday, February 22nd 2025, 5:37:59 pm • Duration: 3 minutes • Successes: 0 / 8 1 Failure PROD_SophosXG Production Sophos Firewall Application • UUID: 4f715b32-ddfb-5818-c7bd-aaaa2a77ce70 • Start time: Saturday, February 22nd 2025, 5:35:38 pm • End time: Saturday, February 22nd 2025, 5:35:38 pm • Duration: a few seconds • ⚠️ the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution. • Error: Lock file is already being held Manually run completed PROD_SophosXG (xcp02) Clean VM directory cleanVm: incorrect backup size in metadata Start: 2025-02-22 18:37 End: 2025-02-22 18:37 Snapshot Start: 2025-02-22 18:37 End: 2025-02-22 18:37 Qnap NFS Backup transfer Start: 2025-02-22 18:37 End: 2025-02-22 18:38 Duration: a minute Size: 12.85 GiB Speed: 185.52 MiB/s Start: 2025-02-22 18:37 End: 2025-02-22 18:44 Duration: 7 minutes Start: 2025-02-22 18:37 End: 2025-02-22 18:44 Duration: 7 minutes Type: delta

Hi jkatz - I am a sales manager for Vates America Corp and I apologize for any confusion but 'complete feature set' is referring to XOA. We list XOSTOR, XOPROXY, and Airgap Support all as 'optional' because they are add-ons and incur additional costs. One of my team members or I will be happy to answer any questions you have and discuss the various costs of our add-on offerings. We can be reached at sales@vates.tech and we'll look forward to hearing from you.

I confirm local configuration is not backup with Pool metadata backup.

It's really hard to tell, have you restarted also all the other pool members?

@starmood For each VM backup run inside of Xen Orchestra it backs up not just the VHD but everything that is needed to restore that VM to any other XCP-ng host. So in a complete loss situation you can load a new XCP-ng Host, setup Xen Orchestra, point the new Xen Orchestra to those backups and restore any of the VM's with all their settings. The metadata backup of XCP-ng is just that, the metadata of the system and it's not granular. It's good to have because all the things like network setting and what VM's are on the hosts will be there, but the VM backups to me are the most important. I have a tutorial covering how the backups work, it's from a bit over a year ago and there are EVEN MORE features now and I will be doing a new video this year to cover that. https://youtu.be/weVoKm8kDb4?si=1z6IDqwnK1cxEGjm I also have a tutorial on how you can also automate the backup validation process https://youtu.be/A0HTRF3dhQE?si=gZLXQUqLJmDkIQs6

@florent of course I'll share everything you need to be able to track what's causing this to happen. This is from the "Win10 22H2 new" job marked as failed in the log (the job in one of the images in the beginning of the thread): (file upload to the forum didn't accept json) { "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1739741544498", "jobId": "e7c936cf-74b4-4873-a1da-ffbd19611a06", "jobName": "Win10 22H2 new", "message": "backup", "scheduleId": "60e57138-e4fd-4779-8627-d69188521c23", "start": 1739741544498, "status": "failure", "infos": [ { "data": { "vms": [ "ef1c8301-35b0-a6e5-a760-aec39ae2c6fb" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "ef1c8301-35b0-a6e5-a760-aec39ae2c6fb", "name_label": "Win10 22H2 new" }, "id": "1739741570271", "message": "backup VM", "start": 1739741570271, "status": "failure", "tasks": [ { "id": "1739741570276", "message": "clean-vm", "start": 1739741570276, "status": "failure", "end": 1739741577400, "result": { "name": "InternalError", "$fault": "client", "$metadata": { "httpStatusCode": 500, "requestId": "E0366AE459C6C2DA", "extendedRequestId": "RTAzNjZBRTQ1OUM2QzJEQUUwMzY2QUU0NTlDNkMyREFFMDM2NkFFNDU5QzZDMkRBRTAzNjZBRTQ1OUM2QzJEQQ==", "attempts": 3, "totalRetryDelay": 126 }, "Code": "InternalError", "message": "Internal Error", "stack": "InternalError: Internal Error\n at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)\n at /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5\n at de_CommandError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4935:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22" } }, { "id": "1739741578015", "message": "snapshot", "start": 1739741578015, "status": "success", "end": 1739741579867, "result": "60547522-f928-7aeb-b952-e0ba43c24903" }, { "data": { "id": "edfede19-7e24-419b-b1dd-1f03d55c09cf", "isFull": false, "type": "remote" }, "id": "1739741579868", "message": "export", "start": 1739741579868, "status": "failure", "tasks": [ { "id": "1739741581176", "message": "transfer", "start": 1739741581176, "status": "success", "end": 1739742166837, "result": { "size": 28490535936 } }, { "id": "1739742169674", "message": "clean-vm", "start": 1739742169674, "status": "failure", "end": 1739742181418, "result": { "name": "InternalError", "$fault": "client", "$metadata": { "httpStatusCode": 500, "requestId": "E0366AE459C6F87F", "extendedRequestId": "RTAzNjZBRTQ1OUM2Rjg3RkUwMzY2QUU0NTlDNkY4N0ZFMDM2NkFFNDU5QzZGODdGRTAzNjZBRTQ1OUM2Rjg3Rg==", "attempts": 3, "totalRetryDelay": 159 }, "Code": "InternalError", "message": "Internal Error", "stack": "InternalError: Internal Error\n at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)\n at /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5\n at de_CommandError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4935:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22" } } ], "end": 1739742181418 } ], "infos": [ { "data": { "vdi": { "uuid": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "name_label": "Windows 10 (64-bit)_obiti", "name_description": "Created by XO", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:35ad91d9-de4b-6202-2362-2bfe39e7d16c", "VBDs": [ "OpaqueRef:a3757dd8-4a65-e2c8-ddb1-b107c970c779" ], "crash_dumps": [], "virtual_size": 32212254720, "physical_utilisation": 67584, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "bf81153f-e790-4b3b-badf-cd35246c434d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:0b0f5873-233c-53c4-ffef-bb3c46b69e95", "snapshots": [], "snapshot_time": "20250216T21:32:59Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:40ec8c23-81a8-e1d3-cb4b-728338907faa" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "vdi": { "uuid": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "name_label": "Windows 10 (64-bit)_obiti", "name_description": "Created by XO", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:35ad91d9-de4b-6202-2362-2bfe39e7d16c", "VBDs": [ "OpaqueRef:a3757dd8-4a65-e2c8-ddb1-b107c970c779" ], "crash_dumps": [], "virtual_size": 32212254720, "physical_utilisation": 67584, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "bf81153f-e790-4b3b-badf-cd35246c434d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:0b0f5873-233c-53c4-ffef-bb3c46b69e95", "snapshots": [], "snapshot_time": "20250216T21:32:59Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" } ], "end": 1739742181419 } ], "end": 1739742181419 } On each of the successful jobs (which logs "ERROR : serve s3: file does not exist" and other to rclone.log), the full backup log looks like this (VM with 3 VDIs): { "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1739932413376", "jobId": "f2038d20-9282-4000-9950-83b6ca3eb709", "jobName": "itlab-2022", "message": "backup", "scheduleId": "5d0b9242-7f4e-4037-b1df-15d0c0dedb42", "start": 1739932413376, "status": "success", "infos": [ { "data": { "vms": [ "ac1e1920-d995-cf99-6e2a-d510baab89b9" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "ac1e1920-d995-cf99-6e2a-d510baab89b9", "name_label": "itlab-2022" }, "id": "1739932677868", "message": "backup VM", "start": 1739932677868, "status": "success", "tasks": [ { "id": "1739932677873", "message": "clean-vm", "start": 1739932677873, "status": "success", "end": 1739932685925, "result": { "merge": false } }, { "id": "1739932687138", "message": "snapshot", "start": 1739932687138, "status": "success", "end": 1739932708145, "result": "579a14a3-fd00-5e46-5926-4ddc64d13ada" }, { "data": { "id": "4ff682de-7873-4cb5-a184-c147773ce982", "isFull": false, "type": "remote" }, "id": "1739932708147", "message": "export", "start": 1739932708147, "status": "success", "tasks": [ { "id": "1739932718998", "message": "transfer", "start": 1739932718998, "status": "success", "end": 1739936096218, "result": { "size": 218480667136 } }, { "id": "1739936121274", "message": "clean-vm", "start": 1739936121274, "status": "success", "end": 1739936149987, "result": { "merge": false } } ], "end": 1739936149988 } ], "infos": [ { "data": { "vdi": { "uuid": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "name_label": "[ESXI]btitlab_1-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:57531c2a-ead2-4993-0a27-80033a7b71ba" ], "crash_dumps": [], "virtual_size": 165190565888, "physical_utilisation": 329728, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "504acaa2-4d91-4bec-8296-b3c6c471e82f" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:7df0a025-2c40-0e83-2c40-cf0944478a94", "snapshots": [], "snapshot_time": "20250219T02:38:24Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "data": { "vdi": { "uuid": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "name_label": "[ESXI]btitlab-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:874d7572-00f5-a6bc-75e7-45612bb6108d" ], "crash_dumps": [], "virtual_size": 513802240, "physical_utilisation": 5632, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "9e257a97-a668-4399-80cb-fb1374947784" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:4ca6f37f-2472-d414-da00-7ff55683fbe9", "snapshots": [], "snapshot_time": "20250219T02:38:22Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "data": { "vdi": { "uuid": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "name_label": "[ESXI]btitlab-2022-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:da1ddc5d-d3d8-08d4-27ad-8be70b72d12f" ], "crash_dumps": [], "virtual_size": 128849018880, "physical_utilisation": 257536, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "212a480b-9992-491a-9a9c-7c132194d86d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:d1e3f3a2-f2e6-59dc-779d-6f37d0e6bd63", "snapshots": [], "snapshot_time": "20250219T02:38:20Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:1013cd8f-ce3d-fc91-b9ce-e889e2958b43" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:7abbed9a-7dd4-2f89-bc32-64b536cc5ea1" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:326b20d3-d85c-1f1b-d46b-43cf35e1168b" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:5f13cf16-6a3a-5048-088f-c81a35198190" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "vdi": { "uuid": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "name_label": "[ESXI]btitlab_1-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:57531c2a-ead2-4993-0a27-80033a7b71ba" ], "crash_dumps": [], "virtual_size": 165190565888, "physical_utilisation": 329728, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "504acaa2-4d91-4bec-8296-b3c6c471e82f" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:7df0a025-2c40-0e83-2c40-cf0944478a94", "snapshots": [], "snapshot_time": "20250219T02:38:24Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" }, { "data": { "vdi": { "uuid": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "name_label": "[ESXI]btitlab-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:874d7572-00f5-a6bc-75e7-45612bb6108d" ], "crash_dumps": [], "virtual_size": 513802240, "physical_utilisation": 5632, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "9e257a97-a668-4399-80cb-fb1374947784" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:4ca6f37f-2472-d414-da00-7ff55683fbe9", "snapshots": [], "snapshot_time": "20250219T02:38:22Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" }, { "data": { "vdi": { "uuid": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "name_label": "[ESXI]btitlab-2022-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:da1ddc5d-d3d8-08d4-27ad-8be70b72d12f" ], "crash_dumps": [], "virtual_size": 128849018880, "physical_utilisation": 257536, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "212a480b-9992-491a-9a9c-7c132194d86d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:d1e3f3a2-f2e6-59dc-779d-6f37d0e6bd63", "snapshots": [], "snapshot_time": "20250219T02:38:20Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" } ], "end": 1739936149988 } ], "end": 1739936149988 }

@jr-m4 Yes, it seems that all dropdown lists are broken by a recent dependency update. XO team has been notified.

@ph7 Thanks for the report, we'll check that

@olivierlambert @DustinB @Forza May I suggest that it goes a bit further, unless it already does so. Can the VMs in Xen Orchestra show if it was backed up successfully in the most recent job. As well as when it was last backed up, if you happen to not have the report yet (or read it) you can see at a glance. That way it makes decoding the orange status for the backup job easier, so you know which ones you need to do a backup job run for. Alternatively show which ones failed and were successful, when you get the details for the job (task) in Xen Orchestra following an orange status on Xen Orchestra 6.

Ah great! It should be documented I suppose, ping @thomas-dkmt

@Disbelief5920 Backup VMs from production cluster in datacenter to separate backup storage in same datacenter. (Delta Backup. Retention <X> Monthly, <X> Weekly, <X> daily). Easily, through the scheduling system you can have the system perform Continuous Replication or a standard Delta job. Copy the backup file from production datacenter backup storage to offsite DR datacenter backup storage. (Retention possibly different than production <Y> Monthly, <Y> Weekly, <Y> daily). I think what you'd be looking at would be the "Mirror Backup" job. This'll copy your backup repo to another available repo. copy the backup from production datacenter or offsite DR datacenter to cloud storage (Backblaze) Yes, this can be completed in XO's interface or from your NAS/SAN solutions (or client side server software). Continuous replication from production cluster in datacenter to offsite DR datacenter (1-2x daily for example). I've had CR jobs running as quickly as every 5 minutes between two datacenters. Obviously bandwidth matters, once or twice a day should be perfectly doable. Is it possible to achieve the above by only backing up the production VM one time so it does not need to do 2x snapshots and 2x data copies? (1 for the backup and 1 for the replication jobs? You'd use the Delta job format for the bulk of these. Is it possible to achieve the above with only 1 transfer across the WAN instead of once for the continuous replication and once for the delta backup? No, CR is very expressly any changes that have happened, it's continuous by it's nature @Disbelief5920 said in Backup and Replication Strategy: For example, is there a scenario that would first create a backup from production to onsite backup storage, then replicate across the WAN to the backup storage, then restore the VM from the backup storage to a DR cluster? This was the production NAS/XCP-NG nodes would only have to process/copy the data once, only 1 copy would transfer across the WAN and only 1 restore from offsite backup storage to DR cluster. I'm not 100% on this scenario.... so I don't think I can readily help explain it.

@rtjdamen I was in the same situation I made a new schedule, BCK-Man with a tag BCK-Man. When I needed a manual backup, I added the tag to that VM

I would like to ask a followup question to confirm. so IF using sequences, we should disable the backup tasks on the overview tab?

Hi @olivierlambert, No performance gain to keep the snapshot. It's more the opposite. Leaving the snapshot means it will be only removed in the next job, meaning you'll have more to coalesce. However, it's very stable, CBT tends to have some quirks from time to time, so worth trying on few VMs first. Thanks - perhaps the backup documentation could have a page that covers each of these options and the pro's / con's of enabling / disabling each of them. They seem to be newer options added as they weren't even shown in the webinar video and I couldn't find any mention of them anywhere. Also, if it's better to have this option enabled, then perhaps it could be enabled by default ? I don't think it's possible now. I can ask internally if it's something that's planned already or doable in the future Thanks - that'll be good. Either option will be good. The ability to set the 'Full Backup Interval' against the individual schedule instead of the whole backup task may be all that needs to change. I'll submit a ticket to get this feature request started. Thanks Kent.

@Danp Thank you very much! Missed it

As a separate thought, do you have another instance of XO (or XOA) running somewhere which may have been configured with the xo-config file?

@florent Clear now, thanks!