Backup issues with S3 remote
-
Using the current version of XO from source (66e67), and any older versions, my VM backups against a S3 remote has given errors in the S3 server log (S3 server using rclone with the Jottacloud backend).
In the rclone.log, I see the following messages when a backup fails (but marks the trasfer as successful)
(repeated multiple times):2025/02/16 22:32:52 ERROR : xcp-ng-windows-user/xo-vm-backups/ef1c8301-35b0-a6e5-a760-aec39ae2c6fb/vdis/e7c936cf-74b4-4873-a1da-ffbd19611a06/1e5f1c8e-ce02-4090-a9db-580ca29bf779/data/: Dir.Remove not empty 2025/02/16 22:32:52 ERROR : xcp-ng-windows-user/xo-vm-backups/ef1c8301-35b0-a6e5-a760-aec39ae2c6fb/vdis/e7c936cf-74b4-4873-a1da-ffbd19611a06/1e5f1c8e-ce02-4090-a9db-580ca29bf779/data/298fef1f-47c3-48d3-bc5e-68b23723d250.vhd/: Dir.Remove not empty 2025/02/16 22:32:52 ERROR : serve s3: Directory not empty 2025/02/16 22:32:52 ERROR : xcp-ng-windows-user/xo-vm-backups/ef1c8301-35b0-a6e5-a760-aec39ae2c6fb/vdis/e7c936cf-74b4-4873-a1da-ffbd19611a06/1e5f1c8e-ce02-4090-a9db-580ca29bf779/data/298fef1f-47c3-48d3-bc5e-68b23723d250.vhd/: Dir.Remove not empty 2025/02/16 22:32:52 ERROR : serve s3: Directory not empty
I have previously spit the backups to go to different S3 buckets, since putting them all in one gave me another error (timeout, or connection error, which shows when testing the remote).
Testing the remotes works well in my current setup (unless it fails tonight).
Backup result (marked as failed)
Remote is working
CBT enabled since a few weeks back:
Backup settings for that machine
Have I found something worth investigating further, or did I do something obvious wrong to make this job fail ?
-
Hi,
I'm not 100% sure to understand what are you doing with rclone exactly? Anything that would interfere with the backup process?
-
@olivierlambert I'm using rclone to be able to use my Jottacloud storage as S3 remote for XO.
So to do this, I have rclone connecting to Jotta, then use "rclone serve s3" to make it available as S3 compatible storage.
I have my detailed installation notes (and for others to try out if they wish) here:
https://tech.webit.nu/how-to-use-jottacloud-as-s3-storage/During the night's (weekly) backup of machines using 'sequences' all of them succeeded (the mentioned above is not included in that batch, but another), except that all gave the message "Can't do delta with this vdi, transfer will be a full" in the backup summary in XO.
Although all backups were supposed to have succeeded in the nightly job, I get some (repeated) messages in the rclone log indicating that XO does something thac could possibly be done in another way or that rclone is a bit sensitive when it comes to some client way of checking if a file exists, and when it comes to overwriting an existing one.
Some samples from the log for the successful backups:2025/02/17 02:10:46 ERROR : serve s3: HTTP error 409 (409 Conflict) returned body: "{\"message\":\"\"}\n" 2025/02/17 02:14:06 ERROR : serve s3: file does not exist 2025/02/17 02:14:06 ERROR : serve s3: file does not exist 2025/02/17 02:14:06 ERROR : serve s3: file does not exist
The last error is repeated many times (29 to be exact) for the machines being backed up (3 Linux + 1 windows + TrueNAS and pfSense installations).
The machines together have 11 VDIs (had to count to check if the messages were multiples of number of VDIs) -
I don't think this will work that way, but I will let @florent answer
-
@peo in object storage world ( S3 compatible) , there is not really the notion of "directory", it's more a convention that / is a logical level in the file key, but rclone has errors on "Directory not empty". It let me think that rclone do things a little differently than aws s3.
We try to be as compatible as possible with S3 implementation, but each object storage has its own quirk. Thus we only supports the most used setups for now.
-
@florent is there any way to (from xo) log what's happening when the errors occur ? Like in the last (and recurring) set of errors where I get a "File does not exist" in the rclone log ?
Tonight I had 3 machines backed up (successfully according to info in xo), a total of 6 VDIs. This matches with the number of errors in the rclone log (one for each machine plus one of for each VDI)2025/02/19 03:20:36 ERROR : serve s3: file does not exist 2025/02/19 03:20:36 ERROR : serve s3: file does not exist 2025/02/19 03:20:37 ERROR : serve s3: file does not exist 2025/02/19 03:24:33 ERROR : xcp-ng/xo-vm-backups/ea7d874c-fb7a-ab24-92a5-062fdf772af3/vdis/6d79cd71-d851-4422-b1e7-8c70f7fd1d97/0706b98e-761a-4e79-ba9b-32229d113d7a/data/7a521b9b-11e8-4b38-96f4-933007fa02f1.vhd/blocks/16/12: WriteFileHandle.New Rcat failed: HTTP error 409 (409 Conflict) returned body: "{\"message\":\"\"}\n" 2025/02/19 03:24:33 ERROR : serve s3: HTTP error 409 (409 Conflict) returned body: "{\"message\":\"\"}\n" 2025/02/19 03:33:12 ERROR : serve s3: file does not exist 2025/02/19 03:33:12 ERROR : serve s3: file does not exist 2025/02/19 03:33:12 ERROR : serve s3: file does not exist 2025/02/19 03:33:12 ERROR : serve s3: file does not exist 2025/02/19 03:33:12 ERROR : serve s3: file does not exist 2025/02/19 03:33:13 ERROR : serve s3: file does not exist
-
@peo do you have anything in the xo logs ? can you post the json of a failed backup job ?
-
@florent of course I'll share everything you need to be able to track what's causing this to happen.
This is from the "Win10 22H2 new" job marked as failed in the log (the job in one of the images in the beginning of the thread):
(file upload to the forum didn't accept json){ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1739741544498", "jobId": "e7c936cf-74b4-4873-a1da-ffbd19611a06", "jobName": "Win10 22H2 new", "message": "backup", "scheduleId": "60e57138-e4fd-4779-8627-d69188521c23", "start": 1739741544498, "status": "failure", "infos": [ { "data": { "vms": [ "ef1c8301-35b0-a6e5-a760-aec39ae2c6fb" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "ef1c8301-35b0-a6e5-a760-aec39ae2c6fb", "name_label": "Win10 22H2 new" }, "id": "1739741570271", "message": "backup VM", "start": 1739741570271, "status": "failure", "tasks": [ { "id": "1739741570276", "message": "clean-vm", "start": 1739741570276, "status": "failure", "end": 1739741577400, "result": { "name": "InternalError", "$fault": "client", "$metadata": { "httpStatusCode": 500, "requestId": "E0366AE459C6C2DA", "extendedRequestId": "RTAzNjZBRTQ1OUM2QzJEQUUwMzY2QUU0NTlDNkMyREFFMDM2NkFFNDU5QzZDMkRBRTAzNjZBRTQ1OUM2QzJEQQ==", "attempts": 3, "totalRetryDelay": 126 }, "Code": "InternalError", "message": "Internal Error", "stack": "InternalError: Internal Error\n at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)\n at /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5\n at de_CommandError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4935:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22" } }, { "id": "1739741578015", "message": "snapshot", "start": 1739741578015, "status": "success", "end": 1739741579867, "result": "60547522-f928-7aeb-b952-e0ba43c24903" }, { "data": { "id": "edfede19-7e24-419b-b1dd-1f03d55c09cf", "isFull": false, "type": "remote" }, "id": "1739741579868", "message": "export", "start": 1739741579868, "status": "failure", "tasks": [ { "id": "1739741581176", "message": "transfer", "start": 1739741581176, "status": "success", "end": 1739742166837, "result": { "size": 28490535936 } }, { "id": "1739742169674", "message": "clean-vm", "start": 1739742169674, "status": "failure", "end": 1739742181418, "result": { "name": "InternalError", "$fault": "client", "$metadata": { "httpStatusCode": 500, "requestId": "E0366AE459C6F87F", "extendedRequestId": "RTAzNjZBRTQ1OUM2Rjg3RkUwMzY2QUU0NTlDNkY4N0ZFMDM2NkFFNDU5QzZGODdGRTAzNjZBRTQ1OUM2Rjg3Rg==", "attempts": 3, "totalRetryDelay": 159 }, "Code": "InternalError", "message": "Internal Error", "stack": "InternalError: Internal Error\n at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)\n at /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5\n at de_CommandError (/opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4935:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14\n at async /opt/xo/xo-builds/xen-orchestra-202502152203/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22" } } ], "end": 1739742181418 } ], "infos": [ { "data": { "vdi": { "uuid": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "name_label": "Windows 10 (64-bit)_obiti", "name_description": "Created by XO", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:35ad91d9-de4b-6202-2362-2bfe39e7d16c", "VBDs": [ "OpaqueRef:a3757dd8-4a65-e2c8-ddb1-b107c970c779" ], "crash_dumps": [], "virtual_size": 32212254720, "physical_utilisation": 67584, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "bf81153f-e790-4b3b-badf-cd35246c434d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:0b0f5873-233c-53c4-ffef-bb3c46b69e95", "snapshots": [], "snapshot_time": "20250216T21:32:59Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:40ec8c23-81a8-e1d3-cb4b-728338907faa" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "vdi": { "uuid": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "name_label": "Windows 10 (64-bit)_obiti", "name_description": "Created by XO", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:35ad91d9-de4b-6202-2362-2bfe39e7d16c", "VBDs": [ "OpaqueRef:a3757dd8-4a65-e2c8-ddb1-b107c970c779" ], "crash_dumps": [], "virtual_size": 32212254720, "physical_utilisation": 67584, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "5f2654ff-6ff5-4b20-b5e3-4acadd7b24ca", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "bf81153f-e790-4b3b-badf-cd35246c434d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:0b0f5873-233c-53c4-ffef-bb3c46b69e95", "snapshots": [], "snapshot_time": "20250216T21:32:59Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" } ], "end": 1739742181419 } ], "end": 1739742181419 }
On each of the successful jobs (which logs "ERROR : serve s3: file does not exist" and other to rclone.log), the full backup log looks like this (VM with 3 VDIs):
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1739932413376", "jobId": "f2038d20-9282-4000-9950-83b6ca3eb709", "jobName": "itlab-2022", "message": "backup", "scheduleId": "5d0b9242-7f4e-4037-b1df-15d0c0dedb42", "start": 1739932413376, "status": "success", "infos": [ { "data": { "vms": [ "ac1e1920-d995-cf99-6e2a-d510baab89b9" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "ac1e1920-d995-cf99-6e2a-d510baab89b9", "name_label": "itlab-2022" }, "id": "1739932677868", "message": "backup VM", "start": 1739932677868, "status": "success", "tasks": [ { "id": "1739932677873", "message": "clean-vm", "start": 1739932677873, "status": "success", "end": 1739932685925, "result": { "merge": false } }, { "id": "1739932687138", "message": "snapshot", "start": 1739932687138, "status": "success", "end": 1739932708145, "result": "579a14a3-fd00-5e46-5926-4ddc64d13ada" }, { "data": { "id": "4ff682de-7873-4cb5-a184-c147773ce982", "isFull": false, "type": "remote" }, "id": "1739932708147", "message": "export", "start": 1739932708147, "status": "success", "tasks": [ { "id": "1739932718998", "message": "transfer", "start": 1739932718998, "status": "success", "end": 1739936096218, "result": { "size": 218480667136 } }, { "id": "1739936121274", "message": "clean-vm", "start": 1739936121274, "status": "success", "end": 1739936149987, "result": { "merge": false } } ], "end": 1739936149988 } ], "infos": [ { "data": { "vdi": { "uuid": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "name_label": "[ESXI]btitlab_1-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:57531c2a-ead2-4993-0a27-80033a7b71ba" ], "crash_dumps": [], "virtual_size": 165190565888, "physical_utilisation": 329728, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "504acaa2-4d91-4bec-8296-b3c6c471e82f" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:7df0a025-2c40-0e83-2c40-cf0944478a94", "snapshots": [], "snapshot_time": "20250219T02:38:24Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "data": { "vdi": { "uuid": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "name_label": "[ESXI]btitlab-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:874d7572-00f5-a6bc-75e7-45612bb6108d" ], "crash_dumps": [], "virtual_size": 513802240, "physical_utilisation": 5632, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "9e257a97-a668-4399-80cb-fb1374947784" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:4ca6f37f-2472-d414-da00-7ff55683fbe9", "snapshots": [], "snapshot_time": "20250219T02:38:22Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "data": { "vdi": { "uuid": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "name_label": "[ESXI]btitlab-2022-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:da1ddc5d-d3d8-08d4-27ad-8be70b72d12f" ], "crash_dumps": [], "virtual_size": 128849018880, "physical_utilisation": 257536, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "212a480b-9992-491a-9a9c-7c132194d86d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:d1e3f3a2-f2e6-59dc-779d-6f37d0e6bd63", "snapshots": [], "snapshot_time": "20250219T02:38:20Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta, will try to get a full stream" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:1013cd8f-ce3d-fc91-b9ce-e889e2958b43" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:7abbed9a-7dd4-2f89-bc32-64b536cc5ea1" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:326b20d3-d85c-1f1b-d46b-43cf35e1168b" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:5f13cf16-6a3a-5048-088f-c81a35198190" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "vdi": { "uuid": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "name_label": "[ESXI]btitlab_1-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:57531c2a-ead2-4993-0a27-80033a7b71ba" ], "crash_dumps": [], "virtual_size": 165190565888, "physical_utilisation": 329728, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "bac77aae-d756-474e-9ddb-fb08d1fe251c", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "504acaa2-4d91-4bec-8296-b3c6c471e82f" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:7df0a025-2c40-0e83-2c40-cf0944478a94", "snapshots": [], "snapshot_time": "20250219T02:38:24Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" }, { "data": { "vdi": { "uuid": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "name_label": "[ESXI]btitlab-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:874d7572-00f5-a6bc-75e7-45612bb6108d" ], "crash_dumps": [], "virtual_size": 513802240, "physical_utilisation": 5632, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "9358f02a-b6bb-41ee-80ea-a6b405cb9452", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "9e257a97-a668-4399-80cb-fb1374947784" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:4ca6f37f-2472-d414-da00-7ff55683fbe9", "snapshots": [], "snapshot_time": "20250219T02:38:22Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" }, { "data": { "vdi": { "uuid": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "name_label": "[ESXI]btitlab-2022-flat.vmdk", "name_description": "fromESXI from esxi", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:eb4f7a63-e973-acc1-5ca4-d5546db8ad1a", "VBDs": [ "OpaqueRef:da1ddc5d-d3d8-08d4-27ad-8be70b72d12f" ], "crash_dumps": [], "virtual_size": 128849018880, "physical_utilisation": 257536, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "a8f9ab4c-b883-4d10-82b6-a17333a46871", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vhd-parent": "212a480b-9992-491a-9a9c-7c132194d86d" }, "is_a_snapshot": true, "snapshot_of": "OpaqueRef:d1e3f3a2-f2e6-59dc-779d-6f37d0e6bd63", "snapshots": [], "snapshot_time": "20250219T02:38:20Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "OpaqueRef:NULL", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": true } }, "message": "Can't do delta with this vdi, transfer will be a full" } ], "end": 1739936149988 } ], "end": 1739936149988 }