CBT: the thread to centralize your feedback
-
Too many error for me..
Time for backup very very long, error at start (with retry) and at the end (with error on backup:Couldn't deleted snapshot data Couldn't deleted snapshot data Retry the VM backup due to an error the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution. Retry the VM backup due to an error Transfer data using NBD will delete snapshot data Snapshot data has been deleted Snapshot data has been deleted Transfer data using NBD Clean VM directory cleanVm: incorrect backup size in metadata Start: 2024-07-11 20:23 End: 2024-07-11 20:24 Snapshot Start: 2024-07-11 20:24 End: 2024-07-11 20:25 Backup XEN OLD Start: 2024-07-11 20:25 End: 2024-07-11 20:27 Duration: 2 minutes Clean VM directory cleanVm: incorrect backup size in metadata Start: 2024-07-12 01:44 End: 2024-07-12 01:44 Snapshot Start: 2024-07-12 01:44 End: 2024-07-12 01:46 Backup XEN OLD transfer Start: 2024-07-12 01:46 End: 2024-07-12 05:39 Duration: 4 hours Size: 397.86 GiB Speed: 29.12 MiB/s Start: 2024-07-12 01:46 End: 2024-07-12 05:39 Duration: 4 hours Start: 2024-07-11 20:23 End: 2024-07-12 05:39 Duration: 9 hours Error: Disk is still attached to DOM0 VM Type: deltanot good..
-
Snapshots not getting deleted most of the time. Purge Snapshot data is enabled.
You can see the error in the picture below
-
The question is about the root cause of the issue: why the VDI is still in use? Before trying again, try to restart the toolstack on all hosts, checking you have 0x VDI attached to the control domain or something like that.
-
No they are not attached.
on the last job there was no error in the log but the vdi snapshots are not going away (see snapshots in picture above)
-
Yes, if they can't be removed because they are used somewhere, there is a problem.
Also, I see you are using XenCenter, but that's not the right tool here. Check the Dashboard/health in XO to see.
-
The snapshots are also show in XO
Subsequent delta backups are working and not leaving orphaned snaps
-
That's not what I'm talking about in my previous post.
-
why are the snapshots of psrv30_base and psrv30_data01 kept after a full backup???
Here what you wanted me tho check:
-
- So first, you still have VDI attached to the control domain. Outside the actual backup jop/export, it's NOT normal to have those here.
- The snapshots you are showing a snapshots without any data, just the metadata. We can improve the UI to detect it and show it's not a regular VDI snapshot. If you don't see the VM snapshot in the VM/snapshot view, it's OK.
-
-
-
Another Backup is running
-
So to summarize: You always get (in th UI) one snapshot which takes no space if you do delta with CBT. The snapshots can be ignored. Is this right?
-
-
Yes. It's not taking any space and won't prevent coalesce immediately after data are purged.
-
Thanks.
Additional Info: The VDI_IN_USE problem was because i always deleted the Snaps manually because i thougt they where orphan. Than the next Backup complains with this error
EDIT: Doesnot seems to be the root cause. VDI_IN_USE seems to be a comon problem when he wants to delete the snapshot-data and is not able to. there seems to be a deeper problem here.
-
I have another problem.
The size of some CBT-Snapshots is greater than 0B after some hours while others are 0B as it is to be expected when "Purge snapshot data" is checked. All disks which begin with psrv3... are backuped with the same job.
coalescing of snapshots is working flawless even with much bigger disks > 300GB. subsequent deltas are creating snaps and removing them so the problem only exists on the snaps taken by the first full. it seems that the problem is again VDI_IN_USE. The logs are at the bottom of the post.
Job config
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1721134380101", "jobId": "67da224e-ca75-47d3-87bf-ef30684b812e", "jobName": "psrv3x", "message": "backup", "scheduleId": "4c4eadfe-8517-47ba-9489-2e155d9d8f67", "start": 1721134380101, "status": "success", "infos": [ { "data": { "vms": [ "c72117ee-5b7e-2cc6-ddd3-5b5e4d2ded3f", "a1f2621a-b9a0-8b8f-a623-e8d1a465b9d1", "227439aa-f1c0-38de-e805-210ca4cfe4f1", "369fb26f-c45b-2911-b8ae-0b891162e55a", "a5a47785-992e-4322-1902-a06e87e1b89b", "b2101bf0-10b2-3296-bf46-9f6ab5417258", "b7cf3c63-f78f-8b13-a8cc-7ca7e99f7f2d", "3d15d157-6ca0-b13b-c6e4-2c49d2f90a82" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c72117ee-5b7e-2cc6-ddd3-5b5e4d2ded3f", "name_label": "psrv30" }, "id": "1721134381225", "message": "backup VM", "start": 1721134381225, "status": "success", "tasks": [ { "id": "1721134381253", "message": "clean-vm", "start": 1721134381253, "status": "success", "end": 1721134382311, "result": { "merge": false } }, { "id": "1721134382324", "message": "snapshot", "start": 1721134382324, "status": "success", "end": 1721134468381, "result": "7a671117-98ff-442c-8a25-0b518159dd91" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134468381:0", "message": "export", "start": 1721134468381, "status": "success", "tasks": [ { "id": "1721134474475", "message": "transfer", "start": 1721134474475, "status": "success", "end": 1721134603873, "result": { "size": 5604999168 } }, { "id": "1721134610042", "message": "clean-vm", "start": 1721134610042, "status": "success", "end": 1721134612114, "result": { "merge": false } } ], "end": 1721134612115 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:6257f019-dd31-8b16-732c-5d29911e0e51" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:a6b4051c-6007-96d7-6053-77ff8c2bcdaa", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:a6b4051c-6007-96d7-6053-77ff8c2bcdaa" ] } }, "vdiRef": "OpaqueRef:a6b4051c-6007-96d7-6053-77ff8c2bcdaa" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721134612115 }, { "data": { "type": "VM", "id": "a1f2621a-b9a0-8b8f-a623-e8d1a465b9d1", "name_label": "psrv32" }, "id": "1721134381231", "message": "backup VM", "start": 1721134381231, "status": "success", "tasks": [ { "id": "1721134381329", "message": "clean-vm", "start": 1721134381329, "status": "success", "end": 1721134381498, "result": { "merge": false } }, { "id": "1721134381509", "message": "snapshot", "start": 1721134381509, "status": "success", "end": 1721134414736, "result": "46548732-64a4-9ec2-8f2f-fcd6695f6127" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134414737", "message": "export", "start": 1721134414737, "status": "success", "tasks": [ { "id": "1721134422205", "message": "transfer", "start": 1721134422205, "status": "success", "end": 1721134749129, "result": { "size": 11214152704 } }, { "id": "1721134755579", "message": "clean-vm", "start": 1721134755579, "status": "success", "end": 1721134756596, "result": { "merge": false } } ], "end": 1721134756596 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:11727e21-ce91-c93b-9381-bd741466aff5" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:453add79-e2b9-f1a3-1940-2fe9fb6e9955", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:453add79-e2b9-f1a3-1940-2fe9fb6e9955" ] } }, "vdiRef": "OpaqueRef:453add79-e2b9-f1a3-1940-2fe9fb6e9955" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721134756597 }, { "data": { "type": "VM", "id": "369fb26f-c45b-2911-b8ae-0b891162e55a", "name_label": "psrv33" }, "id": "1721134756624", "message": "backup VM", "start": 1721134756624, "status": "success", "tasks": [ { "id": "1721134756646", "message": "clean-vm", "start": 1721134756646, "status": "success", "end": 1721134756699, "result": { "merge": false } }, { "id": "1721134756709", "message": "snapshot", "start": 1721134756709, "status": "success", "end": 1721134807757, "result": "3712099c-5138-f9d1-975b-75a8779e0e8c" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134807758", "message": "export", "start": 1721134807758, "status": "success", "tasks": [ { "id": "1721134814098", "message": "transfer", "start": 1721134814098, "status": "success", "end": 1721136709333, "result": { "size": 42135879168 } }, { "id": "1721136715790", "message": "clean-vm", "start": 1721136715790, "status": "success", "end": 1721136717005, "result": { "merge": false } } ], "end": 1721136717005 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:dfc7cdf5-70cc-7161-31d8-2ea8ef39a232" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:98cb909a-79a2-3138-66b9-1ebe15701963", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:98cb909a-79a2-3138-66b9-1ebe15701963" ] } }, "vdiRef": "OpaqueRef:98cb909a-79a2-3138-66b9-1ebe15701963" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721136717005 }, { "data": { "type": "VM", "id": "a5a47785-992e-4322-1902-a06e87e1b89b", "name_label": "psrv38" }, "id": "1721136717009", "message": "backup VM", "start": 1721136717009, "status": "success", "tasks": [ { "id": "1721136717037", "message": "clean-vm", "start": 1721136717037, "status": "success", "end": 1721136717114, "result": { "merge": false } }, { "id": "1721136717126", "message": "snapshot", "start": 1721136717126, "status": "success", "end": 1721136756030, "result": "9161f706-3a4e-aa61-c485-f239bea58c1c" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721136756030:0", "message": "export", "start": 1721136756030, "status": "success", "tasks": [ { "id": "1721136762801", "message": "transfer", "start": 1721136762801, "status": "success", "end": 1721137081725, "result": { "size": 14236907008 } }, { "id": "1721137087610", "message": "clean-vm", "start": 1721137087610, "status": "success", "end": 1721137088697, "result": { "merge": false } } ], "end": 1721137088697 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:8348309e-a904-0ef5-718d-5464aa73bfdf" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:f577ba25-5f0e-79c8-c19b-8f67b9e70817", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:f577ba25-5f0e-79c8-c19b-8f67b9e70817" ] } }, "vdiRef": "OpaqueRef:f577ba25-5f0e-79c8-c19b-8f67b9e70817" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721137088697 }, { "data": { "type": "VM", "id": "227439aa-f1c0-38de-e805-210ca4cfe4f1", "name_label": "psrv31" }, "id": "1721134612122", "message": "backup VM", "start": 1721134612122, "status": "success", "tasks": [ { "id": "1721134612144", "message": "clean-vm", "start": 1721134612144, "status": "success", "end": 1721134612234, "result": { "merge": false } }, { "id": "1721134612261", "message": "snapshot", "start": 1721134612261, "status": "success", "end": 1721134637673, "result": "7fca43e7-50ce-c409-78f9-3f622990231b" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134637674", "message": "export", "start": 1721134637674, "status": "success", "tasks": [ { "id": "1721134644637", "message": "transfer", "start": 1721134644637, "status": "success", "end": 1721137110501, "result": { "size": 67748438528 } }, { "id": "1721137116357", "message": "clean-vm", "start": 1721137116357, "status": "success", "end": 1721137117343, "result": { "merge": false } } ], "end": 1721137117343 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:e5166784-8ccb-599c-2e26-14c1cb7cc455" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:8b8c3e0e-d961-ac5f-807f-8871117ff4e8", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:8b8c3e0e-d961-ac5f-807f-8871117ff4e8" ] } }, "vdiRef": "OpaqueRef:8b8c3e0e-d961-ac5f-807f-8871117ff4e8" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721137117343 }, { "data": { "type": "VM", "id": "b2101bf0-10b2-3296-bf46-9f6ab5417258", "name_label": "psrv37" }, "id": "1721137088704", "message": "backup VM", "start": 1721137088704, "status": "success", "tasks": [ { "id": "1721137088727", "message": "clean-vm", "start": 1721137088727, "status": "success", "end": 1721137088789, "result": { "merge": false } }, { "id": "1721137088814", "message": "snapshot", "start": 1721137088814, "status": "success", "end": 1721137140639, "result": "ce015f67-e251-4719-04e4-3fc1f5a32e4d" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721137140639:0", "message": "export", "start": 1721137140639, "status": "success", "tasks": [ { "id": "1721137149516", "message": "transfer", "start": 1721137149516, "status": "success", "end": 1721137256573, "result": { "size": 5334400512 } }, { "id": "1721137263812", "message": "clean-vm", "start": 1721137263812, "status": "success", "end": 1721137265019, "result": { "merge": false } } ], "end": 1721137265020 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:53a79aeb-5a64-4e2f-b016-bb2d52b913ba" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:8164fe16-721e-e6c9-4a45-c7a435ab84e7" }, "message": "Snapshot data has been deleted" } ], "end": 1721137265020 }, { "data": { "type": "VM", "id": "b7cf3c63-f78f-8b13-a8cc-7ca7e99f7f2d", "name_label": "psrv36" }, "id": "1721137117347", "message": "backup VM", "start": 1721137117347, "status": "success", "tasks": [ { "id": "1721137117366", "message": "clean-vm", "start": 1721137117366, "status": "success", "end": 1721137117421, "result": { "merge": false } }, { "id": "1721137117433", "message": "snapshot", "start": 1721137117433, "status": "success", "end": 1721137182291, "result": "40eaa629-2c0d-cfe5-a969-59511d803d33" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721137182291:0", "message": "export", "start": 1721137182291, "status": "success", "tasks": [ { "id": "1721137188995", "message": "transfer", "start": 1721137188995, "status": "success", "end": 1721137354685, "result": { "size": 6817448960 } }, { "id": "1721137361886", "message": "clean-vm", "start": 1721137361886, "status": "success", "end": 1721137363289, "result": { "merge": false } } ], "end": 1721137363289 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:a4556739-a7a7-0ad9-83d3-d982b7c6b634" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:52e2e793-1301-f647-16cc-a49c49f54844" }, "message": "Snapshot data has been deleted" } ], "end": 1721137363289 }, { "data": { "type": "VM", "id": "3d15d157-6ca0-b13b-c6e4-2c49d2f90a82", "name_label": "psrv34" }, "id": "1721137265039", "message": "backup VM", "start": 1721137265039, "status": "success", "tasks": [ { "id": "1721137265065", "message": "clean-vm", "start": 1721137265065, "status": "success", "end": 1721137265129, "result": { "merge": false } }, { "id": "1721137265190", "message": "snapshot", "start": 1721137265190, "status": "success", "end": 1721137301489, "result": "27fac29e-86ea-bbf9-4232-fd69c8518562" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721137301490", "message": "export", "start": 1721137301490, "status": "success", "tasks": [ { "id": "1721137308542", "message": "transfer", "start": 1721137308542, "status": "success", "end": 1721137521612, "result": { "size": 8711639552 } }, { "id": "1721137527713", "message": "clean-vm", "start": 1721137527713, "status": "success", "end": 1721137528729, "result": { "merge": false } } ], "end": 1721137528729 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:5bf6cd52-e947-fc28-1815-cc4d035f9c68" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:47a39aa7-7dea-0956-00d7-02324dd0495d", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:47a39aa7-7dea-0956-00d7-02324dd0495d" ] } }, "vdiRef": "OpaqueRef:47a39aa7-7dea-0956-00d7-02324dd0495d" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721137528729 } ], "end": 1721137528729 } -
xe vdi-data-destroy does not free the space either.
[root@host3 sm]# xe vdi-list params=uuid,name-label,virtual-size,physical-utilisation,type type="CBT metadata" name-label=psrv34_data01 uuid ( RO) : ceb5a540-7d53-4614-be3e-b329d8d7336b name-label ( RW): psrv34_data01 virtual-size ( RO): 10737418240 physical-utilisation ( RO): 8388608 type ( RO): CBT metadata [root@host3 sm]# xe vdi-data-destroy uuid=ceb5a540-7d53-4614-be3e-b329d8d7336b [root@host3 sm]# xe vdi-list params=uuid,name-label,virtual-size,physical-utilisation,type type="CBT metadata" name-label=psrv34_data01 uuid ( RO) : ceb5a540-7d53-4614-be3e-b329d8d7336b name-label ( RW): psrv34_data01 virtual-size ( RO): 10737418240 physical-utilisation ( RO): 8388608 type ( RO): CBT metadata -
The current cbt backup is working but it contains several bugs that needs to be resolved. @florent is working on fixes but from what i understand it is difficult to be fixed. Hope there will be progress soon!
-
@rtjdamen I noticed this today as well. A VM within a pool migrated from one host to another, after which i received
Error: stream has ended with not enough data (actual: 449, expected: 512)Retrying the backup resulting it in running a Full Backup. Hopefully XO can start to handle this otherwise im not sure CBT backups are worth it over the old method for pools with multiple hosts. Running a Rolling Pool Update or Rolling Pool Reboot would guarantee every VM would fail and require a full backup? Or is this not expected behaviour?
-
@flakpyro no this is a bug, we see this on vms that are not migrated. In our other backup software we do not see this issue, using cbt there as well.
-
I cannot reproduce in my own production here, the only problem is when I have a job on 2 different pool, one pool doesn't have NBD enabled and re-doing a full. The NBD enabled pool works perfectly though
-
@olivierlambert From what i understand about this issue is it occurs when snapshot is deleted at the wrong time, although this is what florent told me. he created a fix for it but that does not resolve the issue.
