CBT: the thread to centralize your feedback
-
I have another problem.
The size of some CBT-Snapshots is greater than 0B after some hours while others are 0B as it is to be expected when "Purge snapshot data" is checked. All disks which begin with psrv3... are backuped with the same job.
coalescing of snapshots is working flawless even with much bigger disks > 300GB. subsequent deltas are creating snaps and removing them so the problem only exists on the snaps taken by the first full. it seems that the problem is again VDI_IN_USE. The logs are at the bottom of the post.
Job config
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1721134380101", "jobId": "67da224e-ca75-47d3-87bf-ef30684b812e", "jobName": "psrv3x", "message": "backup", "scheduleId": "4c4eadfe-8517-47ba-9489-2e155d9d8f67", "start": 1721134380101, "status": "success", "infos": [ { "data": { "vms": [ "c72117ee-5b7e-2cc6-ddd3-5b5e4d2ded3f", "a1f2621a-b9a0-8b8f-a623-e8d1a465b9d1", "227439aa-f1c0-38de-e805-210ca4cfe4f1", "369fb26f-c45b-2911-b8ae-0b891162e55a", "a5a47785-992e-4322-1902-a06e87e1b89b", "b2101bf0-10b2-3296-bf46-9f6ab5417258", "b7cf3c63-f78f-8b13-a8cc-7ca7e99f7f2d", "3d15d157-6ca0-b13b-c6e4-2c49d2f90a82" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c72117ee-5b7e-2cc6-ddd3-5b5e4d2ded3f", "name_label": "psrv30" }, "id": "1721134381225", "message": "backup VM", "start": 1721134381225, "status": "success", "tasks": [ { "id": "1721134381253", "message": "clean-vm", "start": 1721134381253, "status": "success", "end": 1721134382311, "result": { "merge": false } }, { "id": "1721134382324", "message": "snapshot", "start": 1721134382324, "status": "success", "end": 1721134468381, "result": "7a671117-98ff-442c-8a25-0b518159dd91" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134468381:0", "message": "export", "start": 1721134468381, "status": "success", "tasks": [ { "id": "1721134474475", "message": "transfer", "start": 1721134474475, "status": "success", "end": 1721134603873, "result": { "size": 5604999168 } }, { "id": "1721134610042", "message": "clean-vm", "start": 1721134610042, "status": "success", "end": 1721134612114, "result": { "merge": false } } ], "end": 1721134612115 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:6257f019-dd31-8b16-732c-5d29911e0e51" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:a6b4051c-6007-96d7-6053-77ff8c2bcdaa", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:a6b4051c-6007-96d7-6053-77ff8c2bcdaa" ] } }, "vdiRef": "OpaqueRef:a6b4051c-6007-96d7-6053-77ff8c2bcdaa" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721134612115 }, { "data": { "type": "VM", "id": "a1f2621a-b9a0-8b8f-a623-e8d1a465b9d1", "name_label": "psrv32" }, "id": "1721134381231", "message": "backup VM", "start": 1721134381231, "status": "success", "tasks": [ { "id": "1721134381329", "message": "clean-vm", "start": 1721134381329, "status": "success", "end": 1721134381498, "result": { "merge": false } }, { "id": "1721134381509", "message": "snapshot", "start": 1721134381509, "status": "success", "end": 1721134414736, "result": "46548732-64a4-9ec2-8f2f-fcd6695f6127" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134414737", "message": "export", "start": 1721134414737, "status": "success", "tasks": [ { "id": "1721134422205", "message": "transfer", "start": 1721134422205, "status": "success", "end": 1721134749129, "result": { "size": 11214152704 } }, { "id": "1721134755579", "message": "clean-vm", "start": 1721134755579, "status": "success", "end": 1721134756596, "result": { "merge": false } } ], "end": 1721134756596 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:11727e21-ce91-c93b-9381-bd741466aff5" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:453add79-e2b9-f1a3-1940-2fe9fb6e9955", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:453add79-e2b9-f1a3-1940-2fe9fb6e9955" ] } }, "vdiRef": "OpaqueRef:453add79-e2b9-f1a3-1940-2fe9fb6e9955" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721134756597 }, { "data": { "type": "VM", "id": "369fb26f-c45b-2911-b8ae-0b891162e55a", "name_label": "psrv33" }, "id": "1721134756624", "message": "backup VM", "start": 1721134756624, "status": "success", "tasks": [ { "id": "1721134756646", "message": "clean-vm", "start": 1721134756646, "status": "success", "end": 1721134756699, "result": { "merge": false } }, { "id": "1721134756709", "message": "snapshot", "start": 1721134756709, "status": "success", "end": 1721134807757, "result": "3712099c-5138-f9d1-975b-75a8779e0e8c" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134807758", "message": "export", "start": 1721134807758, "status": "success", "tasks": [ { "id": "1721134814098", "message": "transfer", "start": 1721134814098, "status": "success", "end": 1721136709333, "result": { "size": 42135879168 } }, { "id": "1721136715790", "message": "clean-vm", "start": 1721136715790, "status": "success", "end": 1721136717005, "result": { "merge": false } } ], "end": 1721136717005 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:dfc7cdf5-70cc-7161-31d8-2ea8ef39a232" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:98cb909a-79a2-3138-66b9-1ebe15701963", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:98cb909a-79a2-3138-66b9-1ebe15701963" ] } }, "vdiRef": "OpaqueRef:98cb909a-79a2-3138-66b9-1ebe15701963" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721136717005 }, { "data": { "type": "VM", "id": "a5a47785-992e-4322-1902-a06e87e1b89b", "name_label": "psrv38" }, "id": "1721136717009", "message": "backup VM", "start": 1721136717009, "status": "success", "tasks": [ { "id": "1721136717037", "message": "clean-vm", "start": 1721136717037, "status": "success", "end": 1721136717114, "result": { "merge": false } }, { "id": "1721136717126", "message": "snapshot", "start": 1721136717126, "status": "success", "end": 1721136756030, "result": "9161f706-3a4e-aa61-c485-f239bea58c1c" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721136756030:0", "message": "export", "start": 1721136756030, "status": "success", "tasks": [ { "id": "1721136762801", "message": "transfer", "start": 1721136762801, "status": "success", "end": 1721137081725, "result": { "size": 14236907008 } }, { "id": "1721137087610", "message": "clean-vm", "start": 1721137087610, "status": "success", "end": 1721137088697, "result": { "merge": false } } ], "end": 1721137088697 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:8348309e-a904-0ef5-718d-5464aa73bfdf" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:f577ba25-5f0e-79c8-c19b-8f67b9e70817", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:f577ba25-5f0e-79c8-c19b-8f67b9e70817" ] } }, "vdiRef": "OpaqueRef:f577ba25-5f0e-79c8-c19b-8f67b9e70817" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721137088697 }, { "data": { "type": "VM", "id": "227439aa-f1c0-38de-e805-210ca4cfe4f1", "name_label": "psrv31" }, "id": "1721134612122", "message": "backup VM", "start": 1721134612122, "status": "success", "tasks": [ { "id": "1721134612144", "message": "clean-vm", "start": 1721134612144, "status": "success", "end": 1721134612234, "result": { "merge": false } }, { "id": "1721134612261", "message": "snapshot", "start": 1721134612261, "status": "success", "end": 1721134637673, "result": "7fca43e7-50ce-c409-78f9-3f622990231b" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721134637674", "message": "export", "start": 1721134637674, "status": "success", "tasks": [ { "id": "1721134644637", "message": "transfer", "start": 1721134644637, "status": "success", "end": 1721137110501, "result": { "size": 67748438528 } }, { "id": "1721137116357", "message": "clean-vm", "start": 1721137116357, "status": "success", "end": 1721137117343, "result": { "merge": false } } ], "end": 1721137117343 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:e5166784-8ccb-599c-2e26-14c1cb7cc455" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:8b8c3e0e-d961-ac5f-807f-8871117ff4e8", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:8b8c3e0e-d961-ac5f-807f-8871117ff4e8" ] } }, "vdiRef": "OpaqueRef:8b8c3e0e-d961-ac5f-807f-8871117ff4e8" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721137117343 }, { "data": { "type": "VM", "id": "b2101bf0-10b2-3296-bf46-9f6ab5417258", "name_label": "psrv37" }, "id": "1721137088704", "message": "backup VM", "start": 1721137088704, "status": "success", "tasks": [ { "id": "1721137088727", "message": "clean-vm", "start": 1721137088727, "status": "success", "end": 1721137088789, "result": { "merge": false } }, { "id": "1721137088814", "message": "snapshot", "start": 1721137088814, "status": "success", "end": 1721137140639, "result": "ce015f67-e251-4719-04e4-3fc1f5a32e4d" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721137140639:0", "message": "export", "start": 1721137140639, "status": "success", "tasks": [ { "id": "1721137149516", "message": "transfer", "start": 1721137149516, "status": "success", "end": 1721137256573, "result": { "size": 5334400512 } }, { "id": "1721137263812", "message": "clean-vm", "start": 1721137263812, "status": "success", "end": 1721137265019, "result": { "merge": false } } ], "end": 1721137265020 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:53a79aeb-5a64-4e2f-b016-bb2d52b913ba" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:8164fe16-721e-e6c9-4a45-c7a435ab84e7" }, "message": "Snapshot data has been deleted" } ], "end": 1721137265020 }, { "data": { "type": "VM", "id": "b7cf3c63-f78f-8b13-a8cc-7ca7e99f7f2d", "name_label": "psrv36" }, "id": "1721137117347", "message": "backup VM", "start": 1721137117347, "status": "success", "tasks": [ { "id": "1721137117366", "message": "clean-vm", "start": 1721137117366, "status": "success", "end": 1721137117421, "result": { "merge": false } }, { "id": "1721137117433", "message": "snapshot", "start": 1721137117433, "status": "success", "end": 1721137182291, "result": "40eaa629-2c0d-cfe5-a969-59511d803d33" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721137182291:0", "message": "export", "start": 1721137182291, "status": "success", "tasks": [ { "id": "1721137188995", "message": "transfer", "start": 1721137188995, "status": "success", "end": 1721137354685, "result": { "size": 6817448960 } }, { "id": "1721137361886", "message": "clean-vm", "start": 1721137361886, "status": "success", "end": 1721137363289, "result": { "merge": false } } ], "end": 1721137363289 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:a4556739-a7a7-0ad9-83d3-d982b7c6b634" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:52e2e793-1301-f647-16cc-a49c49f54844" }, "message": "Snapshot data has been deleted" } ], "end": 1721137363289 }, { "data": { "type": "VM", "id": "3d15d157-6ca0-b13b-c6e4-2c49d2f90a82", "name_label": "psrv34" }, "id": "1721137265039", "message": "backup VM", "start": 1721137265039, "status": "success", "tasks": [ { "id": "1721137265065", "message": "clean-vm", "start": 1721137265065, "status": "success", "end": 1721137265129, "result": { "merge": false } }, { "id": "1721137265190", "message": "snapshot", "start": 1721137265190, "status": "success", "end": 1721137301489, "result": "27fac29e-86ea-bbf9-4232-fd69c8518562" }, { "data": { "id": "bfa75298-0f2d-4525-a4f4-c27c7b4443d3", "isFull": true, "type": "remote" }, "id": "1721137301490", "message": "export", "start": 1721137301490, "status": "success", "tasks": [ { "id": "1721137308542", "message": "transfer", "start": 1721137308542, "status": "success", "end": 1721137521612, "result": { "size": 8711639552 } }, { "id": "1721137527713", "message": "clean-vm", "start": 1721137527713, "status": "success", "end": 1721137528729, "result": { "merge": false } } ], "end": 1721137528729 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:5bf6cd52-e947-fc28-1815-cc4d035f9c68" }, "message": "Snapshot data has been deleted" } ], "warnings": [ { "data": { "error": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:47a39aa7-7dea-0956-00d7-02324dd0495d", "data_destroy" ], "call": { "method": "VDI.data_destroy", "params": [ "OpaqueRef:47a39aa7-7dea-0956-00d7-02324dd0495d" ] } }, "vdiRef": "OpaqueRef:47a39aa7-7dea-0956-00d7-02324dd0495d" }, "message": "Couldn't deleted snapshot data" } ], "end": 1721137528729 } ], "end": 1721137528729 } -
xe vdi-data-destroy does not free the space either.
[root@host3 sm]# xe vdi-list params=uuid,name-label,virtual-size,physical-utilisation,type type="CBT metadata" name-label=psrv34_data01 uuid ( RO) : ceb5a540-7d53-4614-be3e-b329d8d7336b name-label ( RW): psrv34_data01 virtual-size ( RO): 10737418240 physical-utilisation ( RO): 8388608 type ( RO): CBT metadata [root@host3 sm]# xe vdi-data-destroy uuid=ceb5a540-7d53-4614-be3e-b329d8d7336b [root@host3 sm]# xe vdi-list params=uuid,name-label,virtual-size,physical-utilisation,type type="CBT metadata" name-label=psrv34_data01 uuid ( RO) : ceb5a540-7d53-4614-be3e-b329d8d7336b name-label ( RW): psrv34_data01 virtual-size ( RO): 10737418240 physical-utilisation ( RO): 8388608 type ( RO): CBT metadata -
The current cbt backup is working but it contains several bugs that needs to be resolved. @florent is working on fixes but from what i understand it is difficult to be fixed. Hope there will be progress soon!
-
@rtjdamen I noticed this today as well. A VM within a pool migrated from one host to another, after which i received
Error: stream has ended with not enough data (actual: 449, expected: 512)Retrying the backup resulting it in running a Full Backup. Hopefully XO can start to handle this otherwise im not sure CBT backups are worth it over the old method for pools with multiple hosts. Running a Rolling Pool Update or Rolling Pool Reboot would guarantee every VM would fail and require a full backup? Or is this not expected behaviour?
-
@flakpyro no this is a bug, we see this on vms that are not migrated. In our other backup software we do not see this issue, using cbt there as well.
-
I cannot reproduce in my own production here, the only problem is when I have a job on 2 different pool, one pool doesn't have NBD enabled and re-doing a full. The NBD enabled pool works perfectly though
-
@olivierlambert From what i understand about this issue is it occurs when snapshot is deleted at the wrong time, although this is what florent told me. he created a fix for it but that does not resolve the issue.
-
I do not reproduce that problem here, so it's clearly something subtle and/or configuration dependent. Any other feedback from the community with CBT? More feedback will be helpful to pinpoint the issues left
-
@olivierlambert Maybe it was just a coincidence then that this happened after a migration. Backups with this VM had been working well for about a week until i migrated it to a different host yesterday.
-
@flakpyro we see this happening to a few vms (2 or 3) random on a large pool 400+ so i think it is a coincidence.
-
@flakpyro moving to another SR or just live migrating?
-
@olivierlambert Just live migrating.
So as a test today after the failure last night i ran a full backup with CBT enabled on 2 VMs, i then migrated both from Host 1 to Host 2 (using the same shared SR mounted on each host via NFS 4.1 (TrueNAS)). Tonight the backup failed with the same error:
Error: stream has ended with not enough data (actual: 449, expected: 512)If it would at all help i could open a ticket referencing this thread and enable the remote support tunnel. We can always go back to the old backup method but if i can help make CBT rock solid im always willing to help!
-
@flakpyro i did the same test at out end, live migration of a vm and no issue with the backup after. I can’t reproduce. We use iscsi so maybe the difference is there, i have seen cbt issues on nfs as well with our 3th party tooling. Will do some more testing tonight.
-
@flakpyro Did you move the VM with the snapshots? Or did you delete them before (or had CBT option on to autodelete them)?
I'd like to test this also.
-
@manilx I have the the option "Purge snapshot data when using CBT" in the backup job enabled.
The VMs themselves are staying on the same NFS SR shared between the two hosts in the same pool. I simply migrated the VM from host 1 to host 2 leaving the storage location itself unchanged.
-
Is your XO able to reach both hosts directly?
-
@olivierlambert Yes, the two hosts and XO are all on the same subnet.
-
I've seen some changes/updates on GH by @florent , but I can't tell much more on my side
We can try to test this very scenario though. -
@olivierlambert Using current XO (07024). I'm having problems with one VM and hourly Continuous Replication (Delta, CBT/NBD). The VM has three VDIs attached. Sometimes it works just fine. But it does error out at least once a day and transferring hundreds of gigs of data for its retries.
stream has ended with not enough data (actual: 446, expected: 512) Couldn't deleted snapshot data Disk is still attached to DOM0 VMOn the next CR job it works (and deletes snapshots) but normally leaves a task running from before on the pool master for the export (that's doing nothing, stuck at 0%). A tool stack restart is required to free it.
It's odd that it complains about the disk attached to Dom0 on a run where it leaves a Dom0 export task but then on the next run it works without error (and the VDI task still attached to Dom0).
I can't force the problem to happen, but it keeps happening.
-
@olivierlambert last week @florent implemented some changed code at our end, but this did not seem to resolve the issue, we are using a proxy for backups, i am not shure is the changed code was deployed to our xoa or xoa proxy, as we are still encountering the issue i have now changed the settings to not use the proxy, so far no stream errors yet. I still see the data destory vdi in use but that could be a different problem. I will keep an eye on it to check if indeed florent did not deploy the upgrade to the proxy and we did therefore not see the changes.
Anyone using proxy in this situations where the stream error occurs?
