CBT: the thread to centralize your feedback

[olivierlambert]

@rtjdamen Yes

[rtjdamen]

@olivierlambert just updated, we will run some jobs with snapshot destroy and will report the results

Compliments to all involved fot this great improvement is such a short time!

[manilx]

@Andrew

How do I get rid of this one?

[manilx]

@manilx restart toolstack did it. Thx for pointing this out

[olivierlambert]

@rtjdamen That was a big sprint

[rtjdamen]

@olivierlambert indeed but the first version is here!

I allready noticed some strange behavior, when a backup job is running with both switches on (cbt and data destroy), the first run the job processes succesfull but some snapshots stay orphan after the job is finished and coalesced. @florent any idea what is causing this?

[Andrew]

@olivierlambert With today's update (feat: release 5.96.0, master 96b7673) things are working well again. I have not seen any Orphan VDI/snapshots or stuck Control Domain VDIs (but it may take time to test). Thanks to the Vates team!

[rtjdamen]

@Andrew need to do some more testing but second run does clean them out, keep u all posted on the results over the weekend!

[rtjdamen]

I did some digging on this orphan snapshots. i found this error in the backup logging for these vms

Couldn't deleted snapshot data
error
{"code":"VDI_IN_USE","params":["OpaqueRef:d52c23f7-4008-45e0-9852-6ecb3aeb8567","data_destroy"],"call":{"method":"VDI.data_destroy","params":["OpaqueRef:d52c23f7-4008-45e0-9852-6ecb3aeb8567"]}}
vdiRef
"OpaqueRef:d52c23f7-4008-45e0-9852-6ecb3aeb8567"

i do not see them currently on normal vms and also not on second run

[ajpri1998]

It worked just fine for me during an overnight unattended backup. I also did a manual backup to watch things.

Couple questions:

1. If NBD is used, is there a 100% chance that CBT will also be used, or is it possible to have a NBD backup without CBT?
2. Can it be added to the xo-server-backup-reports emails that NBD/CBT was used?

[rtjdamen]

This post is deleted!

[rtjdamen]

It seems the orphan vdi is showing when the data_destoy fails, maybe the vdi is not yet ready? @florent Is this something you can look into? Maybe a retry when it fails?

[Tristis Oris]

most of backup succed, but few suddenly looks like:

[rtjdamen]

@ajpri1998

From what i have seen if CBT fails for some reason, the chain is broken or the metadata is missing it will fall back on single NDB, most likely a new full.

[probain]

Enabling the new Purge snapshot data when using CBT on Delta backups
results in
cleanVm: incorrect backup size in metadata. But it seems to be successful anyway.
Update: I'm getting this regardless of what "Purge snapshot data when using CBT" is set to. So I'm right now not sure about what causes it.

XOsource: commit 96b76. ?

[Tristis Oris]

[NOBAK] tag is ignored with CBT patch.

before update it works fine. no 2nd disk available for restore.

my share is run out of space)

[Tristis Oris]

@Tristis-Oris is it any way to remove only 1 disk backup chain, to not wait rotation? i can't just clear everything.

[olivierlambert]

@ajpri1998 IIRC:

1. CBT will be enabled as soon you use NBD, no other choice (having 2 different code paths was too prone to problems and bugs, without forgetting the initial VHD diff code)
2. Sure, we can do that! Telling @Bastien-Nollet about it

[Delgado]

I started seeing this error this afternoon.

The failed job also happened to run after a monthly backup job that I have setup that does a full backup to some immutable storage that I have but I am not sure if it is related I just wanted to add that information. Below is the log from the failed backup.
2024-07-07T11_00_11.127Z - backup NG.json.txt

[Andrew]

@Delgado I saw those errors when I disabled/enabled CBT on disks or changed the NBD+CBT option in the backup job. If you have more than one job that works on the same VM you need to have the backup options the same.

I think that migrating the VDI to a different SR also changes the CBT setting.