CBT: the thread to centralize your feedback

rtjdamen

Florent provided us with a fix for the Stream error, so far i did not see it on new runs, we keep an eye on the running jobs. The issues we have remaining is the vdi_in_use error and sometimes the proxy loses connection with the XOA master server in that case the jobs shows as interupted but in real it is still running and finishing the job.

manilx

@olivierlambert It looks like, different from previous versions, the backup keeps one snapshot for each schedule. Was just one for all three before.
Does this seem OK?

olivierlambert

I don't think it was possible to keep one snapshot for many schedules. Because a snapshot is the point of reference, so if your schedules aren't aligned in terms of start date, we must have multiple snapshots (regardless the fact we keep them visible or using CBT to remove them).

manilx

@olivierlambert This would be OK with me. Just that it wasn't like that (pre-CBT) before. I only had one snapshot for the backup job......

olivierlambert

I don't see how that would have been possible in the first place

manilx

@olivierlambert Don't know either but that is how it was

olivierlambert

You can try to go back on XOA stable and see if it's really the case, but I don't see how we could have different schedules not relying on a specific snapshot if they are at different period of time.

manilx

@olivierlambert Running XO from sources. I'm good as it is, no need to go back. Was just wondering why I had multiple snapshots now instead of the single one (per backup job). Thought of a bug but as it's a feature I'm good.

robyt

Too many error for me..
Time for backup very very long, error at start (with retry) and at the end (with error on backup:

Couldn't deleted snapshot data
Couldn't deleted snapshot data
Retry the VM backup due to an error
the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution.
Retry the VM backup due to an error
Transfer data using NBD
will delete snapshot data
Snapshot data has been deleted
Snapshot data has been deleted
Transfer data using NBD

    Clean VM directory
    cleanVm: incorrect backup size in metadata
    Start: 2024-07-11 20:23
    End: 2024-07-11 20:24
    Snapshot
    Start: 2024-07-11 20:24
    End: 2024-07-11 20:25
    Backup XEN OLD
    Start: 2024-07-11 20:25
    End: 2024-07-11 20:27
    Duration: 2 minutes
    Clean VM directory
    cleanVm: incorrect backup size in metadata
    Start: 2024-07-12 01:44
    End: 2024-07-12 01:44
    Snapshot
    Start: 2024-07-12 01:44
    End: 2024-07-12 01:46
    Backup XEN OLD
        transfer
        Start: 2024-07-12 01:46
        End: 2024-07-12 05:39
        Duration: 4 hours
        Size: 397.86 GiB
        Speed: 29.12 MiB/s
    Start: 2024-07-12 01:46
    End: 2024-07-12 05:39
    Duration: 4 hours

Start: 2024-07-11 20:23
End: 2024-07-12 05:39
Duration: 9 hours
Error: Disk is still attached to DOM0 VM
Type: delta

not good..

rfx77

Snapshots not getting deleted most of the time. Purge Snapshot data is enabled.

You can see the error in the picture below

olivierlambert

The question is about the root cause of the issue: why the VDI is still in use? Before trying again, try to restart the toolstack on all hosts, checking you have 0x VDI attached to the control domain or something like that.

rfx77

@olivierlambert

No they are not attached.

on the last job there was no error in the log but the vdi snapshots are not going away (see snapshots in picture above)

olivierlambert

Yes, if they can't be removed because they are used somewhere, there is a problem.

Also, I see you are using XenCenter, but that's not the right tool here. Check the Dashboard/health in XO to see.

rfx77

@olivierlambert

The snapshots are also show in XO

Subsequent delta backups are working and not leaving orphaned snaps

olivierlambert

That's not what I'm talking about in my previous post.

rfx77

@olivierlambert

why are the snapshots of psrv30_base and psrv30_data01 kept after a full backup???

Here what you wanted me tho check:

olivierlambert

So first, you still have VDI attached to the control domain. Outside the actual backup jop/export, it's NOT normal to have those here.
The snapshots you are showing a snapshots without any data, just the metadata. We can improve the UI to detect it and show it's not a regular VDI snapshot. If you don't see the VM snapshot in the VM/snapshot view, it's OK.

olivierlambert

Created https://github.com/vatesfr/xen-orchestra/issues/7837

olivierlambert created this issue in vatesfr/xen-orchestra

open Hide or discriminate metadata VDI in the SR/disk view #7837

rfx77

@olivierlambert

Another Backup is running
So to summarize: You always get (in th UI) one snapshot which takes no space if you do delta with CBT. The snapshots can be ignored. Is this right?

olivierlambert

Yes. It's not taking any space and won't prevent coalesce immediately after data are purged.