Backup-NG Error "invalid header checksum" after VM-Disk-Resize and reaching retention Limit.

AlexD2006

Hi all,

first of all my Environment:
XCP-NG 7.6 Pool (4 Members)
Shared (LVM) Block-Storage (FC and iSCSI)
XenOrchestra (from sources, 5.42.1 on Ubuntu 18).
Backup-Remote via NFS.
Nightly Delta Backup 00:30 - Retention 8

From time to time i have the problem of getting an Error on one or more VMs in my Delta-Backup-Job.
Error (example): "invalid header checksum 1143961443, expected 4294883451"

I have seen this over the last few Months about 10 to 15 Times on different VMs.
(a working solution is to delete/move the existing backups of this VM, so it starts with a new full one)

After some investigation we now found the cause of the problem.
Whenever we resize one or more disks of a VM, all looks good for that moment.
But when the retention Limit is reached, the Backup of the VM(s) with a former resized Disk will fail with error "invalid header checksum".
All later Backups will even fail, so the existing Backups will remain from the day of the resize till the reach of retention.
(in my case, Retention 8. The VM today shows 8 restorable Backups from 01.06. to 08.06. - The Disk was resized on 31.05.)

Does anyone else expieriance the same Problem?
Might this be an exeption that is simply not intended?
Or could it be that i missed some dependencies in my from source built?

Kind regards
Alex

CIT

Hi,

I know this is an 'old' topic, but would like to confirm this bug as we have the same with one of our clients servers, days after expanding the VM disk.

Stil need to move the old backup (it's copying as we speak) and assume after I delete the original backups it will work again.
Will need to see if this still happens in the latest version of XOCE and in XOA.

CIT

I can confirm that Alex's workaround does work.

akurzawa

@AlexD2006 said in Backup-NG Error "invalid header checksum" after VM-Disk-Resize and reaching retention Limit.:

Shared (LVM) Block-Storage (FC and iSCSI)

Little offtop
what hadrware/software You've used to create shared storage?

olivierlambert

Ping @julien-f

I think we tried to reproduce the error without success. Can you tell the exact steps to get the bug in the first place? How big the disk were before and after? How much it was filled?

CIT

In our case it was (fairly certain) 300 GB before and 600 GB after on EXT3 Storage.
I don't know how full the disk was... but we needed to expand is for a reason, so it should have been close to full I guess

We increased the disk size, everything was fine, until the error popped up a week later, then started to search and found this thread...

olivierlambert

So the issue appeared just when the oldest extended delta had to be merged in the full I suppose. But so far we could not reproduce this. We'll retry with a bigger disk to see. Ping @julien-f

julien-f

@olivierlambert said in Backup-NG Error "invalid header checksum" after VM-Disk-Resize and reaching retention Limit.:

So the issue appeared just when the oldest extended delta had to be merged in the full I suppose.

This is exact, but we need to have a proper process to reproduce it.

This might be related to:

increasing the size of the disk above a certain threshold
increasing the content of the disk above a certain threshold