Backup Fail: Trying to add data in unsupported state

[nmadunich]

I am having the same issue with backups. I get the same error Trying to add data in unsupported state and I see the same log entry that @nvoss posted. I think this is related to the backup encryption.

I did some testing after seeing this post. I created a new NFS remote and my VM backed up just fine without encryption. I had never successfully backed up this VM before. I deleted out the backup then cleared out the NFS file directory then added encryption. My VM failed to backup after adding the encryption.

My VM has only a single drive and it's Windows 10 and I have another Win10 VM that backs up fine under the same job. I don't think it's drive numbers or OS related.

I have some more testing tomorrow and will update my post.

Thanks
Nick

[olivierlambert]

So it seems to be something very specific with encryption, right?

[nvoss]

@olivierlambert that's my experience as well, but it's inconsistent in that I have like 6 other VMs on the same backup that work fine to the encrypted repos. In this case I have two -- one that's a local synology and one that's a remote s3 compatible Wasabi. Both show the same failure, but only with these 2 VMs.

[nmadunich]

@olivierlambert I can confidently say that it's an issue with encrypting the VM during backups. I was using 2 Windows 10 VMs for testing. 1 would fail every time I enabled encryption on the backup and 1 would backup with and without encryption enabled. I was able to reproduce this multiple times. I am still trying to figure out what the difference between these 2 VMs is that would cause this issue.

Both VMs are using BIOS boot firmware and I am running full backups. Both of these VMs were imported from VMWare and have the guest tools installed. We are using NetApp NFS storage and AWS S3 for our remotes so I don't think it's storage vendor related.

I updated XOA this morning from 5.95.1 to 5.95.2 and the backup still failed.

[olivierlambert]

Do you have the same behavior on XOA on latest release channel?

[nvoss]

@olivierlambert I'm using XO built from sources. Current commit level: 88c7c. I pulled those updates down ~2 days ago I think.

[nmadunich]

I updated XOA to the latest channel 5.97.0 and I still have the same issue. I can backup the VM without encryption and then it fails when encryption is enabled. We did purchase licensing from VATES so it's not an issue with being built from source.

[Delgado]

I can confirm I am seeing the same thing with full a mirror full backup job to Backblaze B2 utilizing encryption. I am on Master, commit 732ca in my homelab. I have attached the backup log as well.2024-08-28T22_32_37.271Z - backup NG.json.txt

[olivierlambert]

Thanks for the feedback, let me ping @florent internally

[olivierlambert]

@nmadunich and same on XOA stable?

[florent]

Hi,

what are the size of the failing VMs ? is there anything in the syslog before having the cipher error message ?

To be fair, uploading full backup ( > 50GB) , without knowing the size first is full of hurdles. And the xapi dont tell the size of the exports. Incremental backup with block storage completly circumvent this, by uploading separate blocks of known size

Regards

[nmadunich]

@olivierlambert Yes I get the same result on stable vs latest channel.

@florent The ones that fail do seem to be some of my larger VMs the Windows 10 VM that I have been testing with is about 88.3 GB used according to the OS.

All of my VMs are thin provisioned and our NetApp storage is using de-duplication so the size of the VHD on my storage is significantly less in this case it was about 3 GBs.

As a test I created a new storage volume without thin provisioning and de-duplication. I migrated the Disk to the new volume and the VHD is 103 GB. I also removed de-duplication and compression on my remote. I tried the backup again and it failed with the same error.

I do see some errors from the xensource.log around the time it fails and I attached those here.

xensource errors.txt

I am editing my post after looking at the log file @Delgado posted mine are slightly different. I added mine for comparison. At some point during my testing the error also changed slightly and started stating VDI must be free or attached to exactly one VM. It appears after a failed backup it's not cleaning up the snapshots.

2024-08-29T21_32_42.161Z - backup NG.json.txt

2024-08-28T15_05_12.613Z - backup NG.json.txt

[Delgado]

Hello,

My vms are about 150G each. I was using compression when I backed up the vm to the remote before mirroring it to the s3 bucket. I did end up changing to delta backups and the error did go away but I can create another normal backup and mirror it to the bucket again to see if I get the same results.

[olivierlambert]

That's interesting So it's only with full backup (XVAs) then.

[nvoss]

@olivierlambert yeah my experience is also that deltas run without error. Though what they're backing up exactly w/o a full in the remote is pretty questionable. I assume its a delta off of the snapshot full, where the snapshot is completed without issue and it's just the copy to encrypted remote that's failing.

These are definitely my larger VMs -- >100gb total disk.

[daniel.grimm]

Hi,

Same problem here.
Its an encrypted S3 remote to Backblaze.
Full mirror backup with selected VMs.
Small VM like Xen-Orchestra works. As soon as a large VM is added (approx. 500GB), the error occurs after about 3 hours.
Tried several times.

xen-orchestra build from source

[olivierlambert]

And with or without backup compression?

[daniel.grimm]

sorry... i forgot...
with zstd compression

[olivierlambert]

Can you try without it and report?

[daniel.grimm]

Yes.

iam making a non compressed backup now.
And then i try to mirror it with the mirroring job.

Report follows... But uncompressed backup and upload will need some time