Backup Fail: Trying to add data in unsupported state

nvoss

We've got a host of machines backed up via tag selection. Most of them are fine, but during a full backup I have two that fail everytime. The error is "Trying to add data in unsupported state"

These machines happen to have more than 1 disk associated with them whereas all of our other VMs only have a single OS drive. That's the only thing I've got as a potential issue.

Stack trace has some more info in the JSON log, but I'm not sure where to start with it:

Error: Trying to add data in unsupported state
    at Cipheriv.update (node:internal/crypto/cipher:181:29)
    at /opt/xo/xo-builds/xen-orchestra-202408260910/@xen-orchestra/fs/dist/_encryptor.js:52:22
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async pumpToNode (node:internal/streams/pipeline:135:22)

The machines are a Windows Server 2022 VM and a Windows 11 VM.

Final error is all targets have failed, step: writer.run()

I've tried running the backup on one of these machines when it was shutdown and it had the same ultimate error. The Windows management tools are installed on both machines.

Anyone seen this before and had success at resolution?

Thanks!
Nick

ETA: these machines are set to be on an encrypted backup. Other machines on the same backup work fine. CBT is off.

olivierlambert

Hmm weird, question for @florent in case it rings a bell

nvoss

@olivierlambert @florent happy to provide logs or troubleshoot any way that makes sense. The issue definitely seemed to crop up after setting up encrypted repos but that's not to say its the culprit by any stretch. Could easily be coincidental.

nmadunich

I am having the same issue with backups. I get the same error Trying to add data in unsupported state and I see the same log entry that @nvoss posted. I think this is related to the backup encryption.

I did some testing after seeing this post. I created a new NFS remote and my VM backed up just fine without encryption. I had never successfully backed up this VM before. I deleted out the backup then cleared out the NFS file directory then added encryption. My VM failed to backup after adding the encryption.

My VM has only a single drive and it's Windows 10 and I have another Win10 VM that backs up fine under the same job. I don't think it's drive numbers or OS related.

I have some more testing tomorrow and will update my post.

Thanks
Nick

olivierlambert

So it seems to be something very specific with encryption, right?

nvoss

@olivierlambert that's my experience as well, but it's inconsistent in that I have like 6 other VMs on the same backup that work fine to the encrypted repos. In this case I have two -- one that's a local synology and one that's a remote s3 compatible Wasabi. Both show the same failure, but only with these 2 VMs.

nmadunich

@olivierlambert I can confidently say that it's an issue with encrypting the VM during backups. I was using 2 Windows 10 VMs for testing. 1 would fail every time I enabled encryption on the backup and 1 would backup with and without encryption enabled. I was able to reproduce this multiple times. I am still trying to figure out what the difference between these 2 VMs is that would cause this issue.

Both VMs are using BIOS boot firmware and I am running full backups. Both of these VMs were imported from VMWare and have the guest tools installed. We are using NetApp NFS storage and AWS S3 for our remotes so I don't think it's storage vendor related.

I updated XOA this morning from 5.95.1 to 5.95.2 and the backup still failed.

olivierlambert

Do you have the same behavior on XOA on latest release channel?

nvoss

@olivierlambert I'm using XO built from sources. Current commit level: 88c7c. I pulled those updates down ~2 days ago I think.

nmadunich

I updated XOA to the latest channel 5.97.0 and I still have the same issue. I can backup the VM without encryption and then it fails when encryption is enabled. We did purchase licensing from VATES so it's not an issue with being built from source.

Delgado

I can confirm I am seeing the same thing with full a mirror full backup job to Backblaze B2 utilizing encryption. I am on Master, commit 732ca in my homelab. I have attached the backup log as well.2024-08-28T22_32_37.271Z - backup NG.json.txt

olivierlambert

Thanks for the feedback, let me ping @florent internally

olivierlambert

@nmadunich and same on XOA stable?

florent

Hi,

what are the size of the failing VMs ? is there anything in the syslog before having the cipher error message ?

To be fair, uploading full backup ( > 50GB) , without knowing the size first is full of hurdles. And the xapi dont tell the size of the exports. Incremental backup with block storage completly circumvent this, by uploading separate blocks of known size

Regards

nmadunich

@olivierlambert Yes I get the same result on stable vs latest channel.

@florent The ones that fail do seem to be some of my larger VMs the Windows 10 VM that I have been testing with is about 88.3 GB used according to the OS.

All of my VMs are thin provisioned and our NetApp storage is using de-duplication so the size of the VHD on my storage is significantly less in this case it was about 3 GBs.

As a test I created a new storage volume without thin provisioning and de-duplication. I migrated the Disk to the new volume and the VHD is 103 GB. I also removed de-duplication and compression on my remote. I tried the backup again and it failed with the same error.

I do see some errors from the xensource.log around the time it fails and I attached those here.

xensource errors.txt

I am editing my post after looking at the log file @Delgado posted mine are slightly different. I added mine for comparison. At some point during my testing the error also changed slightly and started stating VDI must be free or attached to exactly one VM. It appears after a failed backup it's not cleaning up the snapshots.

2024-08-29T21_32_42.161Z - backup NG.json.txt

2024-08-28T15_05_12.613Z - backup NG.json.txt

Delgado

Hello,

My vms are about 150G each. I was using compression when I backed up the vm to the remote before mirroring it to the s3 bucket. I did end up changing to delta backups and the error did go away but I can create another normal backup and mirror it to the bucket again to see if I get the same results.

olivierlambert

That's interesting So it's only with full backup (XVAs) then.

nvoss

@olivierlambert yeah my experience is also that deltas run without error. Though what they're backing up exactly w/o a full in the remote is pretty questionable. I assume its a delta off of the snapshot full, where the snapshot is completed without issue and it's just the copy to encrypted remote that's failing.

These are definitely my larger VMs -- >100gb total disk.

daniel.grimm

Hi,

Same problem here.
Its an encrypted S3 remote to Backblaze.
Full mirror backup with selected VMs.
Small VM like Xen-Orchestra works. As soon as a large VM is added (approx. 500GB), the error occurs after about 3 hours.
Tried several times.

xen-orchestra build from source

olivierlambert

And with or without backup compression?