Hi All!

We were having problems with our backup remotes not working (on-site Synology, off-site Wasabi) with vdi chain issues. Checked the logs per the referenced article and didn't get anywhere with anything obvious. Noticed we were using an older encryption method with the encrypted remotes. So decide to purge the remotes, setup fresh, and off we go.

Fast forward a week. Full backups went ok on a forced full. We'll see this weekend if it goes well automatically. However deltas continue to fail when run on schedule with the new remotes. I figured for sure that new backups, new snapshots, etc. wouldn't have a coalescing issue. What we've found so far though is "force run" results in a successful backup for the deltas too.

At a bit of a loss on troubleshooting. Anyone else seeing this? Both remotes are encrypted.

Thanks!
Nick

@olivierlambert @florent

Of note from ours is we use Wasabi S3-compatible as the remote in one case and a Synology NAS as our local remote in the other. Both of those remotes fail with the unsupported state error when the backups are encrypted.

In the same encrypted job I have the following machines which have a backup size and duration of:

VM1 - 31.55GB - 47 mins
VM2 - 14.51GB - 22 mins
VM3 - 30.28GB - 48 mins
VM4 - 45.33GB - 24 mins
VM5 - FAIL - 1hr 27 min
VM6 - 2.14GB - 4 mins
VM7 - FAIL - 1hr 28 min
VM8 - 35.95GB - 1hr 5 min

The two machines erroring have thin provisioned disks whose size are
VM5 -- 128GB and 100GB which are 10.94GB and 86MB on disk
VM7 -- 123GB and 128GB which are 11.09GB and 10.3MB on disk

At first I thought it was size related or perhaps duration. But what's causing that extra duration for machines of these sizes? Something about activity on the Windows VMs?

Or perhaps that it was related to having multiple disks on Windows machines?

@olivierlambert yeah my experience is also that deltas run without error. Though what they're backing up exactly w/o a full in the remote is pretty questionable. I assume its a delta off of the snapshot full, where the snapshot is completed without issue and it's just the copy to encrypted remote that's failing.

These are definitely my larger VMs -- >100gb total disk.

@olivierlambert I'm using XO built from sources. Current commit level: 88c7c. I pulled those updates down ~2 days ago I think.

@olivierlambert that's my experience as well, but it's inconsistent in that I have like 6 other VMs on the same backup that work fine to the encrypted repos. In this case I have two -- one that's a local synology and one that's a remote s3 compatible Wasabi. Both show the same failure, but only with these 2 VMs.

@olivierlambert @florent happy to provide logs or troubleshoot any way that makes sense. The issue definitely seemed to crop up after setting up encrypted repos but that's not to say its the culprit by any stretch. Could easily be coincidental.

We've got a host of machines backed up via tag selection. Most of them are fine, but during a full backup I have two that fail everytime. The error is "Trying to add data in unsupported state"

These machines happen to have more than 1 disk associated with them whereas all of our other VMs only have a single OS drive. That's the only thing I've got as a potential issue.

Stack trace has some more info in the JSON log, but I'm not sure where to start with it:

Error: Trying to add data in unsupported state
    at Cipheriv.update (node:internal/crypto/cipher:181:29)
    at /opt/xo/xo-builds/xen-orchestra-202408260910/@xen-orchestra/fs/dist/_encryptor.js:52:22
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async pumpToNode (node:internal/streams/pipeline:135:22)

The machines are a Windows Server 2022 VM and a Windows 11 VM.

Final error is all targets have failed, step: writer.run()

I've tried running the backup on one of these machines when it was shutdown and it had the same ultimate error. The Windows management tools are installed on both machines.

Anyone seen this before and had success at resolution?

Thanks!
Nick

ETA: these machines are set to be on an encrypted backup. Other machines on the same backup work fine. CBT is off.