@florent Finally getting back to this post, I know it's been months, just haven't had a lot of time, sorry!

I think I am still a bit confused, but I will do some additional testing to see if I can confirm my suspicions.

My confusion is that, you can't merge the deltas into the key if the key is locked behind Object Lock, the file isn't writable so you can't do the merge operation, right?

So that being said, it sounds to me like maybe the retention of the object lock/immutability needs to be set to be less than the retention period in XOA, right?

This way the original key is not immutable and can be written to when the merge happens?

Or does XOA just "wait" until the key isn't locked and then do the merge operation?

As an update here, rebooting did resolve the issue, I thought I did so shortly after I did the restore but maybe I forgot to do so.

Anyway, rebooted yesterday and the backups ran last night just as they should.

Perfect, thank you 👍

@planedrop
I usually don't use SMB for remotes - prefer for stability reasons NFS.
But in the past on customer systems we had, from time to time, the problem that the SMB Client/Server (both had the problem, so it might be the case that they stalled each other) processes stalled for a moment, which might cause that file locks are not set or ended correctly.

Maybe it's something like this in your case too.
Catching that is a bit of a pain in the *** , because you need to do process memory tracking on both sides to see if and when they stall for a short moment.

Not sure if there is a way under TrueNAS to check if a file has a lock set or not. If possible it might be already enough to remove the lock of the file (if possible via cli) to make it visible to the XCP-NG host again.

@olivierlambert Sounds good, thanks as always! Not a big deal by any means, just figured I'd mention it.

@florent OK good to know, thank you! I will do what I can to replicate this issue in my lab and then see if changing to block backups fixes it, just want to try and avoid changing things too much in the prod environment.

Appreciate the help!

@olivierlambert Gotcha, totally agree, super low on the list of things to do.

I do wonder if maybe a note should be added somewhere saying you'll have to manually rename the files after importing? Maybe hear near the note about ISO importing?

84aee289-b7e6-4bcc-a1b8-1edfbe5ba107-image.png

@Andrew This was the answer, thank you! I had it set with 8MiB of video memory which was fine with an older install but 22.04 needs more it looks like. Bumped that up to 16 MiB and it's super smooth now.

Thanks for your help!

@olivierlambert increasing the video memory was the solution. But to answer your question this was in BIOS mode rather than UEFI.

@olivierlambert Yeah no worries, was a minor issue overall anyway!

Just figured I'd make my own post about it since the other post mentions CR but not Disaster Recovery, the fix definitely works for both though!

This is all so cool, love to see a platform under this level of development.

Assuming I have the setting in the correct place, I'm not seeing any significant difference whether the setting is there or not.

without setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.5 MiB/s

with setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.87 MiB/s

CPU usage appears to be about the same.

@caponate yup, I ended up doing the same

@olivierlambert This did it, thank you so much!

@Andrew Good suggestion, might as well pick some up, thanks!

Update candidate available to fix certificate download: https://xcp-ng.org/forum/post/49373

@andyhhp well I took way longer than I said I would, but I promise I still wanna work on this lol.

Here is the link to the thread that shows my crash reports. As a reminder, this crash happened on this host randomly and didn't seem directly related to PCI passthrough, or at least not the driver install part which I was having issues with.

https://xcp-ng.org/forum/topic/5900/host-crash-once-in-a-long-while

@Andrew Wanted to go ahead and add that it didn't seem slow to me to do the export, it was only 2GB, but still. It did take a bit to ramp up speed though, I got 4.5KB downloaded and then it sat there for a few minutes, but once it started downloading more data it sped up and the whole 2GB download was done in like 1-2 minutes, not crazy fast but not super slow either.

I will try and test this with a bigger VM when I get the chance to see how it behaves.