@Bastien-Nollet Thanks as always @florent

@Danp I haven't yet, I will do that when I have a free moment and report back.

This VM has 8GB of RAM though, and so far it only seems to happen when viewing the console. I've only tested with 1 VM so far, but will try a few more to see if it's the console view of 1 VM causing it.

It's been about 3 hours of not viewing the console and it's stayed connected, so at least that's good news.

I'll try to get back today after checking those logs, just didn't want to leave this thread empty.

Thanks for the help!!

I have a production environment that I moved to 5.110 yesterday, this resolved the backup errors I was seeing, but now I am seeing the XOA disconnect from my hosts periodically.

So far it seems to primarily be while using the Console on a VM, though I am not 100% sure, it was stable all last night and just started happening while I was consoled into a Win10 VM.

Going into the servers tab and just disconnecting and reconnecting fixes it each time, but it's still odd so I figured I would report it.

@Bastien-Nollet I somehow missed the email about this.

But wanted to come say thanks on this thread anyway. The update fixed the issue, great work!

@Bastien-Nollet Thanks for the suggestion, though I think I am going to avoid doing that in a production setup.

I'll look forward to it being included in a newer version.

Our backups run nightly and it's random which VMs fail with this error, but so far it's almost never been the same one each night, so at worst we are 24hrs behind on a single VM backup which isn't a huge deal.

Our other nightly backup job to S3 works just fine too so we're covered.

Thanks!

@olivierlambert This is a production setup, rather than my lab, so I don't want to use anything that has a chance of being unstable.

Do we know what version this change may end up in? I'm happy to try the Latest channel if/when this fix is in place.

Apologies if I am misunderstanding anything.

@olivierlambert @florent wanted to post an update here. I upgraded to 5.108.1 in this environment and my SMB backups are doing the same thing with the error just being Footer1 !== !Footer2

It seems about half of my backups fail, so it's not all VMs.

I will try to watch it tonight to see if it's possibly a concurrency issue where it's doing all of them at the same time instead of taking my 2 concurrency setting in place.

Let me know if you want any additional info and I can also submit a ticket.

Wanted to post a quick update, it's been over a week now and the backups have been 100% successful.

Figured as such, but thought it was worth at least coming back here and confirming.

@ravenet Yeah another night of successful backups so I think going back to Stable did fix the issue. 2 for 2 on that now.

@ravenet All of my errors seemed related to NBD access, so if the concurrency setting was being ignored, that might be the source of the issue I was seeing.

I'll watch my lab as well and see if the concurrency is being respected or not on the latest from the sources build.

Glad to see you were on 8.3, so not related to me being on 8.2.

@olivierlambert Gotcha. I'll see if I can get this issue to replicate in my lab at all but so far my backups have been smooth over there.

I'll try to re-create more similar backup jobs in the lab as well, maybe it's a specific setting or something on my jobs.

@olivierlambert Happy to help in any way that I can as well!

Notably, I am not seeing any issues doing backups to SMB or S3 with my lab at home which is on the latest. My lab is XCP-ng 8.3 though, rather than 8.2 like this production setup (which will be getting upgraded to 8.3 now that it's LTS), so maybe something specific with the new backup code and 8.2?

@ravenet @olivierlambert yeah going back to 5.106 seems to have resolved the issue. I want to give it one more day before saying 100% that it did, but all VMs in both my backup jobs last night finished properly.

@olivierlambert I will give this a shot and report back. It may be a day or so, one of the backups is still running (very large VM over S3 so takes a while) but once it's done I will go back and see if the failures go away.

@olivierlambert Good question, I am on Latest by mistake in this environment actually.

Is it safe to roll back to Stable channel even though I am already on latest?

Still seeing this issue, trying to pinpoint it but haven't had any luck. It seems like each VM is about a 50/50 chance if it fails or succeeds, but the logs don't really lead me to anything and there's no consistent reason why it would be happening that I can find.

This is only since going to 5.107.2 as well, wasn't happening on the previous version (which I unfortunately don't recall the version number of).

@olivierlambert This is great, thanks for letting us know! I'll give this a shot in my lab as soon as I can.

Going to give it more time, but restarting all the backups seems to have fixed the issue. Unsure why they would fail once and then resume just fine though.

This is a new one, just updated XOA to 5.107.2 and now my backups are no longer working.

I have support and can put in a ticket, but figured it's better to try here first.

I am getting an error: Fail to connect to any Nbd client on the backups to Backblaze and on my SMB backups I just get a Footer1 !== footer2 error.

What's important here is that it's only about half my VMs, and this is a single host setup, so the NBD client issues don't really make sense to me, unless I'm misunderstanding something about NBD.

Anyone else seeing issues with backups after this update?

Also not seeing anything consistent, not like an issue with Windows VMs in specific, it seems random.

@olivierlambert Yeah so far backups have been fast enough to not pose some huge issue though.

IMO if you have a huge VM (many TB) it should just be dealt with on a NAS or something instead of a VHD.

Still glad that qcow2 is coming though!