That worked; thank you!!!

@nraynaud Bizarre isn't it; I'm so very grateful for your efforts.

Some more news - it seems that one big challenge is around concurrency - things improve dramatically if concurrency is set to 1. As soon as something else is running in parallel, we run into the socket failures. I'm expanding things to try your change on another box to see if the outcomes are different - but in summary what I'm seeing so far:

• Concurrency = 1 - works fine first time, fails occasionally (20% of the time?) thereafter
• Concurrency > 1 - almost impossible to get it to run, but sometimes one or two VMs backup ok but not enough to be predictable and never the entire group
• Anything fails - impossible to get a clean run again until the S3 target has been cleaned entirely

So it appears that somewhere there perhaps is a lock occurring when more than one stream is running, and additionally there's some kind of conflict when things have terminated prematurely and the target is therefore not in its expected state on the next run.

@olivierlambert Client is a proxmox shop and whilst tolerating services running in XCP-ng / XO world they require that if everything fails they have the ability to pick up the pieces in-house. Proxmox has easy import of .ova but very manual process for .xva hence....

Correct - through the CLI one could see that the old VM that had been migrated from Host1 to Host2 was still showing under Host1, whilst the migrated copy was showing as running on Host2. Once the Host1 remnant of the migration was removed that cleared things and XO correctly reported the VM as running on Host2 with its disks attached.

TLDR - There were no other conflicts beyond what appeared through XO to be the only version sitting halted on Host1, but through the CLI one could see the halted copy on Host1 and the running copy on Host2. Somehow the running version did not show in XO until the remnant was removed.

Thanks for your help @olivierlambert

Always

Is there any intent to publish the latest xcp-ng 'release' with an XOSTOR iso ? There's an iso for the non-XOSTOR version (xcp-ng-8.3.0-20250606.2.iso) released on 18 Dec 2025, but the latest iso with xostor comparability is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso released in 16 June 2025.

Reason for asking is the last incremental upgrade on 18th Dec partially failed on our pool master and so we need to do a 'clean' upgrade, however there are XOSTOR disks on that machine, and doing a network upgrade after a partial failure and regardless with xostor - is not advised / achievable.

Thank you!

@florent Can't wait!

Ok, spent weekend having backups running continuously across a number of boxes.

Good news - the fix seems to have solved things, providing one only ever uses concurrency “1” and there are no conflicting or overlapping backups.

Restores are working fine for me too.

In short @nraynaud - it’s a substantial improvement and for me makes this now usable. A huge thank you.

@dustinb Concur 100%; my current focus is on confirming the error doesn't reoccur and understanding the change in what I'm seeing compared with previous backups, before this goes into production I 100% agree it should be tested for restores. I shall do so myself once I've got confidence that the symptom has been resolved.

I'll try a fresh install over the next couple of days and see if it reoccurs. Looking at the other boxes, I have the same error on one of the other hosts, but it's not across all of them despite identical installs and hardware.

Thanks for your efforts and help, shame there wasn't an easy answer but let's see if it reoccurs after a completely fresh install.

@olivierlambert Client is a proxmox shop and whilst tolerating services running in XCP-ng / XO world they require that if everything fails they have the ability to pick up the pieces in-house. Proxmox has easy import of .ova but very manual process for .xva hence....

@florent 100% agree, sadly not an option in this particular scenario. Had hoped for a hidden switch that would allow backups to be downgraded to ova but tbh the API will do the job just fine.

Thanks all; appreciated

@MajorP93 Yeah, that's me being daft. I confess I took "consider rebooting the hosts, starting with the pool master' as a parallel not serial task; my bad and my pain as a result.

Fair point sir

As an observation ; I'm going to draw attention to @majorp93's point about rebooting the servers after ALL nodes have been upgraded.

Historically we would move all VMs off master, upgrade master, restart its toolstack, then reboot master, then move VMs from Node 1 to master so we could begin the upgrade on Node1. Normally works ok but last time around it caused all sorts of problems. Previously it had felt right to upgrade master in its entirety including the reboot before moving on to the next host and rinse, repeat - but this cost us a lot of time, corruptions and pain.

TLDR: Perhaps add a footnote to the docs that when upgrading a pool the reboots should take place as a final step across the pool only after all nodes have been updated.

Wondering if there was a simple way to backup (ideal) or manually export as OVA to a remote NFS ?

Doing so as .xva works just fine using the XO backup functions, but for our limited Use Case we need each export as .ova and before we script something was wondering if anyone had any clever ideas?

Thanks!

That worked; thank you!!!

@stormi You star, will let you know. Thank you!

@stormi https://xcp-ng.org/forum/post/101015 V kind, thanks

As per chat in https://xcp-ng.org/forum/post/100993 ; have had a failed update in one our of pools that necessitates using a local ISO to update XCP-ng , however as the pool is using XOSTOR one cannot use the standard release (in this instance xcp-ng-8.3.0-20250606.2.iso published on 18 Dec '25) and instead the update requires the XOSTOR packages to be in that latest iso.

(Context: Due to the issue occurring during an update, we need an iso with the latest patch levels as the update has partially completed leaving the system with updated new packages. We can't use the older xostor friendly iso as there the linstor version is at 1.29.0 whilst the system now has linstor 1.29.2, and we can't use any other route (e.g network or cli) to update as these are not supported for xostor implementations.)

The last xcp-ng iso with XOSTOR packages is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso which was released on 16 Jun '25. Is there a possibility of building an XOSTOR iso image each time we build a new xcp-ng iso so that the two are in sync (or even have a common iso)?

And in the short term is there a way that I or someone cleverer could generate the iso?

Appreciate any help; thank you!

@stormi As a footnote to earlier message, we tried using the previous iso image (xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso) to see if things would magically work out but it results in an unrecoverable error - "Cannot upgrade host with LINSTOR 1.29.2-1.e17_9, upgrade repository has versions 1.29.0-1.e17_9. Please make sure your pool is uptodate [sic - typo in error message] and use the latest dedicated ISO."

So yes, afraid I'm after an iso with 1.29.2...