That worked; thank you!!!

@nraynaud Bizarre isn't it; I'm so very grateful for your efforts.

Some more news - it seems that one big challenge is around concurrency - things improve dramatically if concurrency is set to 1. As soon as something else is running in parallel, we run into the socket failures. I'm expanding things to try your change on another box to see if the outcomes are different - but in summary what I'm seeing so far:

• Concurrency = 1 - works fine first time, fails occasionally (20% of the time?) thereafter
• Concurrency > 1 - almost impossible to get it to run, but sometimes one or two VMs backup ok but not enough to be predictable and never the entire group
• Anything fails - impossible to get a clean run again until the S3 target has been cleaned entirely

So it appears that somewhere there perhaps is a lock occurring when more than one stream is running, and additionally there's some kind of conflict when things have terminated prematurely and the target is therefore not in its expected state on the next run.

Correct - through the CLI one could see that the old VM that had been migrated from Host1 to Host2 was still showing under Host1, whilst the migrated copy was showing as running on Host2. Once the Host1 remnant of the migration was removed that cleared things and XO correctly reported the VM as running on Host2 with its disks attached.

TLDR - There were no other conflicts beyond what appeared through XO to be the only version sitting halted on Host1, but through the CLI one could see the halted copy on Host1 and the running copy on Host2. Somehow the running version did not show in XO until the remnant was removed.

Thanks for your help @olivierlambert

Always

Is there any intent to publish the latest xcp-ng 'release' with an XOSTOR iso ? There's an iso for the non-XOSTOR version (xcp-ng-8.3.0-20250606.2.iso) released on 18 Dec 2025, but the latest iso with xostor comparability is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso released in 16 June 2025.

Reason for asking is the last incremental upgrade on 18th Dec partially failed on our pool master and so we need to do a 'clean' upgrade, however there are XOSTOR disks on that machine, and doing a network upgrade after a partial failure and regardless with xostor - is not advised / achievable.

Thank you!

@florent Can't wait!

Ok, spent weekend having backups running continuously across a number of boxes.

Good news - the fix seems to have solved things, providing one only ever uses concurrency “1” and there are no conflicting or overlapping backups.

Restores are working fine for me too.

In short @nraynaud - it’s a substantial improvement and for me makes this now usable. A huge thank you.

@dustinb Concur 100%; my current focus is on confirming the error doesn't reoccur and understanding the change in what I'm seeing compared with previous backups, before this goes into production I 100% agree it should be tested for restores. I shall do so myself once I've got confidence that the symptom has been resolved.

I'll try a fresh install over the next couple of days and see if it reoccurs. Looking at the other boxes, I have the same error on one of the other hosts, but it's not across all of them despite identical installs and hardware.

Thanks for your efforts and help, shame there wasn't an easy answer but let's see if it reoccurs after a completely fresh install.

@stormi I confess we're now encountering the same error message on nearly all our backups, including CR to a local host. Started from fresh install and cleaned SRs; to avoid memory being the culprit we have upped the memory for Dom0 to 16gb (128gb machine) and XO is running with 16gb of mem of which 12gb is allocated to node.

We've got the same problem occurring across all our hosts. Over 90% of backups error out with the VDI_IO_ERROR, however (weirdly) looking at the target end, I'd say that 75% of the backups 'seem' to complete successfully. Need to restore a couple to find out for sure but confess I've been concentrating on finding out what triggers the error rather than whether it is misleading.

I've gone through the logs in detail and unfortunately nothing jumps out, I'm going to take time to extract the relevant sections from them all to see if you can see something that I can't, but apart from lots of memory messages from squeezed there aren't any obvious errors.

Bizarrely SMlog is pretty clean - it's almost like it receives a termination signal from somewhere rather than erroring out of its own accord - for example tapdisk shuts down with "Closing after 0 errors" and no further explanation. I have found some talk that tapdisk can trigger a target reset after excessive i/o activity but I've not managed to prove that yet.

I'll keep digging into things ; in short it's not something only experienced by @fachex but I haven't yet recreated it in XOA - it's on my todo list.

(If you want me to download all the logs and send them across directly, or to do anything under the covers of XOA, please do let me know. I'm dipping into this when I get time so its not a continuous effort I'm afraid.)

That worked; thank you!!!

@stormi You star, will let you know. Thank you!

@stormi https://xcp-ng.org/forum/post/101015 V kind, thanks

As per chat in https://xcp-ng.org/forum/post/100993 ; have had a failed update in one our of pools that necessitates using a local ISO to update XCP-ng , however as the pool is using XOSTOR one cannot use the standard release (in this instance xcp-ng-8.3.0-20250606.2.iso published on 18 Dec '25) and instead the update requires the XOSTOR packages to be in that latest iso.

(Context: Due to the issue occurring during an update, we need an iso with the latest patch levels as the update has partially completed leaving the system with updated new packages. We can't use the older xostor friendly iso as there the linstor version is at 1.29.0 whilst the system now has linstor 1.29.2, and we can't use any other route (e.g network or cli) to update as these are not supported for xostor implementations.)

The last xcp-ng iso with XOSTOR packages is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso which was released on 16 Jun '25. Is there a possibility of building an XOSTOR iso image each time we build a new xcp-ng iso so that the two are in sync (or even have a common iso)?

And in the short term is there a way that I or someone cleverer could generate the iso?

Appreciate any help; thank you!

@stormi As a footnote to earlier message, we tried using the previous iso image (xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso) to see if things would magically work out but it results in an unrecoverable error - "Cannot upgrade host with LINSTOR 1.29.2-1.e17_9, upgrade repository has versions 1.29.0-1.e17_9. Please make sure your pool is uptodate [sic - typo in error message] and use the latest dedicated ISO."

So yes, afraid I'm after an iso with 1.29.2...

@stormi All our hosts were fully patched. We then went through the upgrade of Dec 19th. Two (single server) pools updated fine, the master for the primary pool then failed after patching but on reboot - this machine happens to have XOSTOR so doing the upgrade manually and recovering via the ISO is not an option as the ISO is not XOSTOR compatible and the other options available to us (network update etc) are not permitted by the installer due to XOSTOR. We're not using the Broadcom drivers.

Installer recognises the old installation, that install was patched and it was the reboot after that caused the problem (no idea why or how) so reluctant to 'upgrade' to the previous install given the patch had completed except for the final reboot.

It might be that we can 'refresh' the install using the older version but was nervous of doing so given we'd end up with (potentially) a mash-up of versions of drivers versus data.

Is there any intent to publish the latest xcp-ng 'release' with an XOSTOR iso ? There's an iso for the non-XOSTOR version (xcp-ng-8.3.0-20250606.2.iso) released on 18 Dec 2025, but the latest iso with xostor comparability is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso released in 16 June 2025.

Reason for asking is the last incremental upgrade on 18th Dec partially failed on our pool master and so we need to do a 'clean' upgrade, however there are XOSTOR disks on that machine, and doing a network upgrade after a partial failure and regardless with xostor - is not advised / achievable.

Thank you!

Correct - through the CLI one could see that the old VM that had been migrated from Host1 to Host2 was still showing under Host1, whilst the migrated copy was showing as running on Host2. Once the Host1 remnant of the migration was removed that cleared things and XO correctly reported the VM as running on Host2 with its disks attached.

TLDR - There were no other conflicts beyond what appeared through XO to be the only version sitting halted on Host1, but through the CLI one could see the halted copy on Host1 and the running copy on Host2. Somehow the running version did not show in XO until the remnant was removed.

Thanks for your help @olivierlambert

Solved.

Thank you - superb deduction skills

Ah, you genius. Yes, disconnecting Host1/Pool1 does indeed have the VM magically appear. So there must be a duplicate on Host1 that 'shields' XO from seeing it - only the dup doesn't show in XO. Will check via CLI - I have enough now to find the issue.

Thank you!!