That worked; thank you!!!

@stormi You star, will let you know. Thank you!

@stormi https://xcp-ng.org/forum/post/101015 V kind, thanks

As per chat in https://xcp-ng.org/forum/post/100993 ; have had a failed update in one our of pools that necessitates using a local ISO to update XCP-ng , however as the pool is using XOSTOR one cannot use the standard release (in this instance xcp-ng-8.3.0-20250606.2.iso published on 18 Dec '25) and instead the update requires the XOSTOR packages to be in that latest iso.

(Context: Due to the issue occurring during an update, we need an iso with the latest patch levels as the update has partially completed leaving the system with updated new packages. We can't use the older xostor friendly iso as there the linstor version is at 1.29.0 whilst the system now has linstor 1.29.2, and we can't use any other route (e.g network or cli) to update as these are not supported for xostor implementations.)

The last xcp-ng iso with XOSTOR packages is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso which was released on 16 Jun '25. Is there a possibility of building an XOSTOR iso image each time we build a new xcp-ng iso so that the two are in sync (or even have a common iso)?

And in the short term is there a way that I or someone cleverer could generate the iso?

Appreciate any help; thank you!

@stormi As a footnote to earlier message, we tried using the previous iso image (xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso) to see if things would magically work out but it results in an unrecoverable error - "Cannot upgrade host with LINSTOR 1.29.2-1.e17_9, upgrade repository has versions 1.29.0-1.e17_9. Please make sure your pool is uptodate [sic - typo in error message] and use the latest dedicated ISO."

So yes, afraid I'm after an iso with 1.29.2...

@stormi All our hosts were fully patched. We then went through the upgrade of Dec 19th. Two (single server) pools updated fine, the master for the primary pool then failed after patching but on reboot - this machine happens to have XOSTOR so doing the upgrade manually and recovering via the ISO is not an option as the ISO is not XOSTOR compatible and the other options available to us (network update etc) are not permitted by the installer due to XOSTOR. We're not using the Broadcom drivers.

Installer recognises the old installation, that install was patched and it was the reboot after that caused the problem (no idea why or how) so reluctant to 'upgrade' to the previous install given the patch had completed except for the final reboot.

It might be that we can 'refresh' the install using the older version but was nervous of doing so given we'd end up with (potentially) a mash-up of versions of drivers versus data.

Is there any intent to publish the latest xcp-ng 'release' with an XOSTOR iso ? There's an iso for the non-XOSTOR version (xcp-ng-8.3.0-20250606.2.iso) released on 18 Dec 2025, but the latest iso with xostor comparability is xcp-ng-8.3.0-20250616-linstor-upgradeonly.iso released in 16 June 2025.

Reason for asking is the last incremental upgrade on 18th Dec partially failed on our pool master and so we need to do a 'clean' upgrade, however there are XOSTOR disks on that machine, and doing a network upgrade after a partial failure and regardless with xostor - is not advised / achievable.

Thank you!

Correct - through the CLI one could see that the old VM that had been migrated from Host1 to Host2 was still showing under Host1, whilst the migrated copy was showing as running on Host2. Once the Host1 remnant of the migration was removed that cleared things and XO correctly reported the VM as running on Host2 with its disks attached.

TLDR - There were no other conflicts beyond what appeared through XO to be the only version sitting halted on Host1, but through the CLI one could see the halted copy on Host1 and the running copy on Host2. Somehow the running version did not show in XO until the remnant was removed.

Thanks for your help @olivierlambert

Solved.

Thank you - superb deduction skills

Ah, you genius. Yes, disconnecting Host1/Pool1 does indeed have the VM magically appear. So there must be a duplicate on Host1 that 'shields' XO from seeing it - only the dup doesn't show in XO. Will check via CLI - I have enough now to find the issue.

Thank you!!

Correct

It's listed in XO as being halted on Host1 without any disks but (correctly) doesn't show on Host1 under xe list, meanwhile it's listed (correctly) in Host2 xe list as running on there.

Have tried from XO on different boxes (Host1, Host2 and also on a different host) but all report exactly as above - halted on Host1 with no disks.

Have restarted toolstack on Host1 and Host2, have rebooted Host1 and Host2, have rebooted VM from ssh (daren't do it from within XO in case it doesn't come up, similarly dare not stop it from ssh and then try to start it from XO), have tried migrating within XO but does not work, have tried booting from XO (despite it already running) but it does not work.

In summary - in reality it is running on Host2 just fine; xe list is reporting correctly but XO is not.

@olivierlambert Correct

@olivierlambert Thanks for the reply.

Console (xl list) on Host1 does not show VM at all (despite XO thinking it's there without any disks but halted), meanwhile the same on Host2 shows it as booted (although XO does not show it at all on that host).

Toolstack has been restarted several times at both ends to no avail.

I'm a little confused!

Strange one for you:

A running VM was (live) migrated from Host 1 (Pool 1) to Host 2 (Pool 2 - each pool only has a single host). The migration failed (too long ago now for me to have the error to hand - but for us it is not an unusual circumstance for live migrations to fail when the disk is > 250gb) and the VM continued to function BUT it shows in XO as not running but present on Host 1 without a disk attached. However in reality it is running on Host 2 - when shutting down Host 1 the VM continues to operate just fine, when shutting down Host 2 it terminates. .

Starting the allegedly stopped VM in XO fails (starting it on Host1 although as above we know it's running on Host2) it shows as it started for about 10 seconds then goes back to stopped - but no error is given.

Migrating it results in the same outcome - it appears to migrate ok but nothing changes and there are no error messages - but remember that XO has it on the wrong host and without an attache disk. Rebooting the VM from its own console direct via SSH comes up just fine, but the issue endures in Xen Orchestra, and rebooting both hosts also fails to resolve the issue.

Is there a way (without damaging the running VM!) to have it re-registered in XO? As an aside, backups using XO are working for the 'stopped' VM except that as it shows in XO as having no attached disk, they take 2s and are worthless.

[Xen Orchestra, commit 00a17 (although will have been updated a few time since the issue presented itself a month or so ago), both hosts on 8.2.1.]

@florent Can't wait!

As an aside, using the same build I am also getting problems when running CR over a slower WAN, with the following error. Just mentioning it here in case it's related.

"message": "VDI_IO_ERROR(Device I/O errors)",
            "name": "XapiError",
            "stack": "XapiError: VDI_IO_ERROR(Device I/O errors)\n    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/_XapiError.js:26:12)\n    at _default (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/_getTaskResult.js:24:38)\n    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:761:51)\n    at /opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:789:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:774:12)\n    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:931:14)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
          }
        }
      ],
      "end": 1620864312824,
      "result": {
        "message": "all targets have failed, step: writer.transfer()",
        "name": "Error",
        "stack": "Error: all targets have failed, step: writer.transfer()\n    at VmBackup._callWriters (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/_VmBackup.js:118:13)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at async VmBackup._copyDelta (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/_VmBackup.js:190:5)\n    at async VmBackup.run (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/_VmBackup.js:371:9)"

CR to a local machine is working fine, the remote one is failing every iteration.

New S3 function (albeit using Backblaze) seems to be working really well for small backups; I'm now seeing almost 100% success.

However larger backups are consistently failing with

"message": "Error calling AWS.S3.upload: aborted",
            "name": "Error",
            "stack": "Error: Error calling AWS.S3.upload: aborted\n    at rethrow (/opt/xo/xo-builds/xen-orchestra-202105081131/node_modules/@sullux/aws-sdk/index.js:254:24)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at async S3Handler._outputStream (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/fs/dist/s3.js:100:5)\n    at async S3Handler.outputStream (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/fs/dist/abstract.js:250:5)\n    at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/RemoteAdapter.js:509:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/writers/FullBackupWriter.js:69:7"

Running with xo-server 5.79.3 , xo-web 5.81.0

I'm having the same issue across different buckets in B2 and from different servers.

Any ideas?