RPU on XOSTOR pool: live migration fails while evacuating the host running the linstor controller, VM left unmanageable

redouann

Hello,

I ran a Rolling Pool Update from Xen Orchestra on a pool with XOSTOR. A live migration failed during the evacuation of the second host. The VM was left in a state that neither XO nor xe could handle (no restart, no shutdown, no reset-powerstate), and I could only recover it manually from the hosts. After that cleanup, a new RPU run completed without errors.

Setup:

XCP-ng 8.3, 3 hosts, HA enabled (the RPU disables it)
XOSTOR SR: replication 2, thin provisioning
DRBD traffic on a dedicated 2.5G network (PrefNic storage)
linstor controller/satellite connections (StltCon) and live migration on the 1G management network (no dedicated migration network)
Same packages on all 3 hosts: sm 3.2.12-25.2, xcp-ng-linstor 1.2-7, python-linstor and linstor-client 1.27.1, LINSTOR controller 1.33.1
Xen Orchestra from sources, commit 83c3e344ba (master of 2026-11-10), xo-server 5.213.0

Timeline of the first run (from the XO RPU trace and the host logs):

22:53:01 to 22:53:17: "Updating LINSTOR packages" (this step stops linstor-controller and restarts linstor-satellite on all hosts)
22:53:49 to 22:54:34: evacuation of gamma (pool master), OK
22:55:53: reboot of gamma. The host was up at 22:56:40, and its satellite was ONLINE again in the controller at 22:57:08
22:58:08: "Waiting for host to be up" finished. At 22:58:09 the RPU started evacuating beta. The linstor controller was running on beta at that moment
22:58:18 to about 23:01: while the VMs were migrating away from beta, the controller (on beta) repeatedly lost its connections to the satellites of alpha and gamma. They reconnected several times, and the controller logged PeerClosingConnectionException ("Fetch thin capacity info") at 22:59:54 and 23:00:10
23:00:10: the migration of one VM (disk on XOSTOR, beta to gamma) failed at synchronisation point 4. The VDI attach on gamma failed:

SR_BACKEND_FAILURE_47 "The SR is not available [opterr=Error: ERRO:NoneType]"
(SMlog on gamma: LinstorSR, line 479, in load, during vdi_attach)

Other XOSTOR VDIs were attached on gamma without problems at 22:59:45, 23:00:15 and 23:00:55. So it looks like a transient failure of the LINSTOR connection during that attach, not an SR that was unavailable on gamma.

State after the failure:

XAPI showed the VM as running on beta
xl list on beta showed the domain in state ---ss-
xenopsd on beta did not know the VM any more. Every action from XO or xe failed with:
INTERNAL_ERROR(Object with type VM and id <uuid>/config does not exist in xenopsd)
xe vm-reset-powerstate --force was refused because the host is live
a paused domain of the VM and an "extra" entry in xenopsd were left on gamma

Following runs:

2nd run: failed immediately with the same "config does not exist in xenopsd" error, because the VM was still in that state
I then recovered the VM on beta (xl destroy, xe-toolstack-restart on beta, vm-start)
3rd run: failed with VM.migrate: Failed during Synchronisation point 4. Already_exists extra/<uuid>, because of the entry left on gamma by the first failure. It left another paused domain on gamma and a ---ss- domain on beta
Manual cleanup on both hosts (below), VM started on gamma
4th run: completed successfully (beta and alpha updated and rebooted, VMs migrated back)

Manual cleanup (on both hosts):

xl destroy of the leftover domains
systemctl stop xenopsd-xc
remove the VM entries (VM/<vm-uuid> and extra/<vm-uuid>) under /var/run/nonpersistent/xenopsd/classic/
xe-toolstack-restart (on beta it took two restarts before XAPI showed the VM as halted)
xe vm-start

What I think happened (I may be wrong):
The linstor controller was running on the host being evacuated. Its connections to the satellites go over the same 1G management network as the live migrations. During the evacuation it lost the satellites several times, and one vdi_attach on the destination hit that window. I can't prove it was network saturation; the logs only show the disconnections and reconnections.

Questions / suggestions:

Is it expected that LinstorSR fails the vdi_attach right away on a transient controller error (ERRO:NoneType), without retrying?
Should the RPU avoid evacuating the host that runs the linstor controller while migrations depend on it, or move the controller first? It might also be worth waiting after each reboot until XOSTOR is fully settled (currently the RPU only waits for host.enabled and metrics.live)
A migration that fails at synchronisation point 4 leaves the VM unmanageable on both hosts and blocks the next RPU runs until it is cleaned up manually
Is a dedicated migration network, or moving the controller/satellite connections to the storage network, the recommended setup for XOSTOR pools?

I can provide the XO RPU trace, xensource.log, SMlog and the linstor-controller error reports if needed.

Thanks!

poddingue

I can only answer one of your questions, and only from the docs.

On moving the controller and satellite connections to the storage network, the XOSTOR page says that "is not recommended" and that the management interface is the best choice for the pool's robustness: https://docs.xcp-ng.org/xostor/#how-to-use-a-specific-network-for-satellites .

As far as I can find, it says nothing about a dedicated migration network for XOSTOR pools. Whether LinstorSR should retry an attach on a transient controller error, or the RPU should move the controller first, is beyond me, so I think @Team-Storage is the right group to call.
The xenopsd leftovers after the failed sync point 4 seem worth them seeing too, and I'd keep the RPU trace, SMlog and controller error reports around, since I suspect they'll ask for them.

redouann

Thanks @poddingue, that's helpful. I'll keep the satellites on the management interface then.

@Team-Storage could you please have a look at the LinstorSR attach failure, the RPU evacuating the controller host, and the xenopsd leftovers after the failed sync point 4? I have the RPU traces, SMlog, xensource.log and the linstor-controller error reports archived if needed.

redouann

Hi, as promised, here are the log excerpts from the incident, in case they help the @Team-Storage folks (or anyone hitting the same thing). All times are CET (host time). VM dd25471a-fbd3-2b0c-df4c-b8182546f6a8, SR 1a97f391-…, migration task R:4fa24c2fefb3 (beta → gamma).

1. The failed attach (SMlog on gamma)

The SR lock is acquired at 23:00:04 and the exception comes 6s later:

Nov 10 23:00:04 gamma SM: [11963] lock: acquired /var/lock/sm/1a97f391-c733-8520-62c3-2f3847235a36/sr
Nov 10 23:00:10 gamma SM: [11963] Raising exception [47, The SR is not available [opterr=Error: ERRO:NoneType]]
Nov 10 23:00:10 gamma SM: [11963] ***** generic exception: vdi_attach: EXCEPTION <class 'xs_errors.SROSError'>, The SR is not available [opterr=Error: ERRO:NoneType]
  File "/opt/xensource/sm/SRCommand.py", line 157, in _run_locked
    target = sr.vdi(self.vdi_uuid)
  File "/opt/xensource/sm/LinstorSR", line 549, in wrap
    return load(self, *args, **kwargs)
  File "/opt/xensource/sm/LinstorSR", line 479, in load
    raise xs_errors.XenError('SRUnavailable', opterr=str(e))

One second later the same SR answers normally, which is why I think the failure was transient:

Nov 10 23:00:11 gamma SM: [12120] Failed to join node(s): {'gamma', 'alpha'}
Nov 10 23:00:11 gamma SM: [12120] Synchronize metadata...
Nov 10 23:00:11 gamma SM: [12120] VDI c90eeb92-... loaded! (path=/dev/drbd/by-res/xcp-persistent-ha-statefile/0, hidden=0)

2. LINSTOR side (controller on beta)

Three error reports between 22:59 and 23:00, all identical:

ERROR REPORT 6AF3924C-00000-000001   (also -000002 and -000003)
Version: 1.33.1   Node: beta
Class name: com.linbit.linstor.netcom.PeerClosingConnectionException
Generated at: Method 'connectionClosing', Source file 'TcpConnectorPeer.java', Line #627
    *__checkpoint ⇢ Fetch thin capacity info

3. How it surfaces in xapi

gamma xapi: [error|VDI.attach2] sm_exec failed with exception Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_47);[S();S(The SR is not available [opterr=Error: ERRO:NoneType]);S()]]])
gamma xapi: [error|Async.VM.pool_migrate R:4fa24c2fefb3|Storage_smapiv1_wrapper] dp:vbd/1/xvda sr:1a97f391-... vdi:60342e65-... op:attach(RW) error:Storage_error (...SR_BACKEND_FAILURE_47...)

4. The cleanup on the source also fails (beta)

23:00:10 beta xenopsd-xc: [error|71|Async.VM.pool_migrate R:4fa24c2fefb3|xenops_server] VM.migrate: Failed during Synchronisation point 4. msg: Xenops_interface.Xenopsd_error([S(Storage_backend_error);[S(SR_BACKEND_FAILURE_47);...]])
23:00:10 beta xenopsd-xc: [error|71|...|task_server] Task 101 failed; Xenopsd_error([S(Does_not_exist);[S(VM);S(dd25471a-.../config)]])
23:00:10 beta xenopsd-xc: [error|71|...|xenops_server] Triggering cleanup actions failed: Xenopsd_error([S(Does_not_exist);[S(VM);S(dd25471a-.../config)]])

So after the migration error, the cleanup fails too because the VM config is no longer in xenopsd on the source. That matches what I saw afterwards: XAPI says running on beta, but xenopsd doesn't know the VM.

5. What is left on the destination (gamma)

The temporary extra/ entry is created at 22:58:09 and renamed to the final uuid at 23:00:03, i.e. before the attach fails at 23:00:10. Nothing removes it afterwards:

22:58:09 gamma xenopsd-xc: [debug|Async.VM.pool_migrate R:4fa24c2fefb3|xenops_utils] TypedTable: Writing extra/dd25471a-fbd3-2b0c-df4c-000000000001
23:00:03 gamma xenopsd-xc: [debug|13|Async.VM.pool_migrate R:4fa24c2fefb3|xenops_utils] TypedTable: Renaming extra/dd25471a-fbd3-2b0c-df4c-000000000001 -> extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8

6. What the following RPU runs report in XO

2nd run (00:03, VM still in that state):

END failure Evacuate  INTERNAL_ERROR(Object with type VM and id dd25471a-fbd3-2b0c-df4c-b8182546f6a8/config does not exist in xenopsd)
END failure Rolling pool update  INTERNAL_ERROR(Object with type VM and id dd25471a-.../config does not exist in xenopsd)

3rd run (00:11–00:14, after I had recovered the VM on beta only):

END failure Evacuate  INTERNAL_ERROR(xenopsd internal error: VM.migrate: Failed during Synchronisation point 4. msg: Xenops_interface.Xenopsd_error([S(Already_exists);[S(extra);S(extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8)]]))

And on gamma for that run:

00:13:42 gamma xenopsd-xc: [debug|94|Async.VM.pool_migrate R:005a41ca1c30|xenops_utils] Key extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8 already exists
00:13:42 gamma xenopsd-xc: [error|94|...|backtrace] Raised Xenopsd_error([S(Already_exists);[S(extra);S(extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8)]])
00:13:42 gamma xenopsd-xc: [debug|94|...|xenops_server] Caught Xenopsd_error([S(Already_exists);...]): cleaning up VM state

Also at 00:14:03, opening the VM console from XO fails on beta with the same Does_not_exist … /config error.

What the logs show

A single transient LINSTOR controller error (ERRO:NoneType) appears to trigger the failure vdi_attach with no retry.
That failure at sync point 4 leaves state on both hosts: on the source the cleanup fails with Does_not_exist …/config, and on the destination the extra/<uuid> entry stays.
Subsequent RPU runs then trips on that leftover (first Does_not_exist, then Already_exists), so it can't recover by itself and requires manual recovery on both hosts.

I still have the full xensource.log, SMlog and ErrorReport files for that window if someone needs more.

Thanks!