RPU on XOSTOR pool: live migration fails while evacuating the host running the linstor controller, VM left unmanageable

[redouann]

Hello,

I ran a Rolling Pool Update from Xen Orchestra on a pool with XOSTOR. A live migration failed during the evacuation of the second host. The VM was left in a state that neither XO nor xe could handle (no restart, no shutdown, no reset-powerstate), and I could only recover it manually from the hosts. After that cleanup, a new RPU run completed without errors.

Setup:

• XCP-ng 8.3, 3 hosts, HA enabled (the RPU disables it)
• XOSTOR SR: replication 2, thin provisioning
• DRBD traffic on a dedicated 2.5G network (PrefNic storage)
• linstor controller/satellite connections (StltCon) and live migration on the 1G management network (no dedicated migration network)
• Same packages on all 3 hosts: sm 3.2.12-25.2, xcp-ng-linstor 1.2-7, python-linstor and linstor-client 1.27.1, LINSTOR controller 1.33.1
• Xen Orchestra from sources, commit 83c3e344ba (master of 2026-11-10), xo-server 5.213.0

Timeline of the first run (from the XO RPU trace and the host logs):

• 22:53:01 to 22:53:17: "Updating LINSTOR packages" (this step stops linstor-controller and restarts linstor-satellite on all hosts)
• 22:53:49 to 22:54:34: evacuation of gamma (pool master), OK
• 22:55:53: reboot of gamma. The host was up at 22:56:40, and its satellite was ONLINE again in the controller at 22:57:08
• 22:58:08: "Waiting for host to be up" finished. At 22:58:09 the RPU started evacuating beta. The linstor controller was running on beta at that moment
• 22:58:18 to about 23:01: while the VMs were migrating away from beta, the controller (on beta) repeatedly lost its connections to the satellites of alpha and gamma. They reconnected several times, and the controller logged PeerClosingConnectionException ("Fetch thin capacity info") at 22:59:54 and 23:00:10
• 23:00:10: the migration of one VM (disk on XOSTOR, beta to gamma) failed at synchronisation point 4. The VDI attach on gamma failed:

SR_BACKEND_FAILURE_47 "The SR is not available [opterr=Error: ERRO:NoneType]"
(SMlog on gamma: LinstorSR, line 479, in load, during vdi_attach)

Other XOSTOR VDIs were attached on gamma without problems at 22:59:45, 23:00:15 and 23:00:55. So it looks like a transient failure of the LINSTOR connection during that attach, not an SR that was unavailable on gamma.

State after the failure:

• XAPI showed the VM as running on beta
• xl list on beta showed the domain in state ---ss-
• xenopsd on beta did not know the VM any more. Every action from XO or xe failed with:
  INTERNAL_ERROR(Object with type VM and id <uuid>/config does not exist in xenopsd)
• xe vm-reset-powerstate --force was refused because the host is live
• a paused domain of the VM and an "extra" entry in xenopsd were left on gamma

Following runs:

• 2nd run: failed immediately with the same "config does not exist in xenopsd" error, because the VM was still in that state
• I then recovered the VM on beta (xl destroy, xe-toolstack-restart on beta, vm-start)
• 3rd run: failed with VM.migrate: Failed during Synchronisation point 4. Already_exists extra/<uuid>, because of the entry left on gamma by the first failure. It left another paused domain on gamma and a ---ss- domain on beta
• Manual cleanup on both hosts (below), VM started on gamma
• 4th run: completed successfully (beta and alpha updated and rebooted, VMs migrated back)

Manual cleanup (on both hosts):

• xl destroy of the leftover domains
• systemctl stop xenopsd-xc
• remove the VM entries (VM/<vm-uuid> and extra/<vm-uuid>) under /var/run/nonpersistent/xenopsd/classic/
• xe-toolstack-restart (on beta it took two restarts before XAPI showed the VM as halted)
• xe vm-start

What I think happened (I may be wrong):
The linstor controller was running on the host being evacuated. Its connections to the satellites go over the same 1G management network as the live migrations. During the evacuation it lost the satellites several times, and one vdi_attach on the destination hit that window. I can't prove it was network saturation; the logs only show the disconnections and reconnections.

Questions / suggestions:

• Is it expected that LinstorSR fails the vdi_attach right away on a transient controller error (ERRO:NoneType), without retrying?
• Should the RPU avoid evacuating the host that runs the linstor controller while migrations depend on it, or move the controller first? It might also be worth waiting after each reboot until XOSTOR is fully settled (currently the RPU only waits for host.enabled and metrics.live)
• A migration that fails at synchronisation point 4 leaves the VM unmanageable on both hosts and blocks the next RPU runs until it is cleaned up manually
• Is a dedicated migration network, or moving the controller/satellite connections to the storage network, the recommended setup for XOSTOR pools?

I can provide the XO RPU trace, xensource.log, SMlog and the linstor-controller error reports if needed.

Thanks!