Slow VM migration on Linstor SR

[KB973]

@Team-Storage

XOSTOR (LINSTOR SR): ~200s extra delay on live migration and cold start when a host accesses a VDI for the first time

Summary

On a 4-host XCP-ng pool with a XOSTOR (LINSTOR/DRBD) shared SR, both live migration and a plain VM start take roughly 3-3.5 minutes longer than expected for an idle 4GB RAM VM, specifically when the target host has not previously held a diskful copy of that VDI's resource. Network, DRBD resync, quorum settings, and package version mismatches have all been ruled out through direct testing. SMlog/xensource.log show two separate, near-identical ~100-105 second gaps with zero log output during the gap itself, which strongly suggests a retry/timeout loop rather than actual data movement.

Environment

• XCP-ng pool, 4 hosts
• Packages (identical on all 4 hosts):
  • linstor-common-1.33.1-1.el7_9, linstor-controller-1.33.1-1.el7_9, linstor-satellite-1.33.1-1.el7_9
  • linstor-client-1.27.1-1.xcpng8.3, python-linstor-1.27.1-1.xcpng8.3
  • xcp-ng-linstor-1.2-6.xcpng8.3, xcp-ng-release-linstor-1.5-1.xcpng8.3
  • No newer version available via yum check-update on the XCP-ng repos at time of testing.
• XOSTOR SR: redundancy=3, provisioning=thin, group-name=linstor_group/thin_device, created via xe sr-create type=linstor (CLI, no XOA wizard).
• Resource-group has DrbdOptions/Resource/on-no-quorum suspend-io and DrbdOptions/Resource/on-no-data-accessible suspend-io set explicitly.
• Dedicated DRBD/storage network: separate bond (active-backup, 2x25Gb/s), tagged VLAN with MTU 9000, PrefNic explicitly set on all nodes to use this network (confirmed via linstor node list-properties, not the management interface).
• Test VM: 4GB RAM, idle at time of test, 2 disks on the XOSTOR SR (20GB and 100GB declared size).

What we ruled out

1. Network — ping between storage-network IPs succeeds, PrefNic confirmed active on all nodes, MTU 9000 verified end to end.
2. DRBD resync — linstor resource list -r <resource> watched live during migration shows the destination host getting the resource as Diskless immediately (no Inconsistent state, no growing InSync %). It is not doing a full data resync.
3. Quorum settings — already applied (suspend-io on both properties), no change in behavior.
4. Package versions — identical across all 4 hosts, no update available.
5. Resource-group properties — linstor resource-group list-properties shows nothing beyond the quorum settings above.
6. DrbdOptions/Resource/auto-quorum property — not whitelisted in this LINSTOR version (1.33.1), had to use the primitive property names instead.

Log evidence (live migration between two hosts)

xensource.log (destination host):

09:10:36  VM.receive_memory: Synchronisation point 2
09:12:18  VM.receive_memory: Synchronisation point 3
09:12:18  VM.receive_memory: Renaming domain
09:12:18  VM.receive_memory: restoring remaining devices and unpausing
09:14:02  VM.receive_memory: Synchronisation point 4
09:14:02  Device.Dm.start domid=2 action=Restore ...
09:14:02  Got QMP event, domain-2: RESUME

→ Gap 1: 09:10:36 → 09:12:18 (~102s), no log output at all during this window.
→ Gap 2: 09:12:18 → 09:14:02 (~104s), same, no log output on this host during the gap.

SMlog (source host), same window:

09:10:36/37  vdi_deactivate {...} (both VDIs, two threads)
[~102s of silence]
09:12:18  tap.deactivate: Shut down Tapdisk(...)
09:12:18  LinstorVDI.detach for <vdi_uuid>
09:12:18  Deflate /dev/drbd/by-res/xcp-volume-.../0 (new size=3410448896, previous=21523156992)
09:12:18  Deflate /dev/drbd/by-res/xcp-volume-.../0 (new size=7558312448, previous=107590782976)

→ vdi_deactivate is called at 09:10:36/37 but tap.deactivate (actually stopping the tapdisk process) doesn't happen until 09:12:18 — the same ~102s gap as on the destination side, in a completely different log, on a different host.
→ Once tap.deactivate starts, the VHD "Deflate" (thin-pool metadata shrink to actual used size) happens immediately, so that step is not the bottleneck.

This behavior is 100% reproducible: it happens both on live migration and on a plain VM start on a host that previously had no diskful/diskless record for that VDI's resource. Hosts that already have an established record for the resource do not show this delay.

What we're asking

Is there a documented reason for this behavior and is there a solution.

We're happy to provide additional diagnostics (full SMlog excerpts, linstor command output, etc.) as needed. Given this also affects plain VM start (not just migration), it directly impacts HA failover time, which is why we're raising it as a priority.

Thanks in advance for any pointers.