XOA loses connection to hosts during VM migration / creation on XOSTOR SR

DAYELA

Hello,
I’m experiencing an issue on an XCP-ng cluster using XOSTOR.
Environment:

3-node XCP-ng cluster
XOSTOR distributed storage (2x2Tob nvme on each host)
XOA for management
Management network 1Gb/s
Storage Network 10Gb/s

MTU 1500 everywhere (no jumbo frames)

So during VM migrations, creation, destroy XOA loses connection to my host pool, VMs keep running normally Hosts remain reachable (SSH / HTTPS / ping OK), Connection comes back after some time 30s to 1min.

Observations:

No significant CPU or RAM saturation
No obvious disk latency issues (iostat looks normal)
No errors reported on NICs
xapi process remains active (no crash or freeze)
The problem is intermittent and seems random.
i've monitored nic with iftop and i see no bandwith bottleneck et and can see that XOSTOR is using 10gb network only.

Has anyone experienced similar behavior with XOSTOR? And how to Fix it ?
Thanks in advance for your help.

poddingue

This is a tricky one, and I'm a bit out of my depth on the XOSTOR internals, but the pattern you describe (XO losing the pool for 30 to 60 seconds while SSH and ping to the hosts stay perfectly fine) reads more like XAPI stalling on a storage call during the LINSTOR coordination than a real network drop.
One detail caught my eye: you mention XOSTOR is on the 10Gb storage network only. The XOSTOR docs actually recommend the satellites stay on the XAPI management interface and call a dedicated network for them "not recommended" for pool robustness (https://docs.xcp-ng.org/xostor/), which feels counterintuitive with a fast 10Gb link sitting right there, so I may well be misreading your setup.
If you're able to grab /var/log/SMlog and /var/log/xensource.log on the master host during one of the disconnect windows, that should show whether XAPI is timing out waiting on a storage operation. Nobody's chimed in yet, and this is pretty XOSTOR-specific, so it might be worth a mention to @Team-Storage; they'll know far more than I do about whether the satellite-network config is what's triggering it.

olivierlambert

Is this cluster using HA or not?

DAYELA

@poddingue I had some issues with my NVMe drives saturating the bandwidth of my 1Gb link, which is why I switched the DRBD sync to 10Gb links; the rest (management, etc.) remains on the 1Gb link.

I’ve since switched back to Proxmox since I couldn’t stay without an infrastructure indefinitely, but I’ll try deploying it again before the end of the week to retrieve some logs.

DAYELA

@olivierlambert yes it is a cluster using HA

olivierlambert

Please disable HA and report if you still have the issue.