@Jonathon Indeed I missed that part in your message, my bad.
If the hosts still crash even after the 9.2.18 module has been loaded (update + reboot), please keep us informed.

Thanks!

@Jonathon Have you rebooted after the update? I should have specified it but the kmod-drbd update is only effective after a reboot.

If the 9.2.16 drbd module was still loaded (i.e. host not rebooted) 16GB would still have been filled quickly with the memory leak unfortunately. 16GB and 9.2.18 should be safe now though.

If unsure you can check the output of modinfo drbd.

Hello,

Indeed this is a known issue with XOSTOR, when causing a sync of too many resources at once it's possible for a DRBD resource to leak memory during the resync. A patch on the 9.2.18 fixes this by throttling the number of resources that can be synced at once to mitigate the issue.

The 9.2.18 update should be in the main xcp-ng-linstor repo very soon.

What you can do too is give more RAM to the dom0 of your hosts. How many RAM have been allocated to dom0 at the moment? We advise 16GB or 32GB depending on the number of volumes.

Hi @jonathon,

If that happens again, could you show me the results of the command linstor n l (listing the nodes and their status) along with the version of each node?

When restarting the controller, did you do it with a systemctl restart linstor-controller or with a systemctl stop linstor-controller, letting drbd-reactor elect a new node?

You mentioned updating all of your nodes but you also mentioned rebooting only one node, is this correct? Depending on the version you were before the update it's possible that you got a version mismatch between your nodes (the satellites of each node + the controller needing to be rebooted so that they all get to the same version). The node appearing offline is due to a bug when updating from < linstor-server v1.32.0 to >= linstor-server v1.32.0 which makes the node with a different version appear as offline (the auth error is a symptom of that).