xcp-ng server crashed/rebooted due to issues with drbd/linstor?
-
Oopsie, I was wrong.
I owe you a correction on the version, and that one is on me. I said 8.3 ships 9.2.18, but I was reading the spec file on the branch rather than what you actually get from the stable repo.
Your yum check-update output answers it better than I did: stable leaves you on 9.2.16, and 9.2.18 only turns up once xcp-ng-linstor-testing is enabled.The screenshot in that same post is the part the still puzzles me. XO says "All good, patches up to date" while the module underneath is two point releases back, which I assume is what everyone else running XOSTOR sees too. If the connection-teardown fixes you found really are in 9.2.19, that is three releases of distance on the exact code path your traces land in, not the single bump I described.
-
Hello,
Indeed this is a known issue with XOSTOR, when causing a sync of too many resources at once it's possible for a DRBD resource to leak memory during the resync. A patch on the 9.2.18 fixes this by throttling the number of resources that can be synced at once to mitigate the issue.
The 9.2.18 update should be in the main xcp-ng-linstor repo very soon.
What you can do too is give more RAM to the dom0 of your hosts. How many RAM have been allocated to dom0 at the moment? We advise 16GB or 32GB depending on the number of volumes.
-
@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.
-
@Jonathon Have you rebooted after the update? I should have specified it but the kmod-drbd update is only effective after a reboot.
If the 9.2.16 drbd module was still loaded (i.e. host not rebooted) 16GB would still have been filled quickly with the memory leak unfortunately. 16GB and 9.2.18 should be safe now though.
If unsure you can check the output of
modinfo drbd. -
@Mathieu-L Yes, the hosts had been restarted after the updates were installed which included 9.2.16
-
@Jonathon I think @mathieu-l asked it you rebooted after installing drbd 9.2.18 from test repository. Your message sounds like you rebooted after installing 9.2.16 but not after applying the 9.2.18 update from test repository.
-
P poddingue marked this topic as a question
-
@MajorP93 My mistake. No I have not rebooted the hosts yet. https://xcp-ng.org/forum/post/107164 I am aware that the reboot is needed for it to be picked up, I went ahead and installed from test so that if host crashes again it is there.
-
@Jonathon Indeed I missed that part in your message, my bad.
If the hosts still crash even after the 9.2.18 module has been loaded (update + reboot), please keep us informed.Thanks!
-
New crash, this time xen02. The time is different because I updated the velero backup schedules to not all be in the middle of the night. Also still happened with the openvas schedule removed, which makes sense.
Xen02 now has 9.2.18 loaded. Will restart other hosts now.
I also uploaded logs if those are desired.
-
All hosts are restarted with
9.2.18loaded now. Will see what happens next.[15:56 ovbh-pprod-xen05 ~]# drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36 DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION_CODE=0x090212 DRBD_KERNEL_VERSION=9.2.18 DRBDADM_VERSION_CODE=0x092100 DRBDADM_VERSION=9.33.0
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login