<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[xcp-ng server crashed/rebooted due to issues with drbd/linstor?]]></title><description><![CDATA[<p dir="auto">Hello, for the last few weeks we have been having random server crashes/reboots. This last crash happened on the <code>June 2026 Updates #1 for XCP-ng 8.3 LTS</code> update. I see a new one just came out.</p>
<p dir="auto">I can share more logs if desired.<br />
<a href="/forum/assets/uploads/files/1782240329984-xen03-jun20-journalctl-crash-summary.txt">xen03-jun20-journalctl-crash-summary.txt</a></p>
<p dir="auto">The timeline is a bit fuzzy, as we started to experience vendor issues and they stated that there was an issue with the power supply. But we have continued to have issues with that server. Now a new server (xen03) has crashed/rebooted. Will add more to this thread as I collect and go through it.</p>
]]></description><link>https://xcp-ng.org/forum/topic/12309/xcp-ng-server-crashed-rebooted-due-to-issues-with-drbd-linstor</link><generator>RSS for Node</generator><lastBuildDate>Mon, 10 Aug 2026 07:42:37 GMT</lastBuildDate><atom:link href="https://xcp-ng.org/forum/topic/12309.rss" rel="self" type="application/rss+xml"/><pubDate>Tue, 23 Jun 2026 18:50:31 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 28 Jul 2026 22:57:35 GMT]]></title><description><![CDATA[<p dir="auto">All hosts are restarted with <code>9.2.18</code> loaded now. Will see what happens next.</p>
<pre><code>[15:56 ovbh-pprod-xen05 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090212
DRBD_KERNEL_VERSION=9.2.18
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0
</code></pre>
]]></description><link>https://xcp-ng.org/forum/post/107317</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107317</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Tue, 28 Jul 2026 22:57:35 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 28 Jul 2026 17:47:45 GMT]]></title><description><![CDATA[<p dir="auto">New crash, this time xen02. The time is different because I updated the velero backup schedules to not all be in the middle of the night. Also still happened with the openvas schedule removed, which makes sense.<br />
Xen02 now has 9.2.18 loaded. Will restart other hosts now.<br />
I also uploaded logs if those are desired.<br />
<img src="/forum/assets/uploads/files/1785260423760-8087b412-eff0-4f2e-9d98-8c87a94e7aad-image-resized.jpeg" alt="8087b412-eff0-4f2e-9d98-8c87a94e7aad-image.jpeg" class=" img-fluid img-markdown" /></p>
]]></description><link>https://xcp-ng.org/forum/post/107299</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107299</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Tue, 28 Jul 2026 17:47:45 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 27 Jul 2026 08:30:13 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/jonathon" aria-label="Profile: Jonathon">@<bdi>Jonathon</bdi></a> Indeed I missed that part in your message, my bad.<br />
If the hosts still crash even after the 9.2.18 module has been loaded (update + reboot), please keep us informed.</p>
<p dir="auto">Thanks!</p>
]]></description><link>https://xcp-ng.org/forum/post/107258</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107258</guid><dc:creator><![CDATA[Mathieu L]]></dc:creator><pubDate>Mon, 27 Jul 2026 08:30:13 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Fri, 24 Jul 2026 16:57:03 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/majorp93" aria-label="Profile: MajorP93">@<bdi>MajorP93</bdi></a> My mistake. No I have not rebooted the hosts yet. <a href="https://xcp-ng.org/forum/post/107164">https://xcp-ng.org/forum/post/107164</a> I am aware that the reboot is needed for it to be picked up, I went ahead and installed from test so that if host crashes again it is there.</p>
]]></description><link>https://xcp-ng.org/forum/post/107228</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107228</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Fri, 24 Jul 2026 16:57:03 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Thu, 23 Jul 2026 19:37:22 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/jonathon" aria-label="Profile: Jonathon">@<bdi>Jonathon</bdi></a> I think <a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/mathieu-l" aria-label="Profile: mathieu-l">@<bdi>mathieu-l</bdi></a> asked it you rebooted after installing drbd 9.2.18 from test repository. Your message sounds like you rebooted after installing 9.2.16 but not after applying the 9.2.18 update from test repository.</p>
]]></description><link>https://xcp-ng.org/forum/post/107207</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107207</guid><dc:creator><![CDATA[MajorP93]]></dc:creator><pubDate>Thu, 23 Jul 2026 19:37:22 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Thu, 23 Jul 2026 17:32:22 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/mathieu-l" aria-label="Profile: Mathieu-L">@<bdi>Mathieu-L</bdi></a> Yes, the hosts had been restarted after the updates were installed which included 9.2.16</p>
]]></description><link>https://xcp-ng.org/forum/post/107205</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107205</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Thu, 23 Jul 2026 17:32:22 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Thu, 23 Jul 2026 16:53:12 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/jonathon" aria-label="Profile: Jonathon">@<bdi>Jonathon</bdi></a> Have you rebooted after the update? I should have specified it but the kmod-drbd update is only effective after a reboot.</p>
<p dir="auto">If the 9.2.16 drbd module was still loaded (i.e. host not rebooted) 16GB would still have been filled quickly with the memory leak unfortunately. 16GB and 9.2.18 should be safe now though.</p>
<p dir="auto">If unsure you can check the output of <code>modinfo drbd</code>.</p>
]]></description><link>https://xcp-ng.org/forum/post/107204</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107204</guid><dc:creator><![CDATA[Mathieu L]]></dc:creator><pubDate>Thu, 23 Jul 2026 16:53:12 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Thu, 23 Jul 2026 16:39:46 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/mathieu-l" aria-label="Profile: Mathieu-L">@<bdi>Mathieu-L</bdi></a>  <a href="https://xcp-ng.org/forum/post/106873">https://xcp-ng.org/forum/post/106873</a> I had increased it to 16G and it still crashed.</p>
]]></description><link>https://xcp-ng.org/forum/post/107203</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107203</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Thu, 23 Jul 2026 16:39:46 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Thu, 23 Jul 2026 15:21:18 GMT]]></title><description><![CDATA[<p dir="auto">Hello,</p>
<p dir="auto">Indeed this is a known issue with XOSTOR, when causing a sync of too many resources at once it's possible for a DRBD resource to leak memory during the resync. A patch on the 9.2.18 fixes this by throttling the number of resources that can be synced at once to mitigate the issue.</p>
<p dir="auto">The 9.2.18 update should be in the main xcp-ng-linstor repo very soon.</p>
<p dir="auto">What you can do too is give more RAM to the dom0 of your hosts. How many RAM have been allocated to dom0 at the moment? We advise 16GB or 32GB depending on the number of volumes.</p>
]]></description><link>https://xcp-ng.org/forum/post/107198</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107198</guid><dc:creator><![CDATA[Mathieu L]]></dc:creator><pubDate>Thu, 23 Jul 2026 15:21:18 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Thu, 23 Jul 2026 09:52:19 GMT]]></title><description><![CDATA[<p dir="auto">Oopsie, I was wrong.<br />
I owe you a correction on the version, and that one is on me. I said 8.3 ships 9.2.18, but I was reading the spec file on the branch rather than what you actually get from the stable repo. <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f937.png?v=f084dd9e819" class="not-responsive emoji emoji-android emoji--shrug" style="height:23px;width:auto;vertical-align:middle" title=":shrug:" alt="🤷" /><br />
Your yum check-update output answers it better than I did: stable leaves you on 9.2.16, and 9.2.18 only turns up once xcp-ng-linstor-testing is enabled.</p>
<p dir="auto">The screenshot in that same post is the part the still puzzles me. XO says "All good, patches up to date" while the module underneath is two point releases back, which I assume is what everyone else running XOSTOR sees too. If the connection-teardown fixes you found really are in 9.2.19, that is three releases of distance on the exact code path your traces land in, not the single bump I described.</p>
]]></description><link>https://xcp-ng.org/forum/post/107175</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107175</guid><dc:creator><![CDATA[poddingue]]></dc:creator><pubDate>Thu, 23 Jul 2026 09:52:19 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Wed, 22 Jul 2026 19:20:32 GMT]]></title><description><![CDATA[<p dir="auto">For now I have upgraded all hosts to use <code>--enablerepo=xcp-ng-linstor-testing</code> to get <code>kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3</code><br />
That way it is there to be picked up if the host crashes again.<br />
<a href="https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec" target="_blank" rel="noopener noreferrer nofollow ugc">https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec</a><br />
<a href="https://koji.xcp-ng.org/buildinfo?buildID=5747" target="_blank" rel="noopener noreferrer nofollow ugc">https://koji.xcp-ng.org/buildinfo?buildID=5747</a></p>
<pre><code>[12:15 ovbh-pprod-xen04 ~]# yum upgrade --enablerepo=xcp-ng-linstor-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
grafana/signature                                                                                                                                                                                                                      |  810 B  00:00:00     
grafana/signature                                                                                                                                                                                                                      | 3.0 kB  00:00:00 !!! 
xcp-ng-linstor-testing/signature                                                                                                                                                                                                       |  473 B  00:00:00     
xcp-ng-linstor-testing/signature                                                                                                                                                                                                       | 3.0 kB  00:00:00 !!! 
zabbix                                                                                                                                                                                                                                 | 3.0 kB  00:00:00     
zabbix-non-supported                                                                                                                                                                                                                   | 2.9 kB  00:00:00     
(1/3): grafana/primary_db                                                                                                                                                                                                              | 879 kB  00:00:00     
(2/3): xcp-ng-linstor-testing/primary_db                                                                                                                                                                                               | 2.4 kB  00:00:00     
(3/3): zabbix/x86_64/primary_db                                                                                                                                                                                                        | 114 kB  00:00:00     
Resolving Dependencies
--&gt; Running transaction check
---&gt; Package kmod-drbd.x86_64 0:9.2.16-1.0.xcpng8.3 will be updated
---&gt; Package kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 will be an update
---&gt; Package zabbix-agent.x86_64 0:7.0.27-release1.el7 will be updated
---&gt; Package zabbix-agent.x86_64 0:7.0.28-release1.el7 will be an update
--&gt; Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================
 Package                                                    Arch                                                 Version                                                           Repository                                                            Size
==============================================================================================================================================================================================================================================================
Updating:
 kmod-drbd                                                  x86_64                                               9.2.18-2.0.xcpng8.3                                               xcp-ng-linstor-testing                                               2.8 M
 zabbix-agent                                               x86_64                                               7.0.28-release1.el7                                               zabbix                                                               660 k

Transaction Summary
==============================================================================================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 3.4 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/2): zabbix-agent-7.0.28-release1.el7.x86_64.rpm                                                                                                                                                                                     | 660 kB  00:00:00     
(2/2): kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64.rpm                                                                                                                                                                                        | 2.8 MB  00:00:01     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         2.8 MB/s | 3.4 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64                                                                                                                                                                                                       1/4 
  Updating   : zabbix-agent-7.0.28-release1.el7.x86_64                                                                                                                                                                                                    2/4 
  Cleanup    : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64                                                                                                                                                                                                       3/4 
  Cleanup    : zabbix-agent-7.0.27-release1.el7.x86_64                                                                                                                                                                                                    4/4 
  Verifying  : zabbix-agent-7.0.28-release1.el7.x86_64                                                                                                                                                                                                    1/4 
  Verifying  : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64                                                                                                                                                                                                       2/4 
  Verifying  : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64                                                                                                                                                                                                       3/4 
  Verifying  : zabbix-agent-7.0.27-release1.el7.x86_64                                                                                                                                                                                                    4/4 

Updated:
  kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3                                                                                       zabbix-agent.x86_64 0:7.0.28-release1.el7                                                                                      

Complete!
</code></pre>
]]></description><link>https://xcp-ng.org/forum/post/107164</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107164</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Wed, 22 Jul 2026 19:20:32 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Wed, 22 Jul 2026 18:09:00 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/poddingue" aria-label="Profile: poddingue">@<bdi>poddingue</bdi></a> I'm fully up to date, but only at version <code>9.2.16</code></p>
<pre><code>[14:57 ovbh-pprod-xen02 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090210
DRBD_KERNEL_VERSION=9.2.16
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0
[14:57 ovbh-pprod-xen02 ~]# drbd-reactor --version
drbd-reactor 1.9.0
[14:57 ovbh-pprod-xen02 ~]# 
</code></pre>
<p dir="auto"><img src="/forum/assets/uploads/files/1784734717818-ae60a96f-99c9-4e31-9f74-362d7d4426d7-image.jpeg" alt="ae60a96f-99c9-4e31-9f74-362d7d4426d7-image.jpeg" class=" img-fluid img-markdown" /></p>
<p dir="auto">There are no updates that I can install</p>
<pre><code>[14:08 ovbh-pprod-xen02 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
zabbix                                                                                                                                                                                                                                 | 3.0 kB  00:00:00     
zabbix-non-supported                                                                                                                                                                                                                   | 2.9 kB  00:00:00     
zabbix/x86_64/primary_db                                                                                                                                                                                                               | 114 kB  00:00:00     

zabbix-agent.x86_64                                                                                                         7.0.28-release1.el7                                                                                                         zabbix
</code></pre>
]]></description><link>https://xcp-ng.org/forum/post/107160</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107160</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Wed, 22 Jul 2026 18:09:00 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Wed, 22 Jul 2026 07:20:02 GMT]]></title><description><![CDATA[<p dir="auto">On the version question, 8.3 ships DRBD 9.2.18 right now (that's what <code>kmod-drbd.spec</code> says on the 8.3 branch), so you'd be asking for a single point bump rather than a big jump.<br />
That probably helps you. Of your two links though, I think the netlink crash fix is a 9.2.17 entry, which would mean it's already in what you're running. <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f937.png?v=f084dd9e819" class="not-responsive emoji emoji-android emoji--shrug" style="height:23px;width:auto;vertical-align:middle" title=":shrug:" alt="🤷" /><br />
The 9.2.19 line that looks closest to your crash to me is the connection-teardown races one rather than the AB-BA deadlock, since a deadlock would hang rather than throw a GPF, but you've read these traces far more closely than I have so correct me if that's backwards.<br />
I can't request a build myself, that's a <a class="plugin-mentions-group plugin-mentions-a" href="/forum/groups/team-storage" aria-label="Profile: Team-Storage">@<bdi>Team-Storage</bdi></a> call rather than mine, though your post #13 timeline next to those changelog lines is a stronger case than most version-bump requests turn up with.</p>
]]></description><link>https://xcp-ng.org/forum/post/107149</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107149</guid><dc:creator><![CDATA[poddingue]]></dc:creator><pubDate>Wed, 22 Jul 2026 07:20:02 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 21 Jul 2026 19:23:21 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/majorp93" aria-label="Profile: MajorP93">@<bdi>MajorP93</bdi></a> No we do not have pro support at the moment</p>
<p dir="auto">hmmm. Really seems like a build with <code>9.2.19</code> could fix all of this.<br />
How can I request for a build <a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/poddingue" aria-label="Profile: poddingue">@<bdi>poddingue</bdi></a> ?</p>
<p dir="auto"><a href="https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L61" target="_blank" rel="noopener noreferrer nofollow ugc">https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L61</a><br />
9.2.17</p>
<blockquote>
<ul>
<li>Fix a kernel crash triggered by a crafted/invalid netlink message</li>
</ul>
</blockquote>
<p dir="auto"><a href="https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L24" target="_blank" rel="noopener noreferrer nofollow ugc">https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L24</a><br />
9.2.19</p>
<blockquote>
<ul>
<li>Fix an AB-BA deadlock between online resize and activity-log  transactions</li>
<li>Fix several races during connection teardown that could crash or<br />
hang (ack_sender requeue, pending ping work, in-progress lb-tcp<br />
connect)</li>
</ul>
</blockquote>
]]></description><link>https://xcp-ng.org/forum/post/107146</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107146</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Tue, 21 Jul 2026 19:23:21 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 21 Jul 2026 16:29:30 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/jonathon" aria-label="Profile: jonathon">@<bdi>jonathon</bdi></a> are you using your LINSTOR/DRBD stack in the form of licensed XOSTOR? Your issue looks a case for pro support. Did you consider opening a case with pro support?</p>
]]></description><link>https://xcp-ng.org/forum/post/107137</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107137</guid><dc:creator><![CDATA[MajorP93]]></dc:creator><pubDate>Tue, 21 Jul 2026 16:29:30 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 21 Jul 2026 07:27:21 GMT]]></title><description><![CDATA[<p dir="auto">That timeline is the clearest version of this yet. xen04 going down the same way as xen05 makes the race look real to me rather than coincidental, unless I am reading the ordering wrong.<br />
The fault lands inside <code>drbd_bm_count_bits</code>, so in the DRBD module itself, which makes me think this is an upstream LINBIT question as much as an XCP-ng one. I do not know enough about how that module gets packaged here to say who owns the fix though. Might be worth another mention to <a class="plugin-mentions-group plugin-mentions-a" href="/forum/groups/team-storage" aria-label="Profile: Team-Storage">@<bdi>Team-Storage</bdi></a> so someone can open your bundles with your post #10 next to them.<br />
On the padding idea I am out of my depth: no idea whether LINSTOR can be told to pre-round a volume at creation, but asking LINBIT directly seems worth a try since it would sidestep <code>drbd_bm_resize</code> entirely. <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f937.png?v=f084dd9e819" class="not-responsive emoji emoji-android emoji--shrug" style="height:23px;width:auto;vertical-align:middle" title=":shrug:" alt="🤷" /></p>
]]></description><link>https://xcp-ng.org/forum/post/107122</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107122</guid><dc:creator><![CDATA[poddingue]]></dc:creator><pubDate>Tue, 21 Jul 2026 07:27:21 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 21 Jul 2026 00:35:12 GMT]]></title><description><![CDATA[<p dir="auto">Logs from xen04 look like they are the same too. Which makes sense.</p>
<pre><code>20:01:53.805: The Linstor DeviceManager on xen04 triggered an asynchronous alignment for a newly created volume (pvc-fac0a18b-2c23-4fe8-b005-b4f492cec92f), padding it from 26220040 KiB to 26222592 KiB.

20:01:54.034: Simultaneously, an orchestrator (like Velero) began tearing the volume down, shifting the DRBD state to conn( Connected -&gt; TearDown ) and pdsk( Diskless -&gt; DUnknown ).

20:01:58.484: The DRBD worker thread evaluated the shrinking bitmap while the alignment was trying to alter it. This caused the general protection fault: 0000 [#1] SMP NOPTI at instruction RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd], crashing the kernel.
</code></pre>
]]></description><link>https://xcp-ng.org/forum/post/107116</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107116</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Tue, 21 Jul 2026 00:35:12 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 20 Jul 2026 20:38:55 GMT]]></title><description><![CDATA[<p dir="auto">I was just moving around the velero backup schedules when it decided to run some again even though it was not time. Crashed xen04 this time. Deleting the possible offending backup schedule until this bug can be identified and fixed.</p>
<p dir="auto">Uploaded new bug bundle to nextcloud too.</p>
]]></description><link>https://xcp-ng.org/forum/post/107114</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107114</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Mon, 20 Jul 2026 20:38:55 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Tue, 21 Jul 2026 00:36:09 GMT]]></title><description><![CDATA[<p dir="auto">My first idea was to have velero use a different storage class. Something like</p>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-ephemeral-retain
parameters:
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/disklessOnRemaining: "true"
  linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool
  linstor.csi.linbit.com/placementCount: "3"
provisioner: linstor.csi.linbit.com
**reclaimPolicy: Retain**
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p dir="auto">Then have a cronjob collected the <code>Released</code> pvcs every hour. The problem is that can not work, as velero uses the storage class of the backed up pvc. So I would have to change every linstor storage class to <code>Retain</code>, and would effectively disable the automatic garbage collection for the cluster. Which <em>might</em> be fine, but would love to avoid it.</p>
<p dir="auto">My second idea is to eliminate the need for the resizing (another work around to hopefully get away from this race condition/bug)<br />
I am also not seeing any way to modify the storageclass to round up or pad the pv. This would be to attempt to eliminate the need for <code>drbd_bm_resize</code>.<br />
Currently looking into Kyverno Mutating Webhook to dynamically pad these volumes on the fly to see if that would help me out at all.<br />
All pvs are already whole numbers of Gi.<br />
Velero calculates the size of the temporary clone pvc based on the exact byte-count of the used/allocated snapshot metadata, not the #Gi specification of the original pvc.<br />
Just confirmed with a test<br />
<img src="/forum/assets/uploads/files/1784589688662-594a3691-1055-4aa3-a3f4-b35e731390d4-image-resized.jpeg" alt="594a3691-1055-4aa3-a3f4-b35e731390d4-image.jpeg" class=" img-fluid img-markdown" /></p>
]]></description><link>https://xcp-ng.org/forum/post/107113</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107113</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Tue, 21 Jul 2026 00:36:09 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 20 Jul 2026 18:54:23 GMT]]></title><description><![CDATA[<p dir="auto">Looks to be similar to last time</p>
<pre><code>Time,Node,Component / Service,Logged Event &amp; Impact
23:17:13,xen01,LINSTOR/Controller,Resource Created: linstor-csi provisions a short-lived volume: pvc-395da199-d455-4004-ac0d-f6c6016ebf4e.
23:17:16,xen01,kernel (drbd),Sync Begins: xen01 connects to xen05 as a SyncSource and begins synchronizing the volume to it.
23:17:49,xen01,LINSTOR/Controller,Teardown Requested: The CSI plugin changes its mind or finishes its task, requesting sequential deletion (DelRsc) of the volume replicas across the cluster.
23:17:51,xen01,LINSTOR/Controller,Source Disk Removed: Linstor removes the volume from xen01 itself (Toggle Disk ... removing disk).
23:17:52,xen01,kernel (drbd),Source Goes Diskless: xen01 detaches the underlying storage block (UpToDate -&gt; Detaching -&gt; Diskless). It alerts peers it can no longer satisfy read requests, calling drbd_bm_resize with capacity == 0.
23:17:53,xen05,LINSTOR/Satellite, While the cluster connection is actively tearing down, xen05's local background DeviceManager wakes up and detects the volume. It attempts an online re-alignment to round up the block layer size to match the 4MiB extent rule (26220040 KiB to 26222592 KiB).
23:17:54,xen05,kernel (drbd),Fatal Crash: xen05's DRBD driver tries to resize and count its synchronization bitmap block pointers. Because its peer source (xen01) dropped to zero capacity mid-flight, DRBD hits memory corruption. It triggers a General Protection Fault at drbd_bm_count_bits+0x223/0x300 and panics the kernel.
</code></pre>
]]></description><link>https://xcp-ng.org/forum/post/107111</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107111</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Mon, 20 Jul 2026 18:54:23 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 20 Jul 2026 16:48:29 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/poddingue" aria-label="Profile: poddingue">@<bdi>poddingue</bdi></a> This time it was host xen05. I got the request link for upload, so I have uploaded the bundle as well as all logs from xen05 and xen01 (xcp-ng pool master and had linstor-controller) So hoping we have everything needed. I will be digging into it to. It is in json format with all labels. Nothing in <code>/var/crash/</code></p>
]]></description><link>https://xcp-ng.org/forum/post/107109</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107109</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Mon, 20 Jul 2026 16:48:29 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 20 Jul 2026 07:24:31 GMT]]></title><description><![CDATA[<p dir="auto">On the bundle, the docs warn it can carry sensitive details about your setup, so better not to attach it publicly: <a href="https://docs.xcp-ng.org/troubleshooting/log-files#produce-a-status-report" target="_blank" rel="noopener noreferrer nofollow ugc">https://docs.xcp-ng.org/troubleshooting/log-files#produce-a-status-report</a>. As far as I can tell there is no self-serve upload spot, it is usually someone from Vates dropping you a private Nextcloud link when you ask, so flagging here that you have the bundle ready is probably the quickest route.</p>
<p dir="auto">Since this was a kernel panic, is there anything in <code>/var/crash</code> on the host that went down, and which host was it this time? The earlier ones were xen04 and xen03.</p>
]]></description><link>https://xcp-ng.org/forum/post/107083</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107083</guid><dc:creator><![CDATA[poddingue]]></dc:creator><pubDate>Mon, 20 Jul 2026 07:24:31 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 20 Jul 2026 07:00:18 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/poddingue" aria-label="Profile: poddingue">@<bdi>poddingue</bdi></a> Just had a new crash. Got the export <code>bug-report-20260719233946.tar.bz2</code> it is ~63MB There is a way to share large files but it has been a bit since I have done that so have forgotten.<br />
I've got all the logs. Will dig into it in the morning. Also looking into ways to share grafana links securely. We have it behind Cloudflare Zero Trust.<br />
<img src="/forum/assets/uploads/files/1784530775326-d16f2a36-3e18-4237-b134-99c29650cd72-image-resized.jpeg" alt="d16f2a36-3e18-4237-b134-99c29650cd72-image.jpeg" class=" img-fluid img-markdown" /><br />
Here you can see when the host was offline.<br />
<img src="/forum/assets/uploads/files/1784530795754-22c5da35-ac6e-4ed4-94ab-1a87fbd1def5-image-resized.jpeg" alt="22c5da35-ac6e-4ed4-94ab-1a87fbd1def5-image.jpeg" class=" img-fluid img-markdown" /></p>
]]></description><link>https://xcp-ng.org/forum/post/107081</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/107081</guid><dc:creator><![CDATA[Jonathon]]></dc:creator><pubDate>Mon, 20 Jul 2026 07:00:18 GMT</pubDate></item><item><title><![CDATA[Reply to xcp-ng server crashed/rebooted due to issues with drbd/linstor? on Mon, 20 Jul 2026 07:27:11 GMT]]></title><description><![CDATA[<p dir="auto">That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes. <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f914.png?v=f084dd9e819" class="not-responsive emoji emoji-android emoji--thinking_face" style="height:23px;width:auto;vertical-align:middle" title=":thinking_face:" alt="🤔" /><br />
Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 926 log lines around the panic suggests dom0 was under real pressure right then. <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f44d.png?v=f084dd9e819" class="not-responsive emoji emoji-android emoji--+1" style="height:23px;width:auto;vertical-align:middle" title=":+1:" alt="👍" /><br />
If it does come back, a <code>xen-bugtool --yestoall</code> bundle from the host that panicked, plus the Loki window you already have, would give <a class="plugin-mentions-group plugin-mentions-a" href="/forum/groups/team-storage" aria-label="Profile: Team-Storage">@<bdi>Team-Storage</bdi></a> something concrete to line up against the DRBD side.<br />
I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful.<br />
Fingers crossed it stays boring from here. <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f91e.png?v=f084dd9e819" class="not-responsive emoji emoji-android emoji--hand_with_index_and_middle_fingers_crossed" style="height:23px;width:auto;vertical-align:middle" title=":hand_with_index_and_middle_fingers_crossed:" alt="🤞" /></p>
]]></description><link>https://xcp-ng.org/forum/post/106903</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/106903</guid><dc:creator><![CDATA[poddingue]]></dc:creator><pubDate>Mon, 20 Jul 2026 07:27:11 GMT</pubDate></item></channel></rss>