xcp-ng server crashed/rebooted due to issues with drbd/linstor?

[poddingue]

The timing evidence here looks strong to me. Your second Grafana window is doing a lot of work: linstor-satellite.service jumps to roughly 8,700 log lines in the three minutes around the panic, well above linstor-controller at 926 and xapi at 460, and the priority chart goes red at the same moment. A backup that fans out volume creates and deletes across three replicas, landing on a DRBD race, fits what you are seeing.

Throttling Velero should tell you a lot. If the crashes stop with clientQPS: 3 and itemBlockWorkerCount: 1, that narrows it to concurrency rather than anything about those particular volumes.

For the logs question, a xen-bugtool --yestoall bundle from a host that has panicked is usually the thing people ask for first (https://docs.xcp-ng.org/troubleshooting/log-files), since it sweeps up the kernel side alongside the storage logs you already have in Loki. I don't know DRBD internals well enough to say which trace matters most, so it might be worth a mention to @Team-Storage. They can say what they actually need rather than have you guess, and a panic that reproduces on a schedule is a good deal easier for them to chase than most.

[Jonathon]

@poddingue

I should already have all the logs, including kernel

Nothing has happened since. I am also increasing dom0 ram for all hosts.

[poddingue]

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes.
Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 926 log lines around the panic suggests dom0 was under real pressure right then.
If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side.
I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful.
Fingers crossed it stays boring from here.

[Jonathon]

@poddingue Just had a new crash. Got the export bug-report-20260719233946.tar.bz2 it is ~63MB There is a way to share large files but it has been a bit since I have done that so have forgotten.
I've got all the logs. Will dig into it in the morning. Also looking into ways to share grafana links securely. We have it behind Cloudflare Zero Trust.

Here you can see when the host was offline.

[poddingue]

On the bundle, the docs warn it can carry sensitive details about your setup, so better not to attach it publicly: https://docs.xcp-ng.org/troubleshooting/log-files#produce-a-status-report. As far as I can tell there is no self-serve upload spot, it is usually someone from Vates dropping you a private Nextcloud link when you ask, so flagging here that you have the bundle ready is probably the quickest route.

Since this was a kernel panic, is there anything in /var/crash on the host that went down, and which host was it this time? The earlier ones were xen04 and xen03.

[Jonathon]

@poddingue This time it was host xen05. I got the request link for upload, so I have uploaded the bundle as well as all logs from xen05 and xen01 (xcp-ng pool master and had linstor-controller) So hoping we have everything needed. I will be digging into it to. It is in json format with all labels. Nothing in /var/crash/

[Jonathon]

Looks to be similar to last time

Time,Node,Component / Service,Logged Event & Impact
23:17:13,xen01,LINSTOR/Controller,Resource Created: linstor-csi provisions a short-lived volume: pvc-395da199-d455-4004-ac0d-f6c6016ebf4e.
23:17:16,xen01,kernel (drbd),Sync Begins: xen01 connects to xen05 as a SyncSource and begins synchronizing the volume to it.
23:17:49,xen01,LINSTOR/Controller,Teardown Requested: The CSI plugin changes its mind or finishes its task, requesting sequential deletion (DelRsc) of the volume replicas across the cluster.
23:17:51,xen01,LINSTOR/Controller,Source Disk Removed: Linstor removes the volume from xen01 itself (Toggle Disk ... removing disk).
23:17:52,xen01,kernel (drbd),Source Goes Diskless: xen01 detaches the underlying storage block (UpToDate -> Detaching -> Diskless). It alerts peers it can no longer satisfy read requests, calling drbd_bm_resize with capacity == 0.
23:17:53,xen05,LINSTOR/Satellite, While the cluster connection is actively tearing down, xen05's local background DeviceManager wakes up and detects the volume. It attempts an online re-alignment to round up the block layer size to match the 4MiB extent rule (26220040 KiB to 26222592 KiB).
23:17:54,xen05,kernel (drbd),Fatal Crash: xen05's DRBD driver tries to resize and count its synchronization bitmap block pointers. Because its peer source (xen01) dropped to zero capacity mid-flight, DRBD hits memory corruption. It triggers a General Protection Fault at drbd_bm_count_bits+0x223/0x300 and panics the kernel.

[Jonathon]

My first idea was to have velero use a different storage class. Something like

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-ephemeral-retain
parameters:
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/disklessOnRemaining: "true"
  linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool
  linstor.csi.linbit.com/placementCount: "3"
provisioner: linstor.csi.linbit.com
**reclaimPolicy: Retain**
volumeBindingMode: WaitForFirstConsumer

Then have a cronjob collected the Released pvcs every hour. The problem is that can not work, as velero uses the storage class of the backed up pvc. So I would have to change every linstor storage class to Retain, and would effectively disable the automatic garbage collection for the cluster. Which might be fine, but would love to avoid it.

My second idea is to eliminate the need for the resizing (another work around to hopefully get away from this race condition/bug)
I am also not seeing any way to modify the storageclass to round up or pad the pv. This would be to attempt to eliminate the need for drbd_bm_resize.
Currently looking into Kyverno Mutating Webhook to dynamically pad these volumes on the fly to see if that would help me out at all.
All pvs are already whole numbers of Gi.
Velero calculates the size of the temporary clone pvc based on the exact byte-count of the used/allocated snapshot metadata, not the #Gi specification of the original pvc.
Just confirmed with a test

[Jonathon]

I was just moving around the velero backup schedules when it decided to run some again even though it was not time. Crashed xen04 this time. Deleting the possible offending backup schedule until this bug can be identified and fixed.

Uploaded new bug bundle to nextcloud too.

[Jonathon]

Logs from xen04 look like they are the same too. Which makes sense.

20:01:53.805: The Linstor DeviceManager on xen04 triggered an asynchronous alignment for a newly created volume (pvc-fac0a18b-2c23-4fe8-b005-b4f492cec92f), padding it from 26220040 KiB to 26222592 KiB.

20:01:54.034: Simultaneously, an orchestrator (like Velero) began tearing the volume down, shifting the DRBD state to conn( Connected -> TearDown ) and pdsk( Diskless -> DUnknown ).

20:01:58.484: The DRBD worker thread evaluated the shrinking bitmap while the alignment was trying to alter it. This caused the general protection fault: 0000 [#1] SMP NOPTI at instruction RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd], crashing the kernel.

[poddingue]

That timeline is the clearest version of this yet. xen04 going down the same way as xen05 makes the race look real to me rather than coincidental, unless I am reading the ordering wrong.
The fault lands inside drbd_bm_count_bits, so in the DRBD module itself, which makes me think this is an upstream LINBIT question as much as an XCP-ng one. I do not know enough about how that module gets packaged here to say who owns the fix though. Might be worth another mention to @Team-Storage so someone can open your bundles with your post #10 next to them.
On the padding idea I am out of my depth: no idea whether LINSTOR can be told to pre-round a volume at creation, but asking LINBIT directly seems worth a try since it would sidestep drbd_bm_resize entirely.