xcp-ng server crashed/rebooted due to issues with drbd/linstor?
-
Happens at around the same time. Which point to something on a schedule, and I think it has been found.
TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash.
I have attempted to slow down velero with the following helm chart value change. Will see if it happens again.configuration: # Lowers API call velocity to stagger volume Create/Delete cycles clientQPS: 3 clientBurst: 5 # Forces serial execution across volume structures, preventing concurrent lifecycle operations itemBlockWorkerCount: 1I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd?
Timeline of crash
On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host
ovbh-pprod-xen04suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module.
It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time.Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDTStage 1: Backup Initialization & Storage Provisioning Burst
Timestamp:
23:15:17 - 23:15:26 PDTVelero starts the
openvas-daily-20260705061500snapshot which has PVC volumes (gvmd-pvc,openvas-pvc,psql-pvc). Thelinstor-csi-controllerissues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings acrossxen01,xen02, andxen05. It then dynamically dropsxen02and redistributes the active target allocation mirror directly ontoxen04for some reason.2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01' 2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02' 2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04' 2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com'Stage 2: Rapid Backup Job Completion & Teardown Execution
Timestamp:
23:15:38 - 23:16:01 PDTThe temporary target volume (
openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops toReleased. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediateDeleteResourcedirective to all replication satellites, commandingxen04to disconnect the live disk configuration mapping.2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""} 2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""} 2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""} 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk 2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1Stage 3: Memory Teardown, GPF, and Hard Kernel Panic
Timestamp:
23:16:03 - 23:16:05 PDTAs
xen04unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault.Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside
rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context.2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI 2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G O 4.19.0+1 #1 2026-07-04T23:16:03-07:00 Hardware name: /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025 2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd] 2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48 2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016 2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0 2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0 2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI 2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G D O 4.19.0+1 #1 2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20 2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660 2026-07-04T23:16:05-07:00 Call Trace: 2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd] 2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd]Stage 4: Cascading Failures & Pool Master Dropout
Timestamp:
23:16:05 - 23:21:50 PDTAs
xen04drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The coreetcdconsensus cluster node onmaster01encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At23:21:50, the Pool Master nodexen01officially declaresxen04dead after exhausting its networkPingAckheartbeat retry limits.2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"} 2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time.Full velero details
jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details Name: openvas-daily-20260705061500 Namespace: velero Labels: app.kubernetes.io/name=openvas velero.io/schedule-name=openvas-daily velero.io/storage-location=default Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=35 Phase: Completed Namespaces: Included: openvas Excluded: <none> Resources: Included cluster-scoped: <none> Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io Included namespace-scoped: * Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: true Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 30m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Expiration: 2026-08-03 23:15:00 -0700 PDT Total items to be backed up: 42 Items backed up: 42 Backup Item Operations: Operation for persistentvolumeclaims openvas/gvmd-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-v86h2 Phase: Completed Progress: 8896914780 of 8896914780 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:08 -0700 PDT Started: 2026-07-04 23:15:30 -0700 PDT Updated: 2026-07-04 23:17:11 -0700 PDT Operation for persistentvolumeclaims openvas/openvas-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-92dtr Phase: Completed Progress: 5540 of 5540 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:13 -0700 PDT Started: 2026-07-04 23:15:38 -0700 PDT Updated: 2026-07-04 23:16:54 -0700 PDT Operation for persistentvolumeclaims openvas/psql-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd Phase: Completed Progress: 24267413354 of 24267413354 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:18 -0700 PDT Started: 2026-07-04 23:15:42 -0700 PDT Updated: 2026-07-04 23:17:15 -0700 PDT Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - certificaterequests.cert-manager.io - certificates.cert-manager.io - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - openvas/openvas - openvas/test apps/v1/ReplicaSet: - openvas/openvas-64786df449 - openvas/openvas-69bb998d7c - openvas/openvas-6db7bcd6ff - openvas/openvas-6dd46f4f79 - openvas/openvas-6f8d45b667 - openvas/openvas-78dc6f74c6 - openvas/openvas-7c5654b9d6 - openvas/openvas-844c5f796d - openvas/openvas-86d9b75d9f - openvas/openvas-b97749f55 - openvas/openvas-d9d48c74 - openvas/test-5879777846 - openvas/test-5bd7f76f8c - openvas/test-6487578786 - openvas/test-6b467857f5 cert-manager.io/v1/Certificate: - openvas/openvas-ingress-cert-fp-pki cert-manager.io/v1/CertificateRequest: - openvas/openvas-ingress-cert-fp-pki-15 discovery.k8s.io/v1/EndpointSlice: - openvas/openvas-service-6zghf networking.k8s.io/v1/Ingress: - openvas/openvas operators.coreos.com/v1alpha1/ClusterServiceVersion: - openvas/node-healthcheck-operator.v0.11.0 - openvas/self-node-remediation.v0.12.1 v1/ConfigMap: - openvas/elasticsearch-es-transport-ca-internal - openvas/init-scripts-config - openvas/kube-root-ca.crt v1/Endpoints: - openvas/openvas-service v1/Namespace: - openvas v1/PersistentVolume: - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2 - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4 v1/PersistentVolumeClaim: - openvas/gvmd-pvc - openvas/openvas-pvc - openvas/psql-pvc v1/Pod: - openvas/openvas-78dc6f74c6-wcwp2 v1/Secret: - openvas/gvmd-secrets - openvas/openvas-ingress-tls-fp-pki v1/Service: - openvas/openvas-service v1/ServiceAccount: - openvas/default Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: openvas/gvmd-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 8896914780 openvas/openvas-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 5540 openvas/psql-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 24267413354 Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0
linstor-pvc-velero-logs.txt
xen04-07042026-panic.txt
linstor-controller-logs.txt -
The timing evidence here looks strong to me. Your second Grafana window is doing a lot of work:
linstor-satellite.servicejumps to roughly 8,700 log lines in the three minutes around the panic, well abovelinstor-controllerat 926 andxapiat 460, and the priority chart goes red at the same moment. A backup that fans out volume creates and deletes across three replicas, landing on a DRBD race, fits what you are seeing.Throttling Velero should tell you a lot. If the crashes stop with
clientQPS: 3anditemBlockWorkerCount: 1, that narrows it to concurrency rather than anything about those particular volumes.For the logs question, a
xen-bugtool --yestoallbundle from a host that has panicked is usually the thing people ask for first (https://docs.xcp-ng.org/troubleshooting/log-files), since it sweeps up the kernel side alongside the storage logs you already have in Loki. I don't know DRBD internals well enough to say which trace matters most, so it might be worth a mention to @Team-Storage. They can say what they actually need rather than have you guess, and a panic that reproduces on a schedule is a good deal easier for them to chase than most. -
I should already have all the logs, including kernel
Nothing has happened since. I am also increasing dom0 ram for all hosts.
-
That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes.
Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 926 log lines around the panic suggests dom0 was under real pressure right then.
If it does come back, axen-bugtool --yestoallbundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side.
I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful.
Fingers crossed it stays boring from here.
-
@poddingue Just had a new crash. Got the export
bug-report-20260719233946.tar.bz2it is ~63MB There is a way to share large files but it has been a bit since I have done that so have forgotten.
I've got all the logs. Will dig into it in the morning. Also looking into ways to share grafana links securely. We have it behind Cloudflare Zero Trust.
Here you can see when the host was offline.
-
On the bundle, the docs warn it can carry sensitive details about your setup, so better not to attach it publicly: https://docs.xcp-ng.org/troubleshooting/log-files#produce-a-status-report. As far as I can tell there is no self-serve upload spot, it is usually someone from Vates dropping you a private Nextcloud link when you ask, so flagging here that you have the bundle ready is probably the quickest route.
Since this was a kernel panic, is there anything in
/var/crashon the host that went down, and which host was it this time? The earlier ones were xen04 and xen03. -
@poddingue This time it was host xen05. I got the request link for upload, so I have uploaded the bundle as well as all logs from xen05 and xen01 (xcp-ng pool master and had linstor-controller) So hoping we have everything needed. I will be digging into it to. It is in json format with all labels. Nothing in
/var/crash/ -
Looks to be similar to last time
Time,Node,Component / Service,Logged Event & Impact 23:17:13,xen01,LINSTOR/Controller,Resource Created: linstor-csi provisions a short-lived volume: pvc-395da199-d455-4004-ac0d-f6c6016ebf4e. 23:17:16,xen01,kernel (drbd),Sync Begins: xen01 connects to xen05 as a SyncSource and begins synchronizing the volume to it. 23:17:49,xen01,LINSTOR/Controller,Teardown Requested: The CSI plugin changes its mind or finishes its task, requesting sequential deletion (DelRsc) of the volume replicas across the cluster. 23:17:51,xen01,LINSTOR/Controller,Source Disk Removed: Linstor removes the volume from xen01 itself (Toggle Disk ... removing disk). 23:17:52,xen01,kernel (drbd),Source Goes Diskless: xen01 detaches the underlying storage block (UpToDate -> Detaching -> Diskless). It alerts peers it can no longer satisfy read requests, calling drbd_bm_resize with capacity == 0. 23:17:53,xen05,LINSTOR/Satellite, While the cluster connection is actively tearing down, xen05's local background DeviceManager wakes up and detects the volume. It attempts an online re-alignment to round up the block layer size to match the 4MiB extent rule (26220040 KiB to 26222592 KiB). 23:17:54,xen05,kernel (drbd),Fatal Crash: xen05's DRBD driver tries to resize and count its synchronization bitmap block pointers. Because its peer source (xen01) dropped to zero capacity mid-flight, DRBD hits memory corruption. It triggers a General Protection Fault at drbd_bm_count_bits+0x223/0x300 and panics the kernel. -
My first idea was to have velero use a different storage class. Something like
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: linstor-ephemeral-retain parameters: linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" linstor.csi.linbit.com/disklessOnRemaining: "true" linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool linstor.csi.linbit.com/placementCount: "3" provisioner: linstor.csi.linbit.com **reclaimPolicy: Retain** volumeBindingMode: WaitForFirstConsumerThen have a cronjob collected the
Releasedpvcs every hour. The problem is that can not work, as velero uses the storage class of the backed up pvc. So I would have to change every linstor storage class toRetain, and would effectively disable the automatic garbage collection for the cluster. Which might be fine, but would love to avoid it.My second idea is to eliminate the need for the resizing (another work around to hopefully get away from this race condition/bug)
I am also not seeing any way to modify the storageclass to round up or pad the pv. This would be to attempt to eliminate the need fordrbd_bm_resize.
Currently looking into Kyverno Mutating Webhook to dynamically pad these volumes on the fly to see if that would help me out at all.
All pvs are already whole numbers of Gi.
Velero calculates the size of the temporary clone pvc based on the exact byte-count of the used/allocated snapshot metadata, not the #Gi specification of the original pvc.
Just confirmed with a test
-
I was just moving around the velero backup schedules when it decided to run some again even though it was not time. Crashed xen04 this time. Deleting the possible offending backup schedule until this bug can be identified and fixed.
Uploaded new bug bundle to nextcloud too.
-
Logs from xen04 look like they are the same too. Which makes sense.
20:01:53.805: The Linstor DeviceManager on xen04 triggered an asynchronous alignment for a newly created volume (pvc-fac0a18b-2c23-4fe8-b005-b4f492cec92f), padding it from 26220040 KiB to 26222592 KiB. 20:01:54.034: Simultaneously, an orchestrator (like Velero) began tearing the volume down, shifting the DRBD state to conn( Connected -> TearDown ) and pdsk( Diskless -> DUnknown ). 20:01:58.484: The DRBD worker thread evaluated the shrinking bitmap while the alignment was trying to alter it. This caused the general protection fault: 0000 [#1] SMP NOPTI at instruction RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd], crashing the kernel. -
That timeline is the clearest version of this yet. xen04 going down the same way as xen05 makes the race look real to me rather than coincidental, unless I am reading the ordering wrong.
The fault lands insidedrbd_bm_count_bits, so in the DRBD module itself, which makes me think this is an upstream LINBIT question as much as an XCP-ng one. I do not know enough about how that module gets packaged here to say who owns the fix though. Might be worth another mention to @Team-Storage so someone can open your bundles with your post #10 next to them.
On the padding idea I am out of my depth: no idea whether LINSTOR can be told to pre-round a volume at creation, but asking LINBIT directly seems worth a try since it would sidestepdrbd_bm_resizeentirely.
-
@jonathon are you using your LINSTOR/DRBD stack in the form of licensed XOSTOR? Your issue looks a case for pro support. Did you consider opening a case with pro support?
-
@MajorP93 No we do not have pro support at the moment
hmmm. Really seems like a build with
9.2.19could fix all of this.
How can I request for a build @poddingue ?https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L61
9.2.17- Fix a kernel crash triggered by a crafted/invalid netlink message
https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L24
9.2.19- Fix an AB-BA deadlock between online resize and activity-log transactions
- Fix several races during connection teardown that could crash or
hang (ack_sender requeue, pending ping work, in-progress lb-tcp
connect)
-
On the version question, 8.3 ships DRBD 9.2.18 right now (that's what
kmod-drbd.specsays on the 8.3 branch), so you'd be asking for a single point bump rather than a big jump.
That probably helps you. Of your two links though, I think the netlink crash fix is a 9.2.17 entry, which would mean it's already in what you're running.
The 9.2.19 line that looks closest to your crash to me is the connection-teardown races one rather than the AB-BA deadlock, since a deadlock would hang rather than throw a GPF, but you've read these traces far more closely than I have so correct me if that's backwards.
I can't request a build myself, that's a @Team-Storage call rather than mine, though your post #13 timeline next to those changelog lines is a stronger case than most version-bump requests turn up with. -
@poddingue I'm fully up to date, but only at version
9.2.16[14:57 ovbh-pprod-xen02 ~]# drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36 DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION_CODE=0x090210 DRBD_KERNEL_VERSION=9.2.16 DRBDADM_VERSION_CODE=0x092100 DRBDADM_VERSION=9.33.0 [14:57 ovbh-pprod-xen02 ~]# drbd-reactor --version drbd-reactor 1.9.0 [14:57 ovbh-pprod-xen02 ~]#
There are no updates that I can install
[14:08 ovbh-pprod-xen02 ~]# yum check-update Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org zabbix | 3.0 kB 00:00:00 zabbix-non-supported | 2.9 kB 00:00:00 zabbix/x86_64/primary_db | 114 kB 00:00:00 zabbix-agent.x86_64 7.0.28-release1.el7 zabbix -
For now I have upgraded all hosts to use
--enablerepo=xcp-ng-linstor-testingto getkmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3
That way it is there to be picked up if the host crashes again.
https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec
https://koji.xcp-ng.org/buildinfo?buildID=5747[12:15 ovbh-pprod-xen04 ~]# yum upgrade --enablerepo=xcp-ng-linstor-testing Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org grafana/signature | 810 B 00:00:00 grafana/signature | 3.0 kB 00:00:00 !!! xcp-ng-linstor-testing/signature | 473 B 00:00:00 xcp-ng-linstor-testing/signature | 3.0 kB 00:00:00 !!! zabbix | 3.0 kB 00:00:00 zabbix-non-supported | 2.9 kB 00:00:00 (1/3): grafana/primary_db | 879 kB 00:00:00 (2/3): xcp-ng-linstor-testing/primary_db | 2.4 kB 00:00:00 (3/3): zabbix/x86_64/primary_db | 114 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package kmod-drbd.x86_64 0:9.2.16-1.0.xcpng8.3 will be updated ---> Package kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 will be an update ---> Package zabbix-agent.x86_64 0:7.0.27-release1.el7 will be updated ---> Package zabbix-agent.x86_64 0:7.0.28-release1.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================== Updating: kmod-drbd x86_64 9.2.18-2.0.xcpng8.3 xcp-ng-linstor-testing 2.8 M zabbix-agent x86_64 7.0.28-release1.el7 zabbix 660 k Transaction Summary ============================================================================================================================================================================================================================================================== Upgrade 2 Packages Total download size: 3.4 M Is this ok [y/d/N]: y Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/2): zabbix-agent-7.0.28-release1.el7.x86_64.rpm | 660 kB 00:00:00 (2/2): kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64.rpm | 2.8 MB 00:00:01 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 2.8 MB/s | 3.4 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64 1/4 Updating : zabbix-agent-7.0.28-release1.el7.x86_64 2/4 Cleanup : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64 3/4 Cleanup : zabbix-agent-7.0.27-release1.el7.x86_64 4/4 Verifying : zabbix-agent-7.0.28-release1.el7.x86_64 1/4 Verifying : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64 2/4 Verifying : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64 3/4 Verifying : zabbix-agent-7.0.27-release1.el7.x86_64 4/4 Updated: kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 zabbix-agent.x86_64 0:7.0.28-release1.el7 Complete! -
Oopsie, I was wrong.
I owe you a correction on the version, and that one is on me. I said 8.3 ships 9.2.18, but I was reading the spec file on the branch rather than what you actually get from the stable repo.
Your yum check-update output answers it better than I did: stable leaves you on 9.2.16, and 9.2.18 only turns up once xcp-ng-linstor-testing is enabled.The screenshot in that same post is the part the still puzzles me. XO says "All good, patches up to date" while the module underneath is two point releases back, which I assume is what everyone else running XOSTOR sees too. If the connection-teardown fixes you found really are in 9.2.19, that is three releases of distance on the exact code path your traces land in, not the single bump I described.
-
Hello,
Indeed this is a known issue with XOSTOR, when causing a sync of too many resources at once it's possible for a DRBD resource to leak memory during the resync. A patch on the 9.2.18 fixes this by throttling the number of resources that can be synced at once to mitigate the issue.
The 9.2.18 update should be in the main xcp-ng-linstor repo very soon.
What you can do too is give more RAM to the dom0 of your hosts. How many RAM have been allocated to dom0 at the moment? We advise 16GB or 32GB depending on the number of volumes.
-
@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login