Happens at around the same time. Which point to something on a schedule, and I think it has been found.
TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash.
I have attempted to slow down velero with the following helm chart value change. Will see if it happens again.
configuration:
# Lowers API call velocity to stagger volume Create/Delete cycles
clientQPS: 3
clientBurst: 5
# Forces serial execution across volume structures, preventing concurrent lifecycle operations
itemBlockWorkerCount: 1
I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd?
Timeline of crash
On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host ovbh-pprod-xen04 suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module.
It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time.
Started: 2026-07-04 23:15:00 -0700 PDT
Completed: 2026-07-04 23:17:33 -0700 PDT
Stage 1: Backup Initialization & Storage Provisioning Burst
Timestamp: 23:15:17 - 23:15:26 PDT
Velero starts the openvas-daily-20260705061500 snapshot which has PVC volumes (gvmd-pvc, openvas-pvc, psql-pvc). The linstor-csi-controller issues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings across xen01, xen02, and xen05. It then dynamically drops xen02 and redistributes the active target allocation mirror directly onto xen04 for some reason.
2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""}
2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01'
2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02'
2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04'
2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com'
Stage 2: Rapid Backup Job Completion & Teardown Execution
Timestamp: 23:15:38 - 23:16:01 PDT
The temporary target volume (openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops to Released. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediate DeleteResource directive to all replication satellites, commanding xen04 to disconnect the live disk configuration mapping.
2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""}
2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""}
2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""}
2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""}
2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk
2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
Stage 3: Memory Teardown, GPF, and Hard Kernel Panic
Timestamp: 23:16:03 - 23:16:05 PDT
As xen04 unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault.
Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context.
2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI
2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G O 4.19.0+1 #1
2026-07-04T23:16:03-07:00 Hardware name: /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025
2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd]
2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48
2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016
2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0
2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0
2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI
2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G D O 4.19.0+1 #1
2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20
2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660
2026-07-04T23:16:05-07:00 Call Trace:
2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd]
2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd]
2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd]
2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd]
Stage 4: Cascading Failures & Pool Master Dropout
Timestamp: 23:16:05 - 23:21:50 PDT
As xen04 drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The core etcd consensus cluster node on master01 encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At 23:21:50, the Pool Master node xen01 officially declares xen04 dead after exhausting its network PingAck heartbeat retry limits.
2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"}
2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time.
Full velero details
jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details
Name: openvas-daily-20260705061500
Namespace: velero
Labels: app.kubernetes.io/name=openvas
velero.io/schedule-name=openvas-daily
velero.io/storage-location=default
Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA
objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f
velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=35
Phase: Completed
Namespaces:
Included: openvas
Excluded: <none>
Resources:
Included cluster-scoped: <none>
Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io
Included namespace-scoped: *
Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io
Label selector: <none>
Or label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: true
Snapshot Move Data: true
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 30m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2026-07-04 23:15:00 -0700 PDT
Completed: 2026-07-04 23:17:33 -0700 PDT
Expiration: 2026-08-03 23:15:00 -0700 PDT
Total items to be backed up: 42
Items backed up: 42
Backup Item Operations:
Operation for persistentvolumeclaims openvas/gvmd-pvc:
Backup Item Action Plugin: velero.io/csi-pvc-backupper
Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5
Items to Update:
datauploads.velero.io velero/openvas-daily-20260705061500-v86h2
Phase: Completed
Progress: 8896914780 of 8896914780 complete (Bytes)
Progress description: Completed
Created: 2026-07-04 23:15:08 -0700 PDT
Started: 2026-07-04 23:15:30 -0700 PDT
Updated: 2026-07-04 23:17:11 -0700 PDT
Operation for persistentvolumeclaims openvas/openvas-pvc:
Backup Item Action Plugin: velero.io/csi-pvc-backupper
Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803
Items to Update:
datauploads.velero.io velero/openvas-daily-20260705061500-92dtr
Phase: Completed
Progress: 5540 of 5540 complete (Bytes)
Progress description: Completed
Created: 2026-07-04 23:15:13 -0700 PDT
Started: 2026-07-04 23:15:38 -0700 PDT
Updated: 2026-07-04 23:16:54 -0700 PDT
Operation for persistentvolumeclaims openvas/psql-pvc:
Backup Item Action Plugin: velero.io/csi-pvc-backupper
Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8
Items to Update:
datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd
Phase: Completed
Progress: 24267413354 of 24267413354 complete (Bytes)
Progress description: Completed
Created: 2026-07-04 23:15:18 -0700 PDT
Started: 2026-07-04 23:15:42 -0700 PDT
Updated: 2026-07-04 23:17:15 -0700 PDT
Resource List:
apiextensions.k8s.io/v1/CustomResourceDefinition:
- certificaterequests.cert-manager.io
- certificates.cert-manager.io
- clusterserviceversions.operators.coreos.com
apps/v1/Deployment:
- openvas/openvas
- openvas/test
apps/v1/ReplicaSet:
- openvas/openvas-64786df449
- openvas/openvas-69bb998d7c
- openvas/openvas-6db7bcd6ff
- openvas/openvas-6dd46f4f79
- openvas/openvas-6f8d45b667
- openvas/openvas-78dc6f74c6
- openvas/openvas-7c5654b9d6
- openvas/openvas-844c5f796d
- openvas/openvas-86d9b75d9f
- openvas/openvas-b97749f55
- openvas/openvas-d9d48c74
- openvas/test-5879777846
- openvas/test-5bd7f76f8c
- openvas/test-6487578786
- openvas/test-6b467857f5
cert-manager.io/v1/Certificate:
- openvas/openvas-ingress-cert-fp-pki
cert-manager.io/v1/CertificateRequest:
- openvas/openvas-ingress-cert-fp-pki-15
discovery.k8s.io/v1/EndpointSlice:
- openvas/openvas-service-6zghf
networking.k8s.io/v1/Ingress:
- openvas/openvas
operators.coreos.com/v1alpha1/ClusterServiceVersion:
- openvas/node-healthcheck-operator.v0.11.0
- openvas/self-node-remediation.v0.12.1
v1/ConfigMap:
- openvas/elasticsearch-es-transport-ca-internal
- openvas/init-scripts-config
- openvas/kube-root-ca.crt
v1/Endpoints:
- openvas/openvas-service
v1/Namespace:
- openvas
v1/PersistentVolume:
- pvc-105299a3-d117-4562-a2e7-8b6be53cfde0
- pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2
- pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4
v1/PersistentVolumeClaim:
- openvas/gvmd-pvc
- openvas/openvas-pvc
- openvas/psql-pvc
v1/Pod:
- openvas/openvas-78dc6f74c6-wcwp2
v1/Secret:
- openvas/gvmd-secrets
- openvas/openvas-ingress-tls-fp-pki
v1/Service:
- openvas/openvas-service
v1/ServiceAccount:
- openvas/default
Backup Volumes:
Velero-Native Snapshots: <none included>
CSI Snapshots:
openvas/gvmd-pvc:
Data Movement:
Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5
Data Mover: velero
Uploader Type: kopia
Moved data Size (bytes): 8896914780
openvas/openvas-pvc:
Data Movement:
Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803
Data Mover: velero
Uploader Type: kopia
Moved data Size (bytes): 5540
openvas/psql-pvc:
Data Movement:
Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8
Data Mover: velero
Uploader Type: kopia
Moved data Size (bytes): 24267413354
Pod Volume Backups: <none included>
HooksAttempted: 0
HooksFailed: 0
linstor-pvc-velero-logs.txt
xen04-07042026-panic.txt
linstor-controller-logs.txt
