xcp-ng server crashed/rebooted due to issues with drbd/linstor?

[Jonathon]

Hello, for the last few weeks we have been having random server crashes/reboots. This last crash happened on the June 2026 Updates #1 for XCP-ng 8.3 LTS update. I see a new one just came out.

I can share more logs if desired.
xen03-jun20-journalctl-crash-summary.txt

The timeline is a bit fuzzy, as we started to experience vendor issues and they stated that there was an issue with the power supply. But we have continued to have issues with that server. Now a new server (xen03) has crashed/rebooted. Will add more to this thread as I collect and go through it.

[Jonathon]

Here is the log from when xen04 last crashed xen04-jun12-journalctl-crash-summary.txt

[Jonathon]

I've installed the latest updates to all hosts, and restarted everything. I also have all xcp-ng logs going into loki now, so next time something happens I will see everything lol.

[Jonathon]

Happens at around the same time. Which point to something on a schedule, and I think it has been found.

TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash.
I have attempted to slow down velero with the following helm chart value change. Will see if it happens again.

configuration:
  # Lowers API call velocity to stagger volume Create/Delete cycles
  clientQPS: 3
  clientBurst: 5
  
  # Forces serial execution across volume structures, preventing concurrent lifecycle operations
  itemBlockWorkerCount: 1

I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd?

---

Timeline of crash

On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host ovbh-pprod-xen04 suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module.
It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time.

Started:    2026-07-04 23:15:00 -0700 PDT
Completed:  2026-07-04 23:17:33 -0700 PDT

Stage 1: Backup Initialization & Storage Provisioning Burst

Timestamp: 23:15:17 - 23:15:26 PDT

Velero starts the openvas-daily-20260705061500 snapshot which has PVC volumes (gvmd-pvc, openvas-pvc, psql-pvc). The linstor-csi-controller issues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings across xen01, xen02, and xen05. It then dynamically drops xen02 and redistributes the active target allocation mirror directly onto xen04 for some reason.

2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982      69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""}
2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO  LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO  LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01'
2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO  LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02'
2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO  LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04'
2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO  LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO  LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com'

Stage 2: Rapid Backup Job Completion & Teardown Execution

Timestamp: 23:15:38 - 23:16:01 PDT

The temporary target volume (openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops to Released. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediate DeleteResource directive to all replication satellites, commanding xen04 to disconnect the live disk configuration mapping.

2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""}
2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""}
2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""}
2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""}
2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO  LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO  LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk
2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO  LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1

Stage 3: Memory Teardown, GPF, and Hard Kernel Panic

Timestamp: 23:16:03 - 23:16:05 PDT

As xen04 unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault.

Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context.

2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI
2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G           O      4.19.0+1 #1
2026-07-04T23:16:03-07:00 Hardware name:  /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025
2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd]
2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48
2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016
2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0
2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0
2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI
2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G      D    O      4.19.0+1 #1
2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20
2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660
2026-07-04T23:16:05-07:00 Call Trace:
2026-07-04T23:16:05-07:00  drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd]
2026-07-04T23:16:05-07:00  ? conn_disconnect+0x19c/0xce0 [drbd]
2026-07-04T23:16:05-07:00  ? drbd_receiver+0x361/0x9a0 [drbd]
2026-07-04T23:16:05-07:00  ? drbd_thread_setup+0xd0/0x230 [drbd]

Stage 4: Cascading Failures & Pool Master Dropout

Timestamp: 23:16:05 - 23:21:50 PDT

As xen04 drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The core etcd consensus cluster node on master01 encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At 23:21:50, the Pool Master node xen01 officially declares xen04 dead after exhausting its network PingAck heartbeat retry limits.

2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"}
2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time.

Full velero details

jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details
Name:         openvas-daily-20260705061500
Namespace:    velero
Labels:       app.kubernetes.io/name=openvas
              velero.io/schedule-name=openvas-daily
              velero.io/storage-location=default
Annotations:  objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA
              objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f
              velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=35

Phase:  Completed


Namespaces:
  Included:  openvas
  Excluded:  <none>

Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    30m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2026-07-04 23:15:00 -0700 PDT
Completed:  2026-07-04 23:17:33 -0700 PDT

Expiration:  2026-08-03 23:15:00 -0700 PDT

Total items to be backed up:  42
Items backed up:              42

Backup Item Operations:
  Operation for persistentvolumeclaims openvas/gvmd-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5
    Items to Update:
                           datauploads.velero.io velero/openvas-daily-20260705061500-v86h2
    Phase:                 Completed
    Progress:              8896914780 of 8896914780 complete (Bytes)
    Progress description:  Completed
    Created:               2026-07-04 23:15:08 -0700 PDT
    Started:               2026-07-04 23:15:30 -0700 PDT
    Updated:               2026-07-04 23:17:11 -0700 PDT
  Operation for persistentvolumeclaims openvas/openvas-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803
    Items to Update:
                           datauploads.velero.io velero/openvas-daily-20260705061500-92dtr
    Phase:                 Completed
    Progress:              5540 of 5540 complete (Bytes)
    Progress description:  Completed
    Created:               2026-07-04 23:15:13 -0700 PDT
    Started:               2026-07-04 23:15:38 -0700 PDT
    Updated:               2026-07-04 23:16:54 -0700 PDT
  Operation for persistentvolumeclaims openvas/psql-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8
    Items to Update:
                           datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd
    Phase:                 Completed
    Progress:              24267413354 of 24267413354 complete (Bytes)
    Progress description:  Completed
    Created:               2026-07-04 23:15:18 -0700 PDT
    Started:               2026-07-04 23:15:42 -0700 PDT
    Updated:               2026-07-04 23:17:15 -0700 PDT
Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - certificaterequests.cert-manager.io
    - certificates.cert-manager.io
    - clusterserviceversions.operators.coreos.com
  apps/v1/Deployment:
    - openvas/openvas
    - openvas/test
  apps/v1/ReplicaSet:
    - openvas/openvas-64786df449
    - openvas/openvas-69bb998d7c
    - openvas/openvas-6db7bcd6ff
    - openvas/openvas-6dd46f4f79
    - openvas/openvas-6f8d45b667
    - openvas/openvas-78dc6f74c6
    - openvas/openvas-7c5654b9d6
    - openvas/openvas-844c5f796d
    - openvas/openvas-86d9b75d9f
    - openvas/openvas-b97749f55
    - openvas/openvas-d9d48c74
    - openvas/test-5879777846
    - openvas/test-5bd7f76f8c
    - openvas/test-6487578786
    - openvas/test-6b467857f5
  cert-manager.io/v1/Certificate:
    - openvas/openvas-ingress-cert-fp-pki
  cert-manager.io/v1/CertificateRequest:
    - openvas/openvas-ingress-cert-fp-pki-15
  discovery.k8s.io/v1/EndpointSlice:
    - openvas/openvas-service-6zghf
  networking.k8s.io/v1/Ingress:
    - openvas/openvas
  operators.coreos.com/v1alpha1/ClusterServiceVersion:
    - openvas/node-healthcheck-operator.v0.11.0
    - openvas/self-node-remediation.v0.12.1
  v1/ConfigMap:
    - openvas/elasticsearch-es-transport-ca-internal
    - openvas/init-scripts-config
    - openvas/kube-root-ca.crt
  v1/Endpoints:
    - openvas/openvas-service
  v1/Namespace:
    - openvas
  v1/PersistentVolume:
    - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0
    - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2
    - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4
  v1/PersistentVolumeClaim:
    - openvas/gvmd-pvc
    - openvas/openvas-pvc
    - openvas/psql-pvc
  v1/Pod:
    - openvas/openvas-78dc6f74c6-wcwp2
  v1/Secret:
    - openvas/gvmd-secrets
    - openvas/openvas-ingress-tls-fp-pki
  v1/Service:
    - openvas/openvas-service
  v1/ServiceAccount:
    - openvas/default

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    openvas/gvmd-pvc:
      Data Movement:
        Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 8896914780
    openvas/openvas-pvc:
      Data Movement:
        Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 5540
    openvas/psql-pvc:
      Data Movement:
        Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 24267413354

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

linstor-pvc-velero-logs.txt
xen04-07042026-panic.txt
linstor-controller-logs.txt