xcp-ng server crashed/rebooted due to issues with drbd/linstor?

Jonathon

Here is the log from when xen04 last crashed xen04-jun12-journalctl-crash-summary.txt

Jonathon

I've installed the latest updates to all hosts, and restarted everything. I also have all xcp-ng logs going into loki now, so next time something happens I will see everything lol.

Jonathon

Happens at around the same time. Which point to something on a schedule, and I think it has been found.

TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash.
I have attempted to slow down velero with the following helm chart value change. Will see if it happens again.

configuration:
  # Lowers API call velocity to stagger volume Create/Delete cycles
  clientQPS: 3
  clientBurst: 5
  
  # Forces serial execution across volume structures, preventing concurrent lifecycle operations
  itemBlockWorkerCount: 1

I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd?

Timeline of crash

On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host ovbh-pprod-xen04 suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module.
It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time.

Started:    2026-07-04 23:15:00 -0700 PDT
Completed:  2026-07-04 23:17:33 -0700 PDT

Stage 1: Backup Initialization & Storage Provisioning Burst

Timestamp: 23:15:17 - 23:15:26 PDT

Velero starts the openvas-daily-20260705061500 snapshot which has PVC volumes (gvmd-pvc, openvas-pvc, psql-pvc). The linstor-csi-controller issues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings across xen01, xen02, and xen05. It then dynamically drops xen02 and redistributes the active target allocation mirror directly onto xen04 for some reason.

2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982      69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""}
2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO  LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO  LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01'
2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO  LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02'
2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO  LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04'
2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO  LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO  LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com'

Stage 2: Rapid Backup Job Completion & Teardown Execution

Timestamp: 23:15:38 - 23:16:01 PDT

The temporary target volume (openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops to Released. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediate DeleteResource directive to all replication satellites, commanding xen04 to disconnect the live disk configuration mapping.

2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""}
2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""}
2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""}
2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""}
2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO  LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1
2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO  LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk
2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO  LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1

Stage 3: Memory Teardown, GPF, and Hard Kernel Panic

Timestamp: 23:16:03 - 23:16:05 PDT

As xen04 unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault.

Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context.

2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI
2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G           O      4.19.0+1 #1
2026-07-04T23:16:03-07:00 Hardware name:  /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025
2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd]
2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48
2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016
2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0
2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0
2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI
2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G      D    O      4.19.0+1 #1
2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20
2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660
2026-07-04T23:16:05-07:00 Call Trace:
2026-07-04T23:16:05-07:00  drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd]
2026-07-04T23:16:05-07:00  ? conn_disconnect+0x19c/0xce0 [drbd]
2026-07-04T23:16:05-07:00  ? drbd_receiver+0x361/0x9a0 [drbd]
2026-07-04T23:16:05-07:00  ? drbd_thread_setup+0xd0/0x230 [drbd]

Stage 4: Cascading Failures & Pool Master Dropout

Timestamp: 23:16:05 - 23:21:50 PDT

As xen04 drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The core etcd consensus cluster node on master01 encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At 23:21:50, the Pool Master node xen01 officially declares xen04 dead after exhausting its network PingAck heartbeat retry limits.

2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"}
2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time.

Full velero details

jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details
Name:         openvas-daily-20260705061500
Namespace:    velero
Labels:       app.kubernetes.io/name=openvas
              velero.io/schedule-name=openvas-daily
              velero.io/storage-location=default
Annotations:  objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA
              objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f
              velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=35

Phase:  Completed


Namespaces:
  Included:  openvas
  Excluded:  <none>

Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    30m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2026-07-04 23:15:00 -0700 PDT
Completed:  2026-07-04 23:17:33 -0700 PDT

Expiration:  2026-08-03 23:15:00 -0700 PDT

Total items to be backed up:  42
Items backed up:              42

Backup Item Operations:
  Operation for persistentvolumeclaims openvas/gvmd-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5
    Items to Update:
                           datauploads.velero.io velero/openvas-daily-20260705061500-v86h2
    Phase:                 Completed
    Progress:              8896914780 of 8896914780 complete (Bytes)
    Progress description:  Completed
    Created:               2026-07-04 23:15:08 -0700 PDT
    Started:               2026-07-04 23:15:30 -0700 PDT
    Updated:               2026-07-04 23:17:11 -0700 PDT
  Operation for persistentvolumeclaims openvas/openvas-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803
    Items to Update:
                           datauploads.velero.io velero/openvas-daily-20260705061500-92dtr
    Phase:                 Completed
    Progress:              5540 of 5540 complete (Bytes)
    Progress description:  Completed
    Created:               2026-07-04 23:15:13 -0700 PDT
    Started:               2026-07-04 23:15:38 -0700 PDT
    Updated:               2026-07-04 23:16:54 -0700 PDT
  Operation for persistentvolumeclaims openvas/psql-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8
    Items to Update:
                           datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd
    Phase:                 Completed
    Progress:              24267413354 of 24267413354 complete (Bytes)
    Progress description:  Completed
    Created:               2026-07-04 23:15:18 -0700 PDT
    Started:               2026-07-04 23:15:42 -0700 PDT
    Updated:               2026-07-04 23:17:15 -0700 PDT
Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - certificaterequests.cert-manager.io
    - certificates.cert-manager.io
    - clusterserviceversions.operators.coreos.com
  apps/v1/Deployment:
    - openvas/openvas
    - openvas/test
  apps/v1/ReplicaSet:
    - openvas/openvas-64786df449
    - openvas/openvas-69bb998d7c
    - openvas/openvas-6db7bcd6ff
    - openvas/openvas-6dd46f4f79
    - openvas/openvas-6f8d45b667
    - openvas/openvas-78dc6f74c6
    - openvas/openvas-7c5654b9d6
    - openvas/openvas-844c5f796d
    - openvas/openvas-86d9b75d9f
    - openvas/openvas-b97749f55
    - openvas/openvas-d9d48c74
    - openvas/test-5879777846
    - openvas/test-5bd7f76f8c
    - openvas/test-6487578786
    - openvas/test-6b467857f5
  cert-manager.io/v1/Certificate:
    - openvas/openvas-ingress-cert-fp-pki
  cert-manager.io/v1/CertificateRequest:
    - openvas/openvas-ingress-cert-fp-pki-15
  discovery.k8s.io/v1/EndpointSlice:
    - openvas/openvas-service-6zghf
  networking.k8s.io/v1/Ingress:
    - openvas/openvas
  operators.coreos.com/v1alpha1/ClusterServiceVersion:
    - openvas/node-healthcheck-operator.v0.11.0
    - openvas/self-node-remediation.v0.12.1
  v1/ConfigMap:
    - openvas/elasticsearch-es-transport-ca-internal
    - openvas/init-scripts-config
    - openvas/kube-root-ca.crt
  v1/Endpoints:
    - openvas/openvas-service
  v1/Namespace:
    - openvas
  v1/PersistentVolume:
    - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0
    - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2
    - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4
  v1/PersistentVolumeClaim:
    - openvas/gvmd-pvc
    - openvas/openvas-pvc
    - openvas/psql-pvc
  v1/Pod:
    - openvas/openvas-78dc6f74c6-wcwp2
  v1/Secret:
    - openvas/gvmd-secrets
    - openvas/openvas-ingress-tls-fp-pki
  v1/Service:
    - openvas/openvas-service
  v1/ServiceAccount:
    - openvas/default

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    openvas/gvmd-pvc:
      Data Movement:
        Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 8896914780
    openvas/openvas-pvc:
      Data Movement:
        Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 5540
    openvas/psql-pvc:
      Data Movement:
        Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 24267413354

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

linstor-pvc-velero-logs.txt
xen04-07042026-panic.txt
linstor-controller-logs.txt

poddingue

The timing evidence here looks strong to me. Your second Grafana window is doing a lot of work: linstor-satellite.service jumps to roughly 8,700 log lines in the three minutes around the panic, well above linstor-controller at 926 and xapi at 460, and the priority chart goes red at the same moment. A backup that fans out volume creates and deletes across three replicas, landing on a DRBD race, fits what you are seeing.

Throttling Velero should tell you a lot. If the crashes stop with clientQPS: 3 and itemBlockWorkerCount: 1, that narrows it to concurrency rather than anything about those particular volumes.

For the logs question, a xen-bugtool --yestoall bundle from a host that has panicked is usually the thing people ask for first (https://docs.xcp-ng.org/troubleshooting/log-files), since it sweeps up the kernel side alongside the storage logs you already have in Loki. I don't know DRBD internals well enough to say which trace matters most, so it might be worth a mention to @Team-Storage. They can say what they actually need rather than have you guess, and a panic that reproduces on a schedule is a good deal easier for them to chase than most.

Jonathon

@poddingue

I should already have all the logs, including kernel

Nothing has happened since. I am also increasing dom0 ram for all hosts.

poddingue

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes.
Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 926 log lines around the panic suggests dom0 was under real pressure right then.
If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side.
I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful.
Fingers crossed it stays boring from here.

Jonathon

@poddingue Just had a new crash. Got the export bug-report-20260719233946.tar.bz2 it is ~63MB There is a way to share large files but it has been a bit since I have done that so have forgotten.
I've got all the logs. Will dig into it in the morning. Also looking into ways to share grafana links securely. We have it behind Cloudflare Zero Trust.

Here you can see when the host was offline.

poddingue

On the bundle, the docs warn it can carry sensitive details about your setup, so better not to attach it publicly: https://docs.xcp-ng.org/troubleshooting/log-files#produce-a-status-report. As far as I can tell there is no self-serve upload spot, it is usually someone from Vates dropping you a private Nextcloud link when you ask, so flagging here that you have the bundle ready is probably the quickest route.

Since this was a kernel panic, is there anything in /var/crash on the host that went down, and which host was it this time? The earlier ones were xen04 and xen03.

Jonathon

@poddingue This time it was host xen05. I got the request link for upload, so I have uploaded the bundle as well as all logs from xen05 and xen01 (xcp-ng pool master and had linstor-controller) So hoping we have everything needed. I will be digging into it to. It is in json format with all labels. Nothing in /var/crash/

Jonathon

Looks to be similar to last time

Time,Node,Component / Service,Logged Event & Impact
23:17:13,xen01,LINSTOR/Controller,Resource Created: linstor-csi provisions a short-lived volume: pvc-395da199-d455-4004-ac0d-f6c6016ebf4e.
23:17:16,xen01,kernel (drbd),Sync Begins: xen01 connects to xen05 as a SyncSource and begins synchronizing the volume to it.
23:17:49,xen01,LINSTOR/Controller,Teardown Requested: The CSI plugin changes its mind or finishes its task, requesting sequential deletion (DelRsc) of the volume replicas across the cluster.
23:17:51,xen01,LINSTOR/Controller,Source Disk Removed: Linstor removes the volume from xen01 itself (Toggle Disk ... removing disk).
23:17:52,xen01,kernel (drbd),Source Goes Diskless: xen01 detaches the underlying storage block (UpToDate -> Detaching -> Diskless). It alerts peers it can no longer satisfy read requests, calling drbd_bm_resize with capacity == 0.
23:17:53,xen05,LINSTOR/Satellite, While the cluster connection is actively tearing down, xen05's local background DeviceManager wakes up and detects the volume. It attempts an online re-alignment to round up the block layer size to match the 4MiB extent rule (26220040 KiB to 26222592 KiB).
23:17:54,xen05,kernel (drbd),Fatal Crash: xen05's DRBD driver tries to resize and count its synchronization bitmap block pointers. Because its peer source (xen01) dropped to zero capacity mid-flight, DRBD hits memory corruption. It triggers a General Protection Fault at drbd_bm_count_bits+0x223/0x300 and panics the kernel.

Jonathon

My first idea was to have velero use a different storage class. Something like

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-ephemeral-retain
parameters:
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/disklessOnRemaining: "true"
  linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool
  linstor.csi.linbit.com/placementCount: "3"
provisioner: linstor.csi.linbit.com
**reclaimPolicy: Retain**
volumeBindingMode: WaitForFirstConsumer

Then have a cronjob collected the Released pvcs every hour. The problem is that can not work, as velero uses the storage class of the backed up pvc. So I would have to change every linstor storage class to Retain, and would effectively disable the automatic garbage collection for the cluster. Which might be fine, but would love to avoid it.

My second idea is to eliminate the need for the resizing (another work around to hopefully get away from this race condition/bug)
I am also not seeing any way to modify the storageclass to round up or pad the pv. This would be to attempt to eliminate the need for drbd_bm_resize.
Currently looking into Kyverno Mutating Webhook to dynamically pad these volumes on the fly to see if that would help me out at all.
All pvs are already whole numbers of Gi.
Velero calculates the size of the temporary clone pvc based on the exact byte-count of the used/allocated snapshot metadata, not the #Gi specification of the original pvc.
Just confirmed with a test

Jonathon

I was just moving around the velero backup schedules when it decided to run some again even though it was not time. Crashed xen04 this time. Deleting the possible offending backup schedule until this bug can be identified and fixed.

Uploaded new bug bundle to nextcloud too.

Jonathon

Logs from xen04 look like they are the same too. Which makes sense.

20:01:53.805: The Linstor DeviceManager on xen04 triggered an asynchronous alignment for a newly created volume (pvc-fac0a18b-2c23-4fe8-b005-b4f492cec92f), padding it from 26220040 KiB to 26222592 KiB.

20:01:54.034: Simultaneously, an orchestrator (like Velero) began tearing the volume down, shifting the DRBD state to conn( Connected -> TearDown ) and pdsk( Diskless -> DUnknown ).

20:01:58.484: The DRBD worker thread evaluated the shrinking bitmap while the alignment was trying to alter it. This caused the general protection fault: 0000 [#1] SMP NOPTI at instruction RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd], crashing the kernel.