Happens at around the same time. Which point to something on a schedule, and I think it has been found. TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash. I have attempted to slow down velero with the following helm chart value change. Will see if it happens again. configuration: # Lowers API call velocity to stagger volume Create/Delete cycles clientQPS: 3 clientBurst: 5 # Forces serial execution across volume structures, preventing concurrent lifecycle operations itemBlockWorkerCount: 1 I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd? Timeline of crash On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host ovbh-pprod-xen04 suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module. It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time. Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Stage 1: Backup Initialization & Storage Provisioning Burst Timestamp: 23:15:17 - 23:15:26 PDT Velero starts the openvas-daily-20260705061500 snapshot which has PVC volumes (gvmd-pvc, openvas-pvc, psql-pvc). The linstor-csi-controller issues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings across xen01, xen02, and xen05. It then dynamically drops xen02 and redistributes the active target allocation mirror directly onto xen04 for some reason. 2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01' 2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02' 2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04' 2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com' Stage 2: Rapid Backup Job Completion & Teardown Execution Timestamp: 23:15:38 - 23:16:01 PDT The temporary target volume (openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops to Released. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediate DeleteResource directive to all replication satellites, commanding xen04 to disconnect the live disk configuration mapping. 2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""} 2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""} 2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""} 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk 2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 Stage 3: Memory Teardown, GPF, and Hard Kernel Panic Timestamp: 23:16:03 - 23:16:05 PDT As xen04 unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault. Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context. 2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI 2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G O 4.19.0+1 #1 2026-07-04T23:16:03-07:00 Hardware name: /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025 2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd] 2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48 2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016 2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0 2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0 2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI 2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G D O 4.19.0+1 #1 2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20 2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660 2026-07-04T23:16:05-07:00 Call Trace: 2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd] 2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd] Stage 4: Cascading Failures & Pool Master Dropout Timestamp: 23:16:05 - 23:21:50 PDT As xen04 drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The core etcd consensus cluster node on master01 encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At 23:21:50, the Pool Master node xen01 officially declares xen04 dead after exhausting its network PingAck heartbeat retry limits. 2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"} 2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time. Full velero details jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details Name: openvas-daily-20260705061500 Namespace: velero Labels: app.kubernetes.io/name=openvas velero.io/schedule-name=openvas-daily velero.io/storage-location=default Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=35 Phase: Completed Namespaces: Included: openvas Excluded: <none> Resources: Included cluster-scoped: <none> Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io Included namespace-scoped: * Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: true Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 30m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Expiration: 2026-08-03 23:15:00 -0700 PDT Total items to be backed up: 42 Items backed up: 42 Backup Item Operations: Operation for persistentvolumeclaims openvas/gvmd-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-v86h2 Phase: Completed Progress: 8896914780 of 8896914780 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:08 -0700 PDT Started: 2026-07-04 23:15:30 -0700 PDT Updated: 2026-07-04 23:17:11 -0700 PDT Operation for persistentvolumeclaims openvas/openvas-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-92dtr Phase: Completed Progress: 5540 of 5540 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:13 -0700 PDT Started: 2026-07-04 23:15:38 -0700 PDT Updated: 2026-07-04 23:16:54 -0700 PDT Operation for persistentvolumeclaims openvas/psql-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd Phase: Completed Progress: 24267413354 of 24267413354 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:18 -0700 PDT Started: 2026-07-04 23:15:42 -0700 PDT Updated: 2026-07-04 23:17:15 -0700 PDT Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - certificaterequests.cert-manager.io - certificates.cert-manager.io - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - openvas/openvas - openvas/test apps/v1/ReplicaSet: - openvas/openvas-64786df449 - openvas/openvas-69bb998d7c - openvas/openvas-6db7bcd6ff - openvas/openvas-6dd46f4f79 - openvas/openvas-6f8d45b667 - openvas/openvas-78dc6f74c6 - openvas/openvas-7c5654b9d6 - openvas/openvas-844c5f796d - openvas/openvas-86d9b75d9f - openvas/openvas-b97749f55 - openvas/openvas-d9d48c74 - openvas/test-5879777846 - openvas/test-5bd7f76f8c - openvas/test-6487578786 - openvas/test-6b467857f5 cert-manager.io/v1/Certificate: - openvas/openvas-ingress-cert-fp-pki cert-manager.io/v1/CertificateRequest: - openvas/openvas-ingress-cert-fp-pki-15 discovery.k8s.io/v1/EndpointSlice: - openvas/openvas-service-6zghf networking.k8s.io/v1/Ingress: - openvas/openvas operators.coreos.com/v1alpha1/ClusterServiceVersion: - openvas/node-healthcheck-operator.v0.11.0 - openvas/self-node-remediation.v0.12.1 v1/ConfigMap: - openvas/elasticsearch-es-transport-ca-internal - openvas/init-scripts-config - openvas/kube-root-ca.crt v1/Endpoints: - openvas/openvas-service v1/Namespace: - openvas v1/PersistentVolume: - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2 - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4 v1/PersistentVolumeClaim: - openvas/gvmd-pvc - openvas/openvas-pvc - openvas/psql-pvc v1/Pod: - openvas/openvas-78dc6f74c6-wcwp2 v1/Secret: - openvas/gvmd-secrets - openvas/openvas-ingress-tls-fp-pki v1/Service: - openvas/openvas-service v1/ServiceAccount: - openvas/default Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: openvas/gvmd-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 8896914780 openvas/openvas-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 5540 openvas/psql-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 24267413354 Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0 [image: 1783370374694-f9797a22-d4cf-40ae-8da5-6a7d580ef137-image-resized.jpeg] linstor-pvc-velero-logs.txt xen04-07042026-panic.txt linstor-controller-logs.txt

Thanks for coming back to close it out. That's useful to know. So I was plenty wrong, as it was the XOSTOR licensing backend rather than the repo side I guessed at; those -32000 errors really don't give much away. For anyone landing here later: sounds like -32000 on XOSTOR creation can come from either end, a licensing/entitlement issue that support sorts, or a host not reaching the package repo, so both are worth checking. @alcoralcor, did support get yours sorted too, or is yours still the repodata / repo-reachability one? Glad you're unblocked either way.

Yes, there's a weird issue when using both XOSTOR and HA, probably due to the HA mechanism itself. @Team-Storage is on it

@Mathieu-L linstor n l was included in my original post. All nodes were updated to May 2026 Security and Maintenance Updates for XCP-ng 8.3 LTS, all nodes were restarted. May 2026 Updates #2 for XCP-ng 8.3 LTS was released, and a couple days later I installed on all hosts. No host restarted. When xen04 was restarted, that is when this issue happened. I had used systemctl restart linstor-controller here (https://xcp-ng.org/forum/post/105309) to restart the controller.

@dthenot said: @ccooke Hello, You should be able to make the XOSTOR SR work again if you update sm and sm-fairlock on the other hosts. yum update sm sm-fairlock Then you should be able to re-plug the SR on the master and proceed with the RPU. Hello, Had the same problem, the command resolved the issue. It needs to be run on every host. Everything is working fine again. However, I had to complete the pool update manually.

Different use cases: Ceph is better with more hosts (at least 6 or 7 minimum) while XOSTOR is better between 3 to 7/8. We might have better Ceph support in the future for large clusters.

And to really round this out, the MTBF for any of these is in the millions of hours (1.2-3M), that's a use time of 136.968 - 342.46 years respectively. Basically, if a drive dies, just replace it no matter what, but in the end the reliability of these drives is meant to outlast all of us. Unless you actually need some specific function provided in some form-factor or model, don't bother.

@tmnguyen You can open a support ticket and request that we reactivate your XOSTOR trial licenses to match your existing XOA trial.

@ronan-a Thanks

@irtaza9 it scans all the VDI's on the SR to see if something has changed, if there's a need for coalesce and so on. I dont think it will be a big issue if you increase the auto-scan-interval value to lets say 5 minutes (300 seconds), but do remember, that everything regarding the VDI's on the SR will take up to 5 minutes to update, as well as triggering coalesce after removing snapshots.

That worked; thank you!!!

@TestForEcho No ETA for now. Even before supporting QCOW2 on LINSTOR, we have several points to robustify (HA performance, potential race conditions, etc.). Regarding other important points: Supporting volumes larger than 2TB has significant impacts on synchronization, RAM usage, coalesce, etc. We need to find a way to cope with these changes. The coalesce algorithm should be changed to no longer depend on the write speed to the SR in order to prevent potential coalesce interruptions; this is even more crucial for LINSTOR. The coalesce behavior is not exactly the same for QCOW2, and we believe that currently this could negatively impact the API impl in the case of LINSTOR. In short: QCOW2 has led to changes that require a long-term investment in several topics before even considering supporting this format on XOSTOR.

You can manually install XOSTOR yes.

@bdenv-r are you trying XOSTOR on 8.2 or 8.3 ? had many problem of performance and tap-drive locking my VDIs on 8.2 xostor would like to know if 8.3 xostor still have issues

@olivierlambert Thanks !

Hello, @DustinB The https://vates.tech/xostor/ says: The maximum size of any single Virtual Disk Image (VDI) will always be limited by the smallest disk in your cluster. But in this case, maybe it can be stored in the "2TB disks" ? Maybe others can answer, i didn't test it.

Hello, I tried on a new pool. a little different scenario since i don't create xostor for now, on my previous example, i tried to add a node as replacement of an existing one.. I just run the install script only on node 1. When i try make node2 join the pool, i reproduce the incompatible sm error i got previously. The things which is "bizarre", is i don't have the license issue i got on Xen-orchestra. ( maybe it was finally not related ? ) Here is the complete logs. pool.mergeInto { "sources": [ "17510fe0-db23-9414-f3df-2941bd34f8dc" ], "target": "cc91fcdc-c7a8-a44c-65b3-a76dced49252", "force": true } { "code": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE", "params": [ "OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd", "" ], "call": { "duration": 1061, "method": "pool.join_force", "params": [ "* session id *", "<MASTER_IP>", "root", "* obfuscated *" ] }, "message": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd, )", "name": "XapiError", "stack": "XapiError: POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd, ) at Function.wrap (file:///etc/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12) at file:///etc/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21 at runNextTicks (node:internal/process/task_queues:60:5) at processImmediate (node:internal/timers:454:9) at process.callbackTrampoline (node:internal/async_hooks:130:17)" }```

Hello @ronan-a I will reproduce the case, i will re-destroy one hypervisor and retrigger the case. Thank you @ronan-a et @olivierlambert If you need me to tests some special case don't hesit, we have a pool dedicated for this