Also seeing the "Body timeout error" only on the metadata and XO-config backups for the past 2 weeks. All the VMs are backing up just fine. The Remote is NFS on a share called Backups: NFS Backups \192.168.191.8:Port:/Backups Click to edit 2 TiB / 4.84 TiB 105.4 MiB/s / 1.8 GiB/s None I recently needed to restore from the metadata and xo-config backs and was able to find an old one, but can't make any new ones, so it is a source of concern.

@PessimistTech That looks a bit odd indeed. In addition to what proposed @yannsionneau, can you also give the output of : xl dmesg (in Dom0) dmesg (in Dom0) dmesg (in the guest) So that we can try to pin-point what may be happening.

@JamesG said: So I ordered some mini DP dummy plugs and took a look at this further today. First thought...Just load Windows on the bare metal and see what the card does. No problem. Loaded Win10, loaded the latest drivers, PCIe Gen4x8 lanes. Sweet. At least that works. Put the XCP-ng drive back in, booted up, passed-through the GPU to a Win11 guest, updated the drivers, PCIe Gen1x1. As I explained briefly in https://xcp-ng.org/forum/post/106479, this is mostly display as the real hardware may be running at a different speed. Ultimately I need to install a Linux distro with a really current kernel on the bare metal and see how it sees the GPU. That might be a good project for tomorrow. I think this just comes down to a combination of too new, not well supported hardware with some quirks and somewhat dated code and support in XCP-ng (ReBAR support in guests for example). I did quick checks and I believe that ReBAR works; at least with UEFI guests (our OVMF build does support resizable bar), and it actually works as suggest https://xcp-ng.org/forum/post/106489 (VRAM BAR is 16 GB); but I didn't do extensive testing in that area.

@pedro_udifar Hi, We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream This should fix the stream issue and fix the warning.

@poddingue Yes I am interested in any and all feedback. As of now it uses the JSON-RPC API and auth token. It will pull the current backup job / settings and create them in the new XO. If there is decent interested in this I will work on adding in the other features as well. As my home lab I am not using all the backup features so can only test basic backup job creating and such. IF other can test the Mirror jobs, DR, or Continuous Replication.

@bvivi57 Hey, merci pour ce tuto ! Je vais comparer ce que tu as fait avec mes template que j'ai créer de mon coté. Si tu veux comparer, le repo est ici https://github.com/disruptivemindseu/xcpng-template-builder/tree/main

Happens at around the same time. Which point to something on a schedule, and I think it has been found. TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash. I have attempted to slow down velero with the following helm chart value change. Will see if it happens again. configuration: # Lowers API call velocity to stagger volume Create/Delete cycles clientQPS: 3 clientBurst: 5 # Forces serial execution across volume structures, preventing concurrent lifecycle operations itemBlockWorkerCount: 1 I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd? Timeline of crash On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host ovbh-pprod-xen04 suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module. It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time. Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Stage 1: Backup Initialization & Storage Provisioning Burst Timestamp: 23:15:17 - 23:15:26 PDT Velero starts the openvas-daily-20260705061500 snapshot which has PVC volumes (gvmd-pvc, openvas-pvc, psql-pvc). The linstor-csi-controller issues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings across xen01, xen02, and xen05. It then dynamically drops xen02 and redistributes the active target allocation mirror directly onto xen04 for some reason. 2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01' 2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02' 2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04' 2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com' Stage 2: Rapid Backup Job Completion & Teardown Execution Timestamp: 23:15:38 - 23:16:01 PDT The temporary target volume (openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops to Released. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediate DeleteResource directive to all replication satellites, commanding xen04 to disconnect the live disk configuration mapping. 2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""} 2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""} 2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""} 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk 2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 Stage 3: Memory Teardown, GPF, and Hard Kernel Panic Timestamp: 23:16:03 - 23:16:05 PDT As xen04 unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault. Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context. 2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI 2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G O 4.19.0+1 #1 2026-07-04T23:16:03-07:00 Hardware name: /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025 2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd] 2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48 2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016 2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0 2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0 2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI 2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G D O 4.19.0+1 #1 2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20 2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660 2026-07-04T23:16:05-07:00 Call Trace: 2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd] 2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd] Stage 4: Cascading Failures & Pool Master Dropout Timestamp: 23:16:05 - 23:21:50 PDT As xen04 drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The core etcd consensus cluster node on master01 encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At 23:21:50, the Pool Master node xen01 officially declares xen04 dead after exhausting its network PingAck heartbeat retry limits. 2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"} 2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time. Full velero details jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details Name: openvas-daily-20260705061500 Namespace: velero Labels: app.kubernetes.io/name=openvas velero.io/schedule-name=openvas-daily velero.io/storage-location=default Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=35 Phase: Completed Namespaces: Included: openvas Excluded: <none> Resources: Included cluster-scoped: <none> Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io Included namespace-scoped: * Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: true Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 30m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Expiration: 2026-08-03 23:15:00 -0700 PDT Total items to be backed up: 42 Items backed up: 42 Backup Item Operations: Operation for persistentvolumeclaims openvas/gvmd-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-v86h2 Phase: Completed Progress: 8896914780 of 8896914780 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:08 -0700 PDT Started: 2026-07-04 23:15:30 -0700 PDT Updated: 2026-07-04 23:17:11 -0700 PDT Operation for persistentvolumeclaims openvas/openvas-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-92dtr Phase: Completed Progress: 5540 of 5540 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:13 -0700 PDT Started: 2026-07-04 23:15:38 -0700 PDT Updated: 2026-07-04 23:16:54 -0700 PDT Operation for persistentvolumeclaims openvas/psql-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd Phase: Completed Progress: 24267413354 of 24267413354 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:18 -0700 PDT Started: 2026-07-04 23:15:42 -0700 PDT Updated: 2026-07-04 23:17:15 -0700 PDT Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - certificaterequests.cert-manager.io - certificates.cert-manager.io - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - openvas/openvas - openvas/test apps/v1/ReplicaSet: - openvas/openvas-64786df449 - openvas/openvas-69bb998d7c - openvas/openvas-6db7bcd6ff - openvas/openvas-6dd46f4f79 - openvas/openvas-6f8d45b667 - openvas/openvas-78dc6f74c6 - openvas/openvas-7c5654b9d6 - openvas/openvas-844c5f796d - openvas/openvas-86d9b75d9f - openvas/openvas-b97749f55 - openvas/openvas-d9d48c74 - openvas/test-5879777846 - openvas/test-5bd7f76f8c - openvas/test-6487578786 - openvas/test-6b467857f5 cert-manager.io/v1/Certificate: - openvas/openvas-ingress-cert-fp-pki cert-manager.io/v1/CertificateRequest: - openvas/openvas-ingress-cert-fp-pki-15 discovery.k8s.io/v1/EndpointSlice: - openvas/openvas-service-6zghf networking.k8s.io/v1/Ingress: - openvas/openvas operators.coreos.com/v1alpha1/ClusterServiceVersion: - openvas/node-healthcheck-operator.v0.11.0 - openvas/self-node-remediation.v0.12.1 v1/ConfigMap: - openvas/elasticsearch-es-transport-ca-internal - openvas/init-scripts-config - openvas/kube-root-ca.crt v1/Endpoints: - openvas/openvas-service v1/Namespace: - openvas v1/PersistentVolume: - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2 - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4 v1/PersistentVolumeClaim: - openvas/gvmd-pvc - openvas/openvas-pvc - openvas/psql-pvc v1/Pod: - openvas/openvas-78dc6f74c6-wcwp2 v1/Secret: - openvas/gvmd-secrets - openvas/openvas-ingress-tls-fp-pki v1/Service: - openvas/openvas-service v1/ServiceAccount: - openvas/default Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: openvas/gvmd-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 8896914780 openvas/openvas-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 5540 openvas/psql-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 24267413354 Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0 linstor-pvc-velero-logs.txt xen04-07042026-panic.txt linstor-controller-logs.txt

Here's the short version: this looks like the known "VDIs show up as snapshots" bug, and you haven't done anything wrong. What happens is a metadata flag on the VDI, is-a-snapshot, gets flipped to true after some of the recent 8.3 host updates, so XO (both 5 and 6) thinks those disks are snapshots and hides them from the per-VM Disks tab, even though the VDIs are fine and xe / XCP-ng Center still show them. A few older threads track the same thing, the main one is https://xcp-ng.org/forum/topic/11715/vdi-not-showing-in-xo-5-from-source . From what's said there, the fix is expected in two parts: stopping it from happening again, and a way to repair the flag on VDIs that are already affected. I don't have a timeline and could be off on the details, so please don't take that as official. In the meantime people have made the disks reappear by migrating the affected VDI to another SR, or snapshot then revert then delete the snapshot, but it's a visual fix and can come back. The VDI_IN_USE you're now hitting on start does sound worth its own detailed report, so yes please file it with the xapi/sm versions and that malformed is-a-snapshot field, and it'd be good to loop in @Team-Storage. Sorry you're stuck on this.

Just to give an update. I still am having the issue with current commit 090ce. Regards, Marc