@MajorP93 My mistake. No I have not rebooted the hosts yet. https://xcp-ng.org/forum/post/107164 I am aware that the reboot is needed for it to be picked up, I went ahead and installed from test so that if host crashes again it is there.
J
Offline
Posts
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
@Mathieu-L Yes, the hosts had been restarted after the updates were installed which included 9.2.16
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
For now I have upgraded all hosts to use
--enablerepo=xcp-ng-linstor-testingto getkmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3
That way it is there to be picked up if the host crashes again.
https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec
https://koji.xcp-ng.org/buildinfo?buildID=5747[12:15 ovbh-pprod-xen04 ~]# yum upgrade --enablerepo=xcp-ng-linstor-testing Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org grafana/signature | 810 B 00:00:00 grafana/signature | 3.0 kB 00:00:00 !!! xcp-ng-linstor-testing/signature | 473 B 00:00:00 xcp-ng-linstor-testing/signature | 3.0 kB 00:00:00 !!! zabbix | 3.0 kB 00:00:00 zabbix-non-supported | 2.9 kB 00:00:00 (1/3): grafana/primary_db | 879 kB 00:00:00 (2/3): xcp-ng-linstor-testing/primary_db | 2.4 kB 00:00:00 (3/3): zabbix/x86_64/primary_db | 114 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package kmod-drbd.x86_64 0:9.2.16-1.0.xcpng8.3 will be updated ---> Package kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 will be an update ---> Package zabbix-agent.x86_64 0:7.0.27-release1.el7 will be updated ---> Package zabbix-agent.x86_64 0:7.0.28-release1.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================== Updating: kmod-drbd x86_64 9.2.18-2.0.xcpng8.3 xcp-ng-linstor-testing 2.8 M zabbix-agent x86_64 7.0.28-release1.el7 zabbix 660 k Transaction Summary ============================================================================================================================================================================================================================================================== Upgrade 2 Packages Total download size: 3.4 M Is this ok [y/d/N]: y Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/2): zabbix-agent-7.0.28-release1.el7.x86_64.rpm | 660 kB 00:00:00 (2/2): kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64.rpm | 2.8 MB 00:00:01 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 2.8 MB/s | 3.4 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64 1/4 Updating : zabbix-agent-7.0.28-release1.el7.x86_64 2/4 Cleanup : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64 3/4 Cleanup : zabbix-agent-7.0.27-release1.el7.x86_64 4/4 Verifying : zabbix-agent-7.0.28-release1.el7.x86_64 1/4 Verifying : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64 2/4 Verifying : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64 3/4 Verifying : zabbix-agent-7.0.27-release1.el7.x86_64 4/4 Updated: kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 zabbix-agent.x86_64 0:7.0.28-release1.el7 Complete! -
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
@poddingue I'm fully up to date, but only at version
9.2.16[14:57 ovbh-pprod-xen02 ~]# drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36 DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION_CODE=0x090210 DRBD_KERNEL_VERSION=9.2.16 DRBDADM_VERSION_CODE=0x092100 DRBDADM_VERSION=9.33.0 [14:57 ovbh-pprod-xen02 ~]# drbd-reactor --version drbd-reactor 1.9.0 [14:57 ovbh-pprod-xen02 ~]#
There are no updates that I can install
[14:08 ovbh-pprod-xen02 ~]# yum check-update Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org zabbix | 3.0 kB 00:00:00 zabbix-non-supported | 2.9 kB 00:00:00 zabbix/x86_64/primary_db | 114 kB 00:00:00 zabbix-agent.x86_64 7.0.28-release1.el7 zabbix -
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
@MajorP93 No we do not have pro support at the moment
hmmm. Really seems like a build with
9.2.19could fix all of this.
How can I request for a build @poddingue ?https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L61
9.2.17- Fix a kernel crash triggered by a crafted/invalid netlink message
https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L24
9.2.19- Fix an AB-BA deadlock between online resize and activity-log transactions
- Fix several races during connection teardown that could crash or
hang (ack_sender requeue, pending ping work, in-progress lb-tcp
connect)
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
Logs from xen04 look like they are the same too. Which makes sense.
20:01:53.805: The Linstor DeviceManager on xen04 triggered an asynchronous alignment for a newly created volume (pvc-fac0a18b-2c23-4fe8-b005-b4f492cec92f), padding it from 26220040 KiB to 26222592 KiB. 20:01:54.034: Simultaneously, an orchestrator (like Velero) began tearing the volume down, shifting the DRBD state to conn( Connected -> TearDown ) and pdsk( Diskless -> DUnknown ). 20:01:58.484: The DRBD worker thread evaluated the shrinking bitmap while the alignment was trying to alter it. This caused the general protection fault: 0000 [#1] SMP NOPTI at instruction RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd], crashing the kernel. -
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
I was just moving around the velero backup schedules when it decided to run some again even though it was not time. Crashed xen04 this time. Deleting the possible offending backup schedule until this bug can be identified and fixed.
Uploaded new bug bundle to nextcloud too.
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
My first idea was to have velero use a different storage class. Something like
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: linstor-ephemeral-retain parameters: linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" linstor.csi.linbit.com/disklessOnRemaining: "true" linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool linstor.csi.linbit.com/placementCount: "3" provisioner: linstor.csi.linbit.com **reclaimPolicy: Retain** volumeBindingMode: WaitForFirstConsumerThen have a cronjob collected the
Releasedpvcs every hour. The problem is that can not work, as velero uses the storage class of the backed up pvc. So I would have to change every linstor storage class toRetain, and would effectively disable the automatic garbage collection for the cluster. Which might be fine, but would love to avoid it.My second idea is to eliminate the need for the resizing (another work around to hopefully get away from this race condition/bug)
I am also not seeing any way to modify the storageclass to round up or pad the pv. This would be to attempt to eliminate the need fordrbd_bm_resize.
Currently looking into Kyverno Mutating Webhook to dynamically pad these volumes on the fly to see if that would help me out at all.
All pvs are already whole numbers of Gi.
Velero calculates the size of the temporary clone pvc based on the exact byte-count of the used/allocated snapshot metadata, not the #Gi specification of the original pvc.
Just confirmed with a test
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
Looks to be similar to last time
Time,Node,Component / Service,Logged Event & Impact 23:17:13,xen01,LINSTOR/Controller,Resource Created: linstor-csi provisions a short-lived volume: pvc-395da199-d455-4004-ac0d-f6c6016ebf4e. 23:17:16,xen01,kernel (drbd),Sync Begins: xen01 connects to xen05 as a SyncSource and begins synchronizing the volume to it. 23:17:49,xen01,LINSTOR/Controller,Teardown Requested: The CSI plugin changes its mind or finishes its task, requesting sequential deletion (DelRsc) of the volume replicas across the cluster. 23:17:51,xen01,LINSTOR/Controller,Source Disk Removed: Linstor removes the volume from xen01 itself (Toggle Disk ... removing disk). 23:17:52,xen01,kernel (drbd),Source Goes Diskless: xen01 detaches the underlying storage block (UpToDate -> Detaching -> Diskless). It alerts peers it can no longer satisfy read requests, calling drbd_bm_resize with capacity == 0. 23:17:53,xen05,LINSTOR/Satellite, While the cluster connection is actively tearing down, xen05's local background DeviceManager wakes up and detects the volume. It attempts an online re-alignment to round up the block layer size to match the 4MiB extent rule (26220040 KiB to 26222592 KiB). 23:17:54,xen05,kernel (drbd),Fatal Crash: xen05's DRBD driver tries to resize and count its synchronization bitmap block pointers. Because its peer source (xen01) dropped to zero capacity mid-flight, DRBD hits memory corruption. It triggers a General Protection Fault at drbd_bm_count_bits+0x223/0x300 and panics the kernel. -
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
@poddingue This time it was host xen05. I got the request link for upload, so I have uploaded the bundle as well as all logs from xen05 and xen01 (xcp-ng pool master and had linstor-controller) So hoping we have everything needed. I will be digging into it to. It is in json format with all labels. Nothing in
/var/crash/ -
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
@poddingue Just had a new crash. Got the export
bug-report-20260719233946.tar.bz2it is ~63MB There is a way to share large files but it has been a bit since I have done that so have forgotten.
I've got all the logs. Will dig into it in the morning. Also looking into ways to share grafana links securely. We have it behind Cloudflare Zero Trust.
Here you can see when the host was offline.
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
I should already have all the logs, including kernel
Nothing has happened since. I am also increasing dom0 ram for all hosts.
-
RE: Not able to create new vdi with linstor at the moment
OK so that resource was a pain and did not want to give up, so I restarted the host.
Migration is running good now.
-
RE: Not able to create new vdi with linstor at the moment
hmmmmm
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource list -r pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ ResourceName │ Node │ Layers │ Usage │ Conns │ State │ Vote │ ╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ │ pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 │ ovbh-pprod-xen04 │ DRBD,STORAGE │ │ Ok │ DELETING │ Yes │ │ pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ │ Connecting(ovbh-pprod-xen01) │ DELETING │ Yes │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource-connection l pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 ╭─────────────────────────────────────────────────────────╮ │ Source │ Target │ Properties │ Port │ ╞═════════════════════════════════════════════════════════╡ │ ovbh-pprod-xen04 │ ovbh-pprod-xen05 │ │ │ ╰─────────────────────────────────────────────────────────╯Lets delete it, nope lol
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource delete ovbh-pprod-xen05 pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 SUCCESS: Description: Node: ovbh-pprod-xen05, Resource: pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 preparing for deletion. Details: Node: ovbh-pprod-xen05, Resource: pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 UUID is: 0485e43e-6dd9-4aa3-962a-99350f04be7e SUCCESS: Preparing deletion of resource on 'ovbh-pprod-xen04' ERROR: Description: (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed Cause: The external command for stopping the DRBD resource failed Correction: - Check whether the required software is installed - Check whether the application's search path includes the location of the external software - Check whether the application has execute permission for the external command Show reports: linstor error-reports show 6A4E98DD-5D3B7-000965 ERROR: Description: Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses Details: Node: ovbh-pprod-xen05, Resource: pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 Show reports: linstor error-reports show 6A4D452B-00000-001312reports
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 error-reports show 6A4E98DD-5D3B7-000965 | cat ERROR REPORT 6A4E98DD-5D3B7-000965 ============================================================ Application: LINBIT® LINSTOR Module: Satellite Version: 1.33.1 Build ID: 95da7940d6efb6a39ea303c5f37b03478a6fab0b Build time: 2025-12-22T16:04:57+00:00 Error time: 2026-07-09 15:42:39 Node: ovbh-pprod-xen05 Thread: DeviceManager ============================================================ Reported error: =============== Category: LinStorException Class name: StorageException Class canonical name: com.linbit.linstor.storage.StorageException Generated at: Method 'deleteDrbd', Source file 'DrbdLayer.java', Line #591 Error message: Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed Error context: An error occurred while processing resource 'Node: 'ovbh-pprod-xen05', Rsc: 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9'' ErrorContext: Description: Operations on resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' were aborted Cause: The external command for stopping the DRBD resource failed Correction: - Check whether the required software is installed - Check whether the application's search path includes the location of the external software - Check whether the application has execute permission for the external command Call backtrace: Method Native Class:Line number deleteDrbd N com.linbit.linstor.layer.drbd.DrbdLayer:591 processResource N com.linbit.linstor.layer.drbd.DrbdLayer:272 lambda$processResource$1 N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424 processGeneric N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467 processResource N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420 processResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394 dispatchResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226 dispatchResources N com.linbit.linstor.core.devmgr.DeviceManagerImpl:341 phaseDispatchDeviceHandlers N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149 devMgrLoop N com.linbit.linstor.core.devmgr.DeviceManagerImpl:786 run N com.linbit.linstor.core.devmgr.DeviceManagerImpl:682 run N java.lang.Thread:829 Caused by: ========== Category: LinStorException Class name: ExtCmdFailedException Class canonical name: com.linbit.extproc.ExtCmdFailedException Generated at: Method 'execute', Source file 'DrbdAdm.java', Line #810 Error message: The external command 'drbdsetup' did not complete within the timeout ErrorContext: Description: Execution of the external command 'drbdsetup' failed. Cause: The external command did not complete within the timeout. Possible causes include: - The system load may be too high to ensure completion of external commands in a timely manner. - The program implementing the external command may not be operating properly. - The operating system may have entered an erroneous state. Correction: Check whether the external program and the operating system are still operating properly. Check whether the system's load is within normal parameters. Details: The full command line executed was: drbdsetup down pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 Call backtrace: Method Native Class:Line number execute N com.linbit.linstor.layer.drbd.utils.DrbdAdm:810 simpleSetupCommand N com.linbit.linstor.layer.drbd.utils.DrbdAdm:735 down N com.linbit.linstor.layer.drbd.utils.DrbdAdm:213 down N com.linbit.linstor.layer.drbd.utils.DrbdAdm:178 deleteDrbd N com.linbit.linstor.layer.drbd.DrbdLayer:553 processResource N com.linbit.linstor.layer.drbd.DrbdLayer:272 lambda$processResource$1 N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424 processGeneric N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467 processResource N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420 processResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394 dispatchResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226 dispatchResources N com.linbit.linstor.core.devmgr.DeviceManagerImpl:341 phaseDispatchDeviceHandlers N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149 devMgrLoop N com.linbit.linstor.core.devmgr.DeviceManagerImpl:786 run N com.linbit.linstor.core.devmgr.DeviceManagerImpl:682 run N java.lang.Thread:829 Caused by: ========== Category: Exception Class name: ChildProcessTimeoutException Class canonical name: com.linbit.ChildProcessTimeoutException Generated at: Method 'waitFor', Source file 'ChildProcessHandler.java', Line #134 Call backtrace: Method Native Class:Line number waitFor N com.linbit.extproc.ChildProcessHandler:134 syncProcess N com.linbit.extproc.ExtCmd:162 pipeExec N com.linbit.extproc.ExtCmd:108 execute N com.linbit.linstor.layer.drbd.utils.DrbdAdm:802 simpleSetupCommand N com.linbit.linstor.layer.drbd.utils.DrbdAdm:735 down N com.linbit.linstor.layer.drbd.utils.DrbdAdm:213 down N com.linbit.linstor.layer.drbd.utils.DrbdAdm:178 deleteDrbd N com.linbit.linstor.layer.drbd.DrbdLayer:553 processResource N com.linbit.linstor.layer.drbd.DrbdLayer:272 lambda$processResource$1 N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424 processGeneric N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467 processResource N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420 processResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394 dispatchResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226 dispatchResources N com.linbit.linstor.core.devmgr.DeviceManagerImpl:341 phaseDispatchDeviceHandlers N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149 devMgrLoop N com.linbit.linstor.core.devmgr.DeviceManagerImpl:786 run N com.linbit.linstor.core.devmgr.DeviceManagerImpl:682 run N java.lang.Thread:829 END OF ERROR REPORT. jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 error-reports show 6A4D452B-00000-001312 | cat ERROR REPORT 6A4D452B-00000-001312 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.33.1 Build ID: 95da7940d6efb6a39ea303c5f37b03478a6fab0b Build time: 2025-12-22T16:04:57+00:00 Error time: 2026-07-09 18:42:39 Node: ovbh-pprod-xen01 Thread: MainWorkerPool-6 Access context information Identity: PUBLIC Role: PUBLIC Domain: PUBLIC Peer: RestClient(10.1.8.104; 'PythonLinstor/1.28.2 (API1.0.4): Client 1.28.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: DelayedApiRcException Class canonical name: com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils.DelayedApiRcException Generated at: Method 'lambda$mergeExtractingApiRcExceptions$6', Source file 'CtrlResponseUtils.java', Line #187 Error message: Exceptions have been converted to responses Error context: Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses Asynchronous stage backtrace: (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed Error has been observed at the following site(s): *__checkpoint ⇢ Prepare resource delete *__checkpoint ⇢ Activating resource if necessary before deletion Original Stack Trace: Call backtrace: Method Native Class:Line number lambda$mergeExtractingApiRcExceptions$6 N com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils:187 Suppressed exception 1 of 2: =============== Category: RuntimeException Class name: ApiRcException Class canonical name: com.linbit.linstor.core.apicallhandler.response.ApiRcException Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #353 Error message: (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed Error context: Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses ApiRcException entries: Message: (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed Cause: The external command for stopping the DRBD resource failed Correction: - Check whether the required software is installed - Check whether the application's search path includes the location of the external software - Check whether the application has execute permission for the external command NumericCode: -4611686018427386906 Call backtrace: Method Native Class:Line number handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:353 handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:306 doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:254 lambda$doProcessMessage$5 N com.linbit.linstor.proto.CommonMessageProcessor:239 subscribe N reactor.core.publisher.FluxDefer:46 subscribe N reactor.core.publisher.Flux:8848 onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430 drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453 drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724 onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256 drainFused N reactor.core.publisher.SinkManyUnicast:321 drain N reactor.core.publisher.SinkManyUnicast:363 tryEmitNext N reactor.core.publisher.SinkManyUnicast:239 tryEmitNext N reactor.core.publisher.SinkManySerialized:100 processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:446 doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:237 lambda$processMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:174 onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185 runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446 run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533 call N reactor.core.scheduler.WorkerTask:84 call N reactor.core.scheduler.WorkerTask:37 run N java.util.concurrent.FutureTask:264 run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304 runWorker N java.util.concurrent.ThreadPoolExecutor:1128 run N java.util.concurrent.ThreadPoolExecutor$Worker:628 run N java.lang.Thread:829 Suppressed exception 2 of 2: =============== Category: RuntimeException Class name: OnAssemblyException Class canonical name: reactor.core.publisher.FluxOnAssembly.OnAssemblyException Generated at: Method 'lambda$mergeExtractingApiRcExceptions$6', Source file 'CtrlResponseUtils.java', Line #187 Error message: Error has been observed at the following site(s): *__checkpoint ⇢ Prepare resource delete *__checkpoint ⇢ Activating resource if necessary before deletion Original Stack Trace: Error context: Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses Call backtrace: Method Native Class:Line number lambda$mergeExtractingApiRcExceptions$6 N com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils:187 subscribe N reactor.core.publisher.FluxDefer:46 subscribe N reactor.core.publisher.Flux:8848 onComplete N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:238 onComplete N reactor.core.publisher.FluxMap$MapSubscriber:144 checkTerminated N reactor.core.publisher.FluxFlatMap$FlatMapMain:850 drainLoop N reactor.core.publisher.FluxFlatMap$FlatMapMain:612 innerComplete N reactor.core.publisher.FluxFlatMap$FlatMapMain:898 onComplete N reactor.core.publisher.FluxFlatMap$FlatMapInner:1001 onComplete N reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2230 request N reactor.core.publisher.Operators$ScalarSubscription:2572 set N reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2366 onSubscribe N reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:74 subscribe N reactor.core.publisher.FluxJust:68 subscribe N reactor.core.publisher.Flux:8848 onError N reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:103 onError N reactor.core.publisher.FluxMap$MapSubscriber:134 onError N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:186 onError N reactor.core.publisher.FluxPeek$PeekSubscriber:222 onError N reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:106 error N reactor.core.publisher.Operators:198 subscribe N reactor.core.publisher.FluxError:43 subscribe N reactor.core.publisher.Flux:8848 onError N reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:103 onError N reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:142 onError N reactor.core.publisher.FluxUsing$UsingSubscriber:217 onError N reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2235 error N reactor.core.publisher.FluxCreate$BaseSink:479 drain N reactor.core.publisher.FluxCreate$BufferAsyncSink:868 error N reactor.core.publisher.FluxCreate$BufferAsyncSink:813 drainLoop N reactor.core.publisher.FluxCreate$SerializedFluxSink:239 drain N reactor.core.publisher.FluxCreate$SerializedFluxSink:215 error N reactor.core.publisher.FluxCreate$SerializedFluxSink:191 apiCallError N com.linbit.linstor.netcom.TcpConnectorPeer:529 handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:365 handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:306 doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:254 lambda$doProcessMessage$5 N com.linbit.linstor.proto.CommonMessageProcessor:239 subscribe N reactor.core.publisher.FluxDefer:46 subscribe N reactor.core.publisher.Flux:8848 onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430 drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453 drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724 onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256 drainFused N reactor.core.publisher.SinkManyUnicast:321 drain N reactor.core.publisher.SinkManyUnicast:363 tryEmitNext N reactor.core.publisher.SinkManyUnicast:239 tryEmitNext N reactor.core.publisher.SinkManySerialized:100 processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:446 doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:237 lambda$processMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:174 onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185 runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446 run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533 call N reactor.core.scheduler.WorkerTask:84 call N reactor.core.scheduler.WorkerTask:37 run N java.util.concurrent.FutureTask:264 run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304 runWorker N java.util.concurrent.ThreadPoolExecutor:1128 run N java.util.concurrent.ThreadPoolExecutor$Worker:628 run N java.lang.Thread:829 END OF ERROR REPORT. -
RE: Not able to create new vdi with linstor at the moment
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 error-reports show 6A4E98DD-5D3B7-000942 | cat ERROR REPORT 6A4E98DD-5D3B7-000942 ============================================================ Application: LINBIT® LINSTOR Module: Satellite Version: 1.33.1 Build ID: 95da7940d6efb6a39ea303c5f37b03478a6fab0b Build time: 2025-12-22T16:04:57+00:00 Error time: 2026-07-09 15:01:11 Node: ovbh-pprod-xen05 Thread: DeviceManager ============================================================ Reported error: =============== Category: LinStorException Class name: ResourceException Class canonical name: com.linbit.linstor.core.devmgr.exceptions.ResourceException Generated at: Method 'adjustDrbd', Source file 'DrbdLayer.java', Line #933 Error message: Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40 Error context: An error occurred while processing resource 'Node: 'ovbh-pprod-xen05', Rsc: 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40'' ErrorContext: Call backtrace: Method Native Class:Line number adjustDrbd N com.linbit.linstor.layer.drbd.DrbdLayer:933 processResource N com.linbit.linstor.layer.drbd.DrbdLayer:281 lambda$processResource$1 N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424 processGeneric N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467 processResource N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420 processResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394 dispatchResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226 dispatchResources N com.linbit.linstor.core.devmgr.DeviceManagerImpl:341 phaseDispatchDeviceHandlers N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149 devMgrLoop N com.linbit.linstor.core.devmgr.DeviceManagerImpl:786 run N com.linbit.linstor.core.devmgr.DeviceManagerImpl:682 run N java.lang.Thread:829 Caused by: ========== Category: LinStorException Class name: ExtCmdFailedException Class canonical name: com.linbit.extproc.ExtCmdFailedException Generated at: Method 'execute', Source file 'DrbdAdm.java', Line #805 Error message: The external command 'drbdadm' exited with error code 10 ErrorContext: Description: Execution of the external command 'drbdadm' failed. Cause: The external command exited with error code 10. Correction: - Check whether the external program is operating properly. - Check whether the command line is correct. Contact a system administrator or a developer if the command line is no longer valid for the installed version of the external program. Details: The full command line executed was: drbdadm -vvv adjust xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40 The external command sent the following output data: The external command sent the following error information: /var/lib/linstor.d/xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40.res:104: in resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40 ipv4:10.2.4.6:7033 is also used /var/lib/linstor.d/pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9.res:84 (resource pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9) Call backtrace: Method Native Class:Line number execute N com.linbit.linstor.layer.drbd.utils.DrbdAdm:805 adjust N com.linbit.linstor.layer.drbd.utils.DrbdAdm:134 adjustDrbd N com.linbit.linstor.layer.drbd.DrbdLayer:890 processResource N com.linbit.linstor.layer.drbd.DrbdLayer:281 lambda$processResource$1 N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424 processGeneric N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467 processResource N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420 processResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394 dispatchResources N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226 dispatchResources N com.linbit.linstor.core.devmgr.DeviceManagerImpl:341 phaseDispatchDeviceHandlers N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149 devMgrLoop N com.linbit.linstor.core.devmgr.DeviceManagerImpl:786 run N com.linbit.linstor.core.devmgr.DeviceManagerImpl:682 run N java.lang.Thread:829 END OF ERROR REPORT. -
RE: Not able to create new vdi with linstor at the moment
SM logs
"Date","Time","Line","tsNs","id","clusterLoggingSource","detected_level","hostname","job","priority","service_name","syslog_identifier","unit" 2026-07-09T22:01:14.842Z,1783634474842,"[2060240][MainThread] ***** Failed to snapshot!: EXCEPTION <class 'linstorvolumemanager.LinstorVolumeManagerError'>, Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01'",1783634474842687000,1783634474842687000_2194f1ea,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service 2026-07-09T22:01:15.206Z,1783634475206,"[2060240][MainThread] Raising exception [81, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']]",1783634475206274000,1783634475206274000_2137a4ec,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service 2026-07-09T22:01:15.215Z,1783634475215,"[2060240][MainThread] ***** generic exception: vdi_snapshot: EXCEPTION <class 'xs_errors.SROSError'>, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']",1783634475215733000,1783634475215733000_8881a7dc,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service 2026-07-09T22:01:15.228Z,1783634475228,"[2060240][MainThread] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'xs_errors.SROSError'>, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']",1783634475228432000,1783634475228432000_9ae0439a,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.serviceLinstor controller and satellite logs Logs-A-data-2026-07-09 15_11_07.csv.txt
-
Not able to create new vdi with linstor at the moment
Cluster is fully up to date and restarted all hosts.
vm migration from local storage to linstor is failing. It seems like it is trying to put it on all hosts, which is wrong. It should only be placement group three. For this migration the vm is on xen01 local storage, and I am attempt to move it to xen02 backed by linstor sr
Also happens when I attempt to create a new vm from terraform targeting linstor sr.
vm.migrate { "vm": "1a04aabe-1f93-94b3-ae24-d770b25d9c99", "migrationNetwork": "f1462aa4-d4f3-67d7-4d72-8784f56858e5", "sr": "479ca676-20a1-4051-7189-a4a9ca47e00d", "targetHost": "783c27ac-2dcb-4798-9ca8-27f5f30791f6" } { "code": "INTERNAL_ERROR", "params": [ "Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))])" ], "task": { "uuid": "9985ac33-a790-5600-c4a8-01ea18f31935", "name_label": "Async.VM.migrate_send", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20260709T22:00:43Z", "finished": "20260709T22:01:18Z", "status": "failure", "resident_on": "OpaqueRef:b3c659f5-efb3-4202-9268-b824e3eed523", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "INTERNAL_ERROR", "Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))])" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1815))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2654))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))" }, "message": "INTERNAL_ERROR(Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))]))", "name": "XapiError", "stack": "XapiError: INTERNAL_ERROR(Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))])) at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/_XapiError.mjs:16:12) at default (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/_getTaskResult.mjs:13:29) at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1078:24) at file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1112:14 at Array.forEach (<anonymous>) at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1102:12) at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1275:14)" }I have all the logs, let me know if there is something else I should post.
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource-group l ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ ResourceGroup │ SelectFilter │ VlmNrs │ Properties │ Description │ ╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ │ DfltRscGrp │ PlaceCount: 2 │ │ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ sc-51cd301c-e04b-5129-b9ac-d08d730d10b7 │ PlaceCount: 2 │ 0 │ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ sc-74e1434b-b435-587e-9dea-fa067deec898 │ PlaceCount: 3 │ 0 │ │ │ │ │ StoragePool(s): xcp-sr-linstor_group_thin_device │ │ │ │ │ │ DisklessOnRemaining: True │ │ │ │ │ │ LayerStack: ['DRBD', 'STORAGE'] │ │ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ sc-b066e430-6206-5588-a490-cc91ecef53d6 │ PlaceCount: 1 │ 0 │ │ │ │ │ StoragePool(s): xcp-sr-linstor_group_thin_device │ │ │ │ │ │ DisklessOnRemaining: True │ │ │ │ │ │ LayerStack: ['DRBD', 'STORAGE'] │ │ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ xcp-ha-linstor_group_thin_device │ PlaceCount: 3 │ 0 │ │ │ │ │ StoragePool(s): xcp-sr-linstor_group_thin_device │ │ │ │ │ │ DisklessOnRemaining: False │ │ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ xcp-sr-linstor_group_thin_device │ PlaceCount: 3 │ 0 │ │ │ │ │ StoragePool(s): xcp-sr-linstor_group_thin_device │ │ │ │ │ │ DisklessOnRemaining: False │ │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
Happens at around the same time. Which point to something on a schedule, and I think it has been found.
TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash.
I have attempted to slow down velero with the following helm chart value change. Will see if it happens again.configuration: # Lowers API call velocity to stagger volume Create/Delete cycles clientQPS: 3 clientBurst: 5 # Forces serial execution across volume structures, preventing concurrent lifecycle operations itemBlockWorkerCount: 1I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd?
Timeline of crash
On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host
ovbh-pprod-xen04suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module.
It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time.Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDTStage 1: Backup Initialization & Storage Provisioning Burst
Timestamp:
23:15:17 - 23:15:26 PDTVelero starts the
openvas-daily-20260705061500snapshot which has PVC volumes (gvmd-pvc,openvas-pvc,psql-pvc). Thelinstor-csi-controllerissues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings acrossxen01,xen02, andxen05. It then dynamically dropsxen02and redistributes the active target allocation mirror directly ontoxen04for some reason.2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01' 2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02' 2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04' 2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com'Stage 2: Rapid Backup Job Completion & Teardown Execution
Timestamp:
23:15:38 - 23:16:01 PDTThe temporary target volume (
openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops toReleased. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediateDeleteResourcedirective to all replication satellites, commandingxen04to disconnect the live disk configuration mapping.2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""} 2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""} 2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""} 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk 2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1Stage 3: Memory Teardown, GPF, and Hard Kernel Panic
Timestamp:
23:16:03 - 23:16:05 PDTAs
xen04unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault.Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside
rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context.2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI 2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G O 4.19.0+1 #1 2026-07-04T23:16:03-07:00 Hardware name: /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025 2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd] 2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48 2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016 2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0 2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0 2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI 2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G D O 4.19.0+1 #1 2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20 2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660 2026-07-04T23:16:05-07:00 Call Trace: 2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd] 2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd]Stage 4: Cascading Failures & Pool Master Dropout
Timestamp:
23:16:05 - 23:21:50 PDTAs
xen04drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The coreetcdconsensus cluster node onmaster01encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At23:21:50, the Pool Master nodexen01officially declaresxen04dead after exhausting its networkPingAckheartbeat retry limits.2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"} 2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time.Full velero details
jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details Name: openvas-daily-20260705061500 Namespace: velero Labels: app.kubernetes.io/name=openvas velero.io/schedule-name=openvas-daily velero.io/storage-location=default Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=35 Phase: Completed Namespaces: Included: openvas Excluded: <none> Resources: Included cluster-scoped: <none> Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io Included namespace-scoped: * Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: true Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 30m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Expiration: 2026-08-03 23:15:00 -0700 PDT Total items to be backed up: 42 Items backed up: 42 Backup Item Operations: Operation for persistentvolumeclaims openvas/gvmd-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-v86h2 Phase: Completed Progress: 8896914780 of 8896914780 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:08 -0700 PDT Started: 2026-07-04 23:15:30 -0700 PDT Updated: 2026-07-04 23:17:11 -0700 PDT Operation for persistentvolumeclaims openvas/openvas-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-92dtr Phase: Completed Progress: 5540 of 5540 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:13 -0700 PDT Started: 2026-07-04 23:15:38 -0700 PDT Updated: 2026-07-04 23:16:54 -0700 PDT Operation for persistentvolumeclaims openvas/psql-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd Phase: Completed Progress: 24267413354 of 24267413354 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:18 -0700 PDT Started: 2026-07-04 23:15:42 -0700 PDT Updated: 2026-07-04 23:17:15 -0700 PDT Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - certificaterequests.cert-manager.io - certificates.cert-manager.io - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - openvas/openvas - openvas/test apps/v1/ReplicaSet: - openvas/openvas-64786df449 - openvas/openvas-69bb998d7c - openvas/openvas-6db7bcd6ff - openvas/openvas-6dd46f4f79 - openvas/openvas-6f8d45b667 - openvas/openvas-78dc6f74c6 - openvas/openvas-7c5654b9d6 - openvas/openvas-844c5f796d - openvas/openvas-86d9b75d9f - openvas/openvas-b97749f55 - openvas/openvas-d9d48c74 - openvas/test-5879777846 - openvas/test-5bd7f76f8c - openvas/test-6487578786 - openvas/test-6b467857f5 cert-manager.io/v1/Certificate: - openvas/openvas-ingress-cert-fp-pki cert-manager.io/v1/CertificateRequest: - openvas/openvas-ingress-cert-fp-pki-15 discovery.k8s.io/v1/EndpointSlice: - openvas/openvas-service-6zghf networking.k8s.io/v1/Ingress: - openvas/openvas operators.coreos.com/v1alpha1/ClusterServiceVersion: - openvas/node-healthcheck-operator.v0.11.0 - openvas/self-node-remediation.v0.12.1 v1/ConfigMap: - openvas/elasticsearch-es-transport-ca-internal - openvas/init-scripts-config - openvas/kube-root-ca.crt v1/Endpoints: - openvas/openvas-service v1/Namespace: - openvas v1/PersistentVolume: - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2 - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4 v1/PersistentVolumeClaim: - openvas/gvmd-pvc - openvas/openvas-pvc - openvas/psql-pvc v1/Pod: - openvas/openvas-78dc6f74c6-wcwp2 v1/Secret: - openvas/gvmd-secrets - openvas/openvas-ingress-tls-fp-pki v1/Service: - openvas/openvas-service v1/ServiceAccount: - openvas/default Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: openvas/gvmd-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 8896914780 openvas/openvas-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 5540 openvas/psql-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 24267413354 Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0
linstor-pvc-velero-logs.txt
xen04-07042026-panic.txt
linstor-controller-logs.txt -
RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?
I've installed the latest updates to all hosts, and restarted everything. I also have all xcp-ng logs going into loki now, so next time something happens I will see everything lol.
