XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Not able to create new vdi with linstor at the moment

    Scheduled Pinned Locked Moved XOSTOR
    5 Posts 1 Posters 8 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Online
      Jonathon
      last edited by

      Cluster is fully up to date and restarted all hosts.

      vm migration from local storage to linstor is failing. It seems like it is trying to put it on all hosts, which is wrong. It should only be placement group three. For this migration the vm is on xen01 local storage, and I am attempt to move it to xen02 backed by linstor sr

      Also happens when I attempt to create a new vm from terraform targeting linstor sr.

      vm.migrate
      {
        "vm": "1a04aabe-1f93-94b3-ae24-d770b25d9c99",
        "migrationNetwork": "f1462aa4-d4f3-67d7-4d72-8784f56858e5",
        "sr": "479ca676-20a1-4051-7189-a4a9ca47e00d",
        "targetHost": "783c27ac-2dcb-4798-9ca8-27f5f30791f6"
      }
      {
        "code": "INTERNAL_ERROR",
        "params": [
          "Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))])"
        ],
        "task": {
          "uuid": "9985ac33-a790-5600-c4a8-01ea18f31935",
          "name_label": "Async.VM.migrate_send",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20260709T22:00:43Z",
          "finished": "20260709T22:01:18Z",
          "status": "failure",
          "resident_on": "OpaqueRef:b3c659f5-efb3-4202-9268-b824e3eed523",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "INTERNAL_ERROR",
            "Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))])"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1815))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2654))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
        },
        "message": "INTERNAL_ERROR(Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))]))",
        "name": "XapiError",
        "stack": "XapiError: INTERNAL_ERROR(Storage_error ([S(Internal_error);S(Storage_error ([S(Migration_preparation_failure);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_81);[S();S(Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']);S()]]]))]))]))
          at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/_XapiError.mjs:16:12)
          at default (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/_getTaskResult.mjs:13:29)
          at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1078:24)
          at file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1112:14
          at Array.forEach (<anonymous>)
          at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1102:12)
          at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202606162125/packages/xen-api/index.mjs:1275:14)"
      }
      

      I have all the logs, let me know if there is something else I should post.
      cd80a27d-225b-44b8-a1ab-1ff8a226ee22-image.jpeg

      jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource-group  l
      ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
      │ ResourceGroup                           │ SelectFilter                                     │ VlmNrs │ Properties │ Description │
      ╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
      │ DfltRscGrp                              │ PlaceCount: 2                                    │        │            │             │
      ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
      │ sc-51cd301c-e04b-5129-b9ac-d08d730d10b7 │ PlaceCount: 2                                    │ 0      │            │             │
      ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
      │ sc-74e1434b-b435-587e-9dea-fa067deec898 │ PlaceCount: 3                                    │ 0      │            │             │
      │                                         │ StoragePool(s): xcp-sr-linstor_group_thin_device │        │            │             │
      │                                         │ DisklessOnRemaining: True                        │        │            │             │
      │                                         │ LayerStack: ['DRBD', 'STORAGE']                  │        │            │             │
      ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
      │ sc-b066e430-6206-5588-a490-cc91ecef53d6 │ PlaceCount: 1                                    │ 0      │            │             │
      │                                         │ StoragePool(s): xcp-sr-linstor_group_thin_device │        │            │             │
      │                                         │ DisklessOnRemaining: True                        │        │            │             │
      │                                         │ LayerStack: ['DRBD', 'STORAGE']                  │        │            │             │
      ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
      │ xcp-ha-linstor_group_thin_device        │ PlaceCount: 3                                    │ 0      │            │             │
      │                                         │ StoragePool(s): xcp-sr-linstor_group_thin_device │        │            │             │
      │                                         │ DisklessOnRemaining: False                       │        │            │             │
      ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
      │ xcp-sr-linstor_group_thin_device        │ PlaceCount: 3                                    │ 0      │            │             │
      │                                         │ StoragePool(s): xcp-sr-linstor_group_thin_device │        │            │             │
      │                                         │ DisklessOnRemaining: False                       │        │            │             │
      ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
      

      31a9e0af-8f52-47cd-81e8-71923a45e281-image.jpeg

      J 1 Reply Last reply Reply Quote 0
      • J Online
        Jonathon @Jonathon
        last edited by

        SM logs

        "Date","Time","Line","tsNs","id","clusterLoggingSource","detected_level","hostname","job","priority","service_name","syslog_identifier","unit"
        2026-07-09T22:01:14.842Z,1783634474842,"[2060240][MainThread] ***** Failed to snapshot!: EXCEPTION <class 'linstorvolumemanager.LinstorVolumeManagerError'>, Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01'",1783634474842687000,1783634474842687000_2194f1ea,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service
        2026-07-09T22:01:15.206Z,1783634475206,"[2060240][MainThread] Raising exception [81, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']]",1783634475206274000,1783634475206274000_2137a4ec,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service
        2026-07-09T22:01:15.215Z,1783634475215,"[2060240][MainThread] ***** generic exception: vdi_snapshot: EXCEPTION <class 'xs_errors.SROSError'>, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']",1783634475215733000,1783634475215733000_8881a7dc,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service
        2026-07-09T22:01:15.228Z,1783634475228,"[2060240][MainThread] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'xs_errors.SROSError'>, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']",1783634475228432000,1783634475228432000_9ae0439a,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service
        

        Linstor controller and satellite logs Logs-A-data-2026-07-09 15_11_07.csv.txt

        J 1 Reply Last reply Reply Quote 0
        • J Online
          Jonathon @Jonathon
          last edited by

          jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 error-reports show 6A4E98DD-5D3B7-000942 | cat
          ERROR REPORT 6A4E98DD-5D3B7-000942
          
          ============================================================
          
          Application:                        LINBIT® LINSTOR
          Module:                             Satellite
          Version:                            1.33.1
          Build ID:                           95da7940d6efb6a39ea303c5f37b03478a6fab0b
          Build time:                         2025-12-22T16:04:57+00:00
          Error time:                         2026-07-09 15:01:11
          Node:                               ovbh-pprod-xen05
          Thread:                             DeviceManager
          
          ============================================================
          
          Reported error:
          ===============
          
          Category:                           LinStorException
          Class name:                         ResourceException
          Class canonical name:               com.linbit.linstor.core.devmgr.exceptions.ResourceException
          Generated at:                       Method 'adjustDrbd', Source file 'DrbdLayer.java', Line #933
          
          Error message:                      Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40
          
          Error context:
                  An error occurred while processing resource 'Node: 'ovbh-pprod-xen05', Rsc: 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40''
          ErrorContext:
          
          
          Call backtrace:
          
              Method                                   Native Class:Line number
              adjustDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:933
              processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:281
              lambda$processResource$1                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424
              processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467
              processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420
              processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394
              dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226
              dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:341
              phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149
              devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:786
              run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:682
              run                                      N      java.lang.Thread:829
          
          Caused by:
          ==========
          
          Category:                           LinStorException
          Class name:                         ExtCmdFailedException
          Class canonical name:               com.linbit.extproc.ExtCmdFailedException
          Generated at:                       Method 'execute', Source file 'DrbdAdm.java', Line #805
          
          Error message:                      The external command 'drbdadm' exited with error code 10
          
          
          ErrorContext:
            Description: Execution of the external command 'drbdadm' failed.
            Cause:       The external command exited with error code 10.
            Correction:  - Check whether the external program is operating properly.
          - Check whether the command line is correct.
            Contact a system administrator or a developer if the command line is no longer valid
            for the installed version of the external program.
            Details:     The full command line executed was:
          drbdadm -vvv adjust xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40
          
          The external command sent the following output data:
          
          
          The external command sent the following error information:
          /var/lib/linstor.d/xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40.res:104: in resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40
              ipv4:10.2.4.6:7033 is also used /var/lib/linstor.d/pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9.res:84 (resource pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9)
          
          
          
          
          Call backtrace:
          
              Method                                   Native Class:Line number
              execute                                  N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:805
              adjust                                   N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:134
              adjustDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:890
              processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:281
              lambda$processResource$1                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424
              processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467
              processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420
              processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394
              dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226
              dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:341
              phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149
              devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:786
              run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:682
              run                                      N      java.lang.Thread:829
          
          
          END OF ERROR REPORT.
          
          J 1 Reply Last reply Reply Quote 0
          • J Online
            Jonathon @Jonathon
            last edited by

            hmmmmm

            jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource list -r pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9
            ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
            │ ResourceName                             │ Node             │ Layers       │ Usage │ Conns                        │    State │ Vote │
            ╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
            │ pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 │ ovbh-pprod-xen04 │ DRBD,STORAGE │       │ Ok                           │ DELETING │ Yes  │
            │ pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 │ ovbh-pprod-xen05 │ DRBD,STORAGE │       │ Connecting(ovbh-pprod-xen01) │ DELETING │ Yes  │
            ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
            jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource-connection l pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9
            ╭─────────────────────────────────────────────────────────╮
            │ Source           │ Target           │ Properties │ Port │
            ╞═════════════════════════════════════════════════════════╡
            │ ovbh-pprod-xen04 │ ovbh-pprod-xen05 │            │      │
            ╰─────────────────────────────────────────────────────────╯
            

            Lets delete it, nope lol

            jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 resource delete ovbh-pprod-xen05 pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9
            SUCCESS:
            Description:
                Node: ovbh-pprod-xen05, Resource: pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 preparing for deletion.
            Details:
                Node: ovbh-pprod-xen05, Resource: pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 UUID is: 0485e43e-6dd9-4aa3-962a-99350f04be7e
            SUCCESS:
                Preparing deletion of resource on 'ovbh-pprod-xen04'
            ERROR:
            Description:
                (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed
            Cause:
                The external command for stopping the DRBD resource failed
            Correction:
                - Check whether the required software is installed
                - Check whether the application's search path includes the location
                  of the external software
                - Check whether the application has execute permission for the external command
            Show reports:
                linstor error-reports show 6A4E98DD-5D3B7-000965
            ERROR:
            Description:
                Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
            Details:
                Node: ovbh-pprod-xen05, Resource: pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9
            Show reports:
                linstor error-reports show 6A4D452B-00000-001312
            

            reports

            jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 error-reports show 6A4E98DD-5D3B7-000965 | cat
            ERROR REPORT 6A4E98DD-5D3B7-000965
            
            ============================================================
            
            Application:                        LINBIT® LINSTOR
            Module:                             Satellite
            Version:                            1.33.1
            Build ID:                           95da7940d6efb6a39ea303c5f37b03478a6fab0b
            Build time:                         2025-12-22T16:04:57+00:00
            Error time:                         2026-07-09 15:42:39
            Node:                               ovbh-pprod-xen05
            Thread:                             DeviceManager
            
            ============================================================
            
            Reported error:
            ===============
            
            Category:                           LinStorException
            Class name:                         StorageException
            Class canonical name:               com.linbit.linstor.storage.StorageException
            Generated at:                       Method 'deleteDrbd', Source file 'DrbdLayer.java', Line #591
            
            Error message:                      Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed
            
            Error context:
                    An error occurred while processing resource 'Node: 'ovbh-pprod-xen05', Rsc: 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9''
            ErrorContext:
              Description: Operations on resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' were aborted
              Cause:       The external command for stopping the DRBD resource failed
              Correction:  - Check whether the required software is installed
            - Check whether the application's search path includes the location
              of the external software
            - Check whether the application has execute permission for the external command
            
            
            
            Call backtrace:
            
                Method                                   Native Class:Line number
                deleteDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:591
                processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:272
                lambda$processResource$1                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424
                processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467
                processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420
                processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394
                dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226
                dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:341
                phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149
                devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:786
                run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:682
                run                                      N      java.lang.Thread:829
            
            Caused by:
            ==========
            
            Category:                           LinStorException
            Class name:                         ExtCmdFailedException
            Class canonical name:               com.linbit.extproc.ExtCmdFailedException
            Generated at:                       Method 'execute', Source file 'DrbdAdm.java', Line #810
            
            Error message:                      The external command 'drbdsetup' did not complete within the timeout
            
            ErrorContext:
              Description: Execution of the external command 'drbdsetup' failed.
              Cause:       The external command did not complete within the timeout.
            Possible causes include:
            - The system load may be too high to ensure completion of external commands in a timely manner.
            - The program implementing the external command may not be operating properly.
            - The operating system may have entered an erroneous state.
              Correction:  Check whether the external program and the operating system are still operating properly.
            Check whether the system's load is within normal parameters.
            
              Details:     The full command line executed was:
            drbdsetup down pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9
            
            
            Call backtrace:
            
                Method                                   Native Class:Line number
                execute                                  N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:810
                simpleSetupCommand                       N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:735
                down                                     N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:213
                down                                     N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:178
                deleteDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:553
                processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:272
                lambda$processResource$1                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424
                processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467
                processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420
                processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394
                dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226
                dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:341
                phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149
                devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:786
                run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:682
                run                                      N      java.lang.Thread:829
            
            Caused by:
            ==========
            
            Category:                           Exception
            Class name:                         ChildProcessTimeoutException
            Class canonical name:               com.linbit.ChildProcessTimeoutException
            Generated at:                       Method 'waitFor', Source file 'ChildProcessHandler.java', Line #134
            
            
            Call backtrace:
            
                Method                                   Native Class:Line number
                waitFor                                  N      com.linbit.extproc.ChildProcessHandler:134
                syncProcess                              N      com.linbit.extproc.ExtCmd:162
                pipeExec                                 N      com.linbit.extproc.ExtCmd:108
                execute                                  N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:802
                simpleSetupCommand                       N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:735
                down                                     N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:213
                down                                     N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:178
                deleteDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:553
                processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:272
                lambda$processResource$1                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1424
                processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1467
                processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1420
                processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:394
                dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:226
                dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:341
                phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1149
                devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:786
                run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:682
                run                                      N      java.lang.Thread:829
            
            
            END OF ERROR REPORT.
            
            jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10 error-reports show 6A4D452B-00000-001312 | cat
            ERROR REPORT 6A4D452B-00000-001312
            
            ============================================================
            
            Application:                        LINBIT® LINSTOR
            Module:                             Controller
            Version:                            1.33.1
            Build ID:                           95da7940d6efb6a39ea303c5f37b03478a6fab0b
            Build time:                         2025-12-22T16:04:57+00:00
            Error time:                         2026-07-09 18:42:39
            Node:                               ovbh-pprod-xen01
            Thread:                             MainWorkerPool-6
            Access context information
            
            Identity:                           PUBLIC
            Role:                               PUBLIC
            Domain:                             PUBLIC
            
            Peer:                               RestClient(10.1.8.104; 'PythonLinstor/1.28.2 (API1.0.4): Client 1.28.1')
            
            ============================================================
            
            Reported error:
            ===============
            
            Category:                           RuntimeException
            Class name:                         DelayedApiRcException
            Class canonical name:               com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils.DelayedApiRcException
            Generated at:                       Method 'lambda$mergeExtractingApiRcExceptions$6', Source file 'CtrlResponseUtils.java', Line #187
            
            Error message:                      Exceptions have been converted to responses
            
            Error context:
                    Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
            Asynchronous stage backtrace:
                    (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed
                
                Error has been observed at the following site(s):
                	*__checkpoint ⇢ Prepare resource delete
                	*__checkpoint ⇢ Activating resource if necessary before deletion
                Original Stack Trace:
            
            Call backtrace:
            
                Method                                   Native Class:Line number
                lambda$mergeExtractingApiRcExceptions$6  N      com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils:187
            
            Suppressed exception 1 of 2:
            ===============
            Category:                           RuntimeException
            Class name:                         ApiRcException
            Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
            Generated at:                       Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #353
            
            Error message:                      (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed
            
            Error context:
                    Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
            ApiRcException entries: 
              Message:     (ovbh-pprod-xen05) Shutdown of the DRBD resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9 failed
              Cause:       The external command for stopping the DRBD resource failed
              Correction:  - Check whether the required software is installed
            - Check whether the application's search path includes the location
              of the external software
            - Check whether the application has execute permission for the external command
            
              NumericCode: -4611686018427386906
            
            
            Call backtrace:
            
                Method                                   Native Class:Line number
                handleAnswer                             N      com.linbit.linstor.proto.CommonMessageProcessor:353
                handleDataMessage                        N      com.linbit.linstor.proto.CommonMessageProcessor:306
                doProcessInOrderMessage                  N      com.linbit.linstor.proto.CommonMessageProcessor:254
                lambda$doProcessMessage$5                N      com.linbit.linstor.proto.CommonMessageProcessor:239
                subscribe                                N      reactor.core.publisher.FluxDefer:46
                subscribe                                N      reactor.core.publisher.Flux:8848
                onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:430
                drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
                drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
                onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
                drainFused                               N      reactor.core.publisher.SinkManyUnicast:321
                drain                                    N      reactor.core.publisher.SinkManyUnicast:363
                tryEmitNext                              N      reactor.core.publisher.SinkManyUnicast:239
                tryEmitNext                              N      reactor.core.publisher.SinkManySerialized:100
                processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:446
                doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:237
                lambda$processMessage$3                  N      com.linbit.linstor.proto.CommonMessageProcessor:174
                onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:185
                runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
                run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
                call                                     N      reactor.core.scheduler.WorkerTask:84
                call                                     N      reactor.core.scheduler.WorkerTask:37
                run                                      N      java.util.concurrent.FutureTask:264
                run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
                runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1128
                run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:628
                run                                      N      java.lang.Thread:829
            
            Suppressed exception 2 of 2:
            ===============
            Category:                           RuntimeException
            Class name:                         OnAssemblyException
            Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
            Generated at:                       Method 'lambda$mergeExtractingApiRcExceptions$6', Source file 'CtrlResponseUtils.java', Line #187
            
            Error message:                      
            Error has been observed at the following site(s):
            	*__checkpoint ⇢ Prepare resource delete
            	*__checkpoint ⇢ Activating resource if necessary before deletion
            Original Stack Trace:
            
            Error context:
                    Deletion of resource 'pvc-a432777b-b5ee-4b98-8c9d-a01f35a84fc9' on node 'ovbh-pprod-xen05' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
            Call backtrace:
            
                Method                                   Native Class:Line number
                lambda$mergeExtractingApiRcExceptions$6  N      com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils:187
                subscribe                                N      reactor.core.publisher.FluxDefer:46
                subscribe                                N      reactor.core.publisher.Flux:8848
                onComplete                               N      reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:238
                onComplete                               N      reactor.core.publisher.FluxMap$MapSubscriber:144
                checkTerminated                          N      reactor.core.publisher.FluxFlatMap$FlatMapMain:850
                drainLoop                                N      reactor.core.publisher.FluxFlatMap$FlatMapMain:612
                innerComplete                            N      reactor.core.publisher.FluxFlatMap$FlatMapMain:898
                onComplete                               N      reactor.core.publisher.FluxFlatMap$FlatMapInner:1001
                onComplete                               N      reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2230
                request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2572
                set                                      N      reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2366
                onSubscribe                              N      reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:74
                subscribe                                N      reactor.core.publisher.FluxJust:68
                subscribe                                N      reactor.core.publisher.Flux:8848
                onError                                  N      reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:103
                onError                                  N      reactor.core.publisher.FluxMap$MapSubscriber:134
                onError                                  N      reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:186
                onError                                  N      reactor.core.publisher.FluxPeek$PeekSubscriber:222
                onError                                  N      reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:106
                error                                    N      reactor.core.publisher.Operators:198
                subscribe                                N      reactor.core.publisher.FluxError:43
                subscribe                                N      reactor.core.publisher.Flux:8848
                onError                                  N      reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber:103
                onError                                  N      reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:142
                onError                                  N      reactor.core.publisher.FluxUsing$UsingSubscriber:217
                onError                                  N      reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2235
                error                                    N      reactor.core.publisher.FluxCreate$BaseSink:479
                drain                                    N      reactor.core.publisher.FluxCreate$BufferAsyncSink:868
                error                                    N      reactor.core.publisher.FluxCreate$BufferAsyncSink:813
                drainLoop                                N      reactor.core.publisher.FluxCreate$SerializedFluxSink:239
                drain                                    N      reactor.core.publisher.FluxCreate$SerializedFluxSink:215
                error                                    N      reactor.core.publisher.FluxCreate$SerializedFluxSink:191
                apiCallError                             N      com.linbit.linstor.netcom.TcpConnectorPeer:529
                handleAnswer                             N      com.linbit.linstor.proto.CommonMessageProcessor:365
                handleDataMessage                        N      com.linbit.linstor.proto.CommonMessageProcessor:306
                doProcessInOrderMessage                  N      com.linbit.linstor.proto.CommonMessageProcessor:254
                lambda$doProcessMessage$5                N      com.linbit.linstor.proto.CommonMessageProcessor:239
                subscribe                                N      reactor.core.publisher.FluxDefer:46
                subscribe                                N      reactor.core.publisher.Flux:8848
                onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:430
                drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
                drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
                onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
                drainFused                               N      reactor.core.publisher.SinkManyUnicast:321
                drain                                    N      reactor.core.publisher.SinkManyUnicast:363
                tryEmitNext                              N      reactor.core.publisher.SinkManyUnicast:239
                tryEmitNext                              N      reactor.core.publisher.SinkManySerialized:100
                processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:446
                doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:237
                lambda$processMessage$3                  N      com.linbit.linstor.proto.CommonMessageProcessor:174
                onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:185
                runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
                run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
                call                                     N      reactor.core.scheduler.WorkerTask:84
                call                                     N      reactor.core.scheduler.WorkerTask:37
                run                                      N      java.util.concurrent.FutureTask:264
                run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
                runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1128
                run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:628
                run                                      N      java.lang.Thread:829
            
            
            END OF ERROR REPORT.
            
            J 1 Reply Last reply Reply Quote 0
            • J Online
              Jonathon @Jonathon
              last edited by

              OK so that resource was a pain and did not want to give up, so I restarted the host.

              Migration is running good now.

              1 Reply Last reply Reply Quote 0

              Hello! It looks like you're interested in this conversation, but you don't have an account yet.

              Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

              With your input, this post could be even better 💗

              Register Login
              • First post
                Last post