XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XOSTOR hyperconvergence preview

    Scheduled Pinned Locked Moved XOSTOR
    446 Posts 47 Posters 479.7k Views 48 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ronan-aR Offline
      ronan-a Vates šŸŖ XCP-ng Team @geoffbland
      last edited by

      @geoffbland Okay so it's probably not related to the driver itself, I will take a look to the logs after reception. šŸ™‚

      G 1 Reply Last reply Reply Quote 0
      • G Offline
        geoffbland
        last edited by

        Failure trying to revert a VM to a snapshot with XOSTOR.

        Created a VM with main VDI on XOSTOR (24GB) and with 6 disks each also on XOSTOR (2GB each).
        All is running OK.
        Now create a snapshot of the VM - this takes quite a while but does eventually succeed.
        Now using XO (from sources) click the "Revert VM to this snapshot". This errors and the VM stops.

        vm.revert
{
  "snapshot": "6032fc73-eb7f-cf64-2481-4346b7b57204"
}
{
  "code": "VM_REVERT_FAILED",
  "params": [
    "OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378",
    "OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add"
  ],
  "task": {
    "uuid": "4804fefd-0037-d7dd-9a7c-769230728483",
    "name_label": "Async.VM.revert",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20220527T15:01:42Z",
    "finished": "20220527T15:01:46Z",
    "status": "failure",
    "resident_on": "OpaqueRef:a1e9a8f3-0a79-4824-b29f-d81b3246d190",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_REVERT_FAILED",
      "OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378",
      "OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 492))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "VM_REVERT_FAILED(OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378, OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add)",
  "name": "XapiError",
  "stack": "XapiError: VM_REVERT_FAILED(OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378, OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:949:24)
    at forEach (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:983:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:973:12)
    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:1139:14)"
}

        Now viewing the VM with XO on the disks tab shows no attached disks - disk tab is blank.

        XOSTOR01 Storage No Disks.png

        But linstor appears to still have the disks and the snapshot disks too.

        ā”Š XCPNG01 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š           ā”Š InUse  ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-30bf014b-025d-4f3f-a068-f9a9bf34fab2 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1013 ā”Š /dev/drbd1013 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-30bf014b-025d-4f3f-a068-f9a9bf34fab2 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1013 ā”Š /dev/drbd1013 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-30bf014b-025d-4f3f-a068-f9a9bf34fab2 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1013 ā”Š /dev/drbd1013 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-7aaa7a6e-98c4-4a57-a4f1-4fea0a36b17a ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1011 ā”Š /dev/drbd1011 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-7aaa7a6e-98c4-4a57-a4f1-4fea0a36b17a ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1011 ā”Š /dev/drbd1011 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-7aaa7a6e-98c4-4a57-a4f1-4fea0a36b17a ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1011 ā”Š /dev/drbd1011 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-b341848b-01d1-4019-a62f-85c6108a53e3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1006 ā”Š /dev/drbd1006 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-b341848b-01d1-4019-a62f-85c6108a53e3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1006 ā”Š /dev/drbd1006 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-b341848b-01d1-4019-a62f-85c6108a53e3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1006 ā”Š /dev/drbd1006 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG01 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
ā”Š XCPNG02 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG03 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG04 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
ā”Š XCPNG05 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š           ā”Š Unused ā”Š Diskless ā”Š

        From the VM disks tab if I try to Attach the disks, two of the disks created on XOSTOR are missing (data1 and data4).

        XOSTOR01 Storage Disks Missing.png

        Finally if I go to storage and bring up the XOSTOR storage and then press "Rescan all disks" I get this error:

        sr.scan
{
  "id": "cf896912-cd71-d2b2-488a-5792b7147c87"
}
{
  "code": "SR_BACKEND_FAILURE_46",
  "params": [
    "",
    "The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']]",
    ""
  ],
  "task": {
    "uuid": "4dcac885-dfaa-784a-eb2d-02335efde0fb",
    "name_label": "Async.SR.scan",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20220527T16:27:36Z",
    "finished": "20220527T16:27:50Z",
    "status": "failure",
    "resident_on": "OpaqueRef:a1e9a8f3-0a79-4824-b29f-d81b3246d190",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "SR_BACKEND_FAILURE_46",
      "",
      "The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']]",
      ""
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_access.ml)(line 32))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 128))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']], )
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:949:24)
    at forEach (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:983:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:973:12)
    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:1139:14)"
}
        1 Reply Last reply Reply Quote 0
        • G Offline
          geoffbland @ronan-a
          last edited by

          @ronan-a said in XOSTOR hyperconvergence preview:

          Okay so it's probably not related to the driver itself, I will take a look to the logs after reception. šŸ™‚

          Did you get chance to look at the logs I sent?

          ronan-aR 1 Reply Last reply Reply Quote 0
          • ronan-aR Offline
            ronan-a Vates šŸŖ XCP-ng Team @geoffbland
            last edited by ronan-a

            @geoffbland So, I didn't notice useful info outside of:

            FIXME drbd_a_xcp-volu[24302] op clear, bitmap locked for 'set_n_write sync_handshake' by drbd_r_xcp-volu[24231]
...
FIXME drbd_a_xcp-volu[24328] op clear, bitmap locked for 'demote' by drbd_w_xcp-volu[24188]

            Like I said in my e-mail, maybe there are more details in another log file. I hope. šŸ™‚

            G 2 Replies Last reply Reply Quote 0
            • G Offline
              geoffbland @ronan-a
              last edited by

              @ronan-a said in [XOSTOR hyperconvergence preview]

              Like I said in my e-mail, maybe there are more details in another log file. I hope. šŸ™‚

              In the end I realised I was more trying to "use" XOSTOR whilst testing rather than properly test it. So I decided to rip it all down and start again and retest it again - this time properly recording each step so any issues can be replicated. I will let you know how this goes.

              1 Reply Last reply Reply Quote 1
              • G Offline
                geoffbland @ronan-a
                last edited by

                @ronan-a Could you please take a look at this issue I raised elsewhere on the forums.

                I am currently unable to create new VMs, getting a No such Tapdisk error - checking down the stack trace - it seems to be coming from a get() call in /opt/xensource/sm/blktap2.py and this code seems to have been changed in a XOSTOR release from 24th May.

                ronan-aR 1 Reply Last reply Reply Quote 0
                • ronan-aR Offline
                  ronan-a Vates šŸŖ XCP-ng Team @geoffbland
                  last edited by

                  @geoffbland Hi, I was away last week, I will take a look. šŸ™‚

                  1 Reply Last reply Reply Quote 0
                  • Y Offline
                    yrrips
                    last edited by

                    This is a supercool and amazing thing. Coming from Proxmox with Ceph I really feel this was a missing piece. I want to migrate my whole homelab to this!

                    I've been playing around with it a fair bit, and it works well, but when it came to enabling HA I ran into trouble. Is XOSTOR not a valid shared storage target to enable HA on?

                    Having multiple shared storages (NFS/CIFS etc) in production is a given to put backups and whatnot on, but I thought it was weird that I couldn't use XOSTOR storage to enable HA.

                    G ronan-aR 2 Replies Last reply Reply Quote 0
                    • G Offline
                      geoffbland @yrrips
                      last edited by

                      @yrrips said in XOSTOR hyperconvergence preview:

                      when it came to enabling HA I ran into trouble. Is XOSTOR not a valid shared storage target to enable HA on?

                      I'm also really hoping that XOSTOR works well and also feel like this is something XCP-NG really needs. I've tried other distributed storage solutions, notably GlusterFS but never found anything that really works 100% when outages occur.

                      Note I plan to use XOSTOR just for VM disks and any data they use - not for the HA share and not for backups. My logic is that:

                      • backups should not be anywhere near XCP-NG and should be isolated software and hardware - and hopefully location wise.
                      • HA share needs to survive a issue where XCP-NG HA fails; if XCP-NG quorum is not working then XOSTOR (Linstor) quorum may be affected in a similar way - so I keep my HA share on a reliable NFS share. Note that in testing I found if the HA share is not available for a while that XCP-NG stays running OK (just don't make any configuration changes until HA share is back).
                      1 Reply Last reply Reply Quote 0
                      • ronan-aR Offline
                        ronan-a Vates šŸŖ XCP-ng Team @yrrips
                        last edited by

                        @yrrips I fixed several problems with the HA and the linstor driver. I don't know what's your sm version, but I updated it few weeks ago (current: sm-2.30.6-1.2.0.linstor.1.xcpng8.2.x86_64). Could you give me more details?

                        1 Reply Last reply Reply Quote 1
                        • M Offline
                          markhewitt1978
                          last edited by

                          I have been waiting for this for some time. after XOSAN didn't really work for me, but neither did OVH's SAN offering.

                          Is there a documentation page to look at rather than going through the thread?

                          In the absense of that is there a minumum recommanded host count?

                          Would love to give this a try but would mean playing with production VMs!

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates šŸŖ Co-Founder CEO
                            last edited by

                            Hello @markhewitt1978 !

                            • First post is a complete guide šŸ™‚
                            • 3 hosts is fine, no less. No problem with more.

                            We are still investigating a bug we discovered recently, so I wouldn't play in production right now (except if you are very confident and you have a lot of backup).

                            G 1 Reply Last reply Reply Quote 0
                            • G Offline
                              geoffbland @olivierlambert
                              last edited by

                              Is it possible to use a separate network for the XOSTOR/Linstor disk replication from the "main" network used for XCP-ng servers?

                              If so when the SR is created with this command:

                              xe sr-create type=linstor name-label=XOSTOR host-uuid=bc3cd3af-3f09-48cf-ae55-515ba21930f5 device-config:hosts=host-a,host-b,host-c,host-d device-config:group-name=linstor_group/thin_device device-config:redundancy=4 shared=true device-config:provisioning=thin

                              Do the device-config:hosts need to be XCP-ng hosts - or can IP address of the "data-replication" network be provided here.

                              For example, my XCP-NG servers have dual NICs, I could use the second NIC on a private network/switch with a different subnet to the "main" hosts and use this solely for XOSTOR/Linstor disk replication. Is this possible?

                              G 1 Reply Last reply Reply Quote 0
                              • G Offline
                                geoffbland @geoffbland
                                last edited by geoffbland

                                Doing some more testing on XOSTOR and starting from scratch again. A brand-new XCP-ng installation made onto a new 3 server pool, each server with a blank 4GB disk.
                                All servers have xcpng patched up to date.
                                Then I installed XOSTOR onto each of these servers, Linstor installed OK and I have the linstor group on each.

                                [19:22 XCPNG30 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize    VFree
  VG_XenStorage-a776b6b1-9a96-e179-ea12-f2419ae512b6   1   1   0 wz--n- <405.62g 405.61g
  linstor_group                                        1   0   0 wz--n-   <3.64t  <3.64t
[19:22 XCPNG30 ~]# rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
xha-10.1.0-2.2.0.linstor.2.xcpng8.2.x86_64
sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64
sm-rawhba-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64

[19:21 XCPNG31 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize    VFree
  VG_XenStorage-f75785ef-df30-b54c-2af4-84d19c966453   1   1   0 wz--n- <405.62g 405.61g
  linstor_group                                        1   0   0 wz--n-   <3.64t  <3.64t
[19:21 XCPNG31 ~]# rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
xha-10.1.0-2.2.0.linstor.2.xcpng8.2.x86_64
sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64
sm-rawhba-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64

[19:23 XCPNG32 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize    VFree
  VG_XenStorage-abaf8356-fc58-9124-a23b-c29e7e67c983   1   1   0 wz--n- <405.62g 405.61g
  linstor_group                                        1   0   0 wz--n-   <3.64t  <3.64t
[19:23 XCPNG32 ~]# rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
xha-10.1.0-2.2.0.linstor.2.xcpng8.2.x86_64
sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64
sm-rawhba-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64

[19:26 XCPNG31 ~]# xe host-list
uuid ( RO)                : 7c3f2fae-0456-4155-a9ad-43790fcb4155
          name-label ( RW): XCPNG32
    name-description ( RW): Default install

uuid ( RO)                : 2e48b46a-c420-4957-9233-3e029ea39305
          name-label ( RW): XCPNG30
    name-description ( RW): Default install

uuid ( RO)                : 7aaaf4a5-0e43-442e-a9b1-38620c87fd69
          name-label ( RW): XCPNG31
    name-description ( RW): Default install

                                But I am not able to create the SR.

                                xe sr-create type=linstor name-label=XOSTOR01 host-uuid=7aaaf4a5-0e43-442e-a9b1-38620c87fd69 device-config:hosts=xcpng30,xcpng31,xcpng32 device-config:group-name=linstor_group device-config:redundancy=2 shared=true device-config:provisioning=thick

                                This gives the following error:

                                Error code: SR_BACKEND_FAILURE_5006
Error parameters: , LINSTOR SR creation error [opterr=Not enough online hosts],

                                Here's the error in the SMLog

                                Jul 15 19:29:22 XCPNG31 SM: [9747] sr_create {'sr_uuid': '14aa2b8b-430f-34e5-fb74-c37667cb18ec', 'subtask_of': 'DummyRef:|d39839f1-ee3a-4bfe-8a41-7a077f4f2640|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:196f738d-24fa-4598-8e96-4a13390abc87', 'session_ref': 'OpaqueRef:e806b347-1e5f-4644-842f-26a7b06b2561', 'device_config': {'group-name': 'linstor_group', 'redundancy': '2', 'hosts': 'xcpng30,xcpng31,xcpng32', 'SRmaster': 'true', 'provisioning': 'thick'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:7ded7feb-729f-47c3-9893-1b62db0b7e17'}
Jul 15 19:29:22 XCPNG31 SM: [9747] LinstorSR.create for 14aa2b8b-430f-34e5-fb74-c37667cb18ec
Jul 15 19:29:22 XCPNG31 SM: [9747] Raising exception [5006, LINSTOR SR creation error [opterr=Not enough online hosts]]
Jul 15 19:29:22 XCPNG31 SM: [9747] lock: released /var/lock/sm/14aa2b8b-430f-34e5-fb74-c37667cb18ec/sr
Jul 15 19:29:22 XCPNG31 SM: [9747] ***** generic exception: sr_create: EXCEPTION <class 'SR.SROSError'>, LINSTOR SR creation error [opterr=Not enough online hosts]
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return self._run_locked(sr)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 15 19:29:22 XCPNG31 SM: [9747]     rv = self._run(sr, target)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 612, in wrap
Jul 15 19:29:22 XCPNG31 SM: [9747]     return load(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 597, in load
Jul 15 19:29:22 XCPNG31 SM: [9747]     return wrapped_method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 443, in wrapped_method
Jul 15 19:29:22 XCPNG31 SM: [9747]     return method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 688, in create
Jul 15 19:29:22 XCPNG31 SM: [9747]     opterr='Not enough online hosts'
Jul 15 19:29:22 XCPNG31 SM: [9747]
Jul 15 19:29:22 XCPNG31 SM: [9747] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'SR.SROSError'>, LINSTOR SR creation error [opterr=Not enough online hosts]
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Jul 15 19:29:22 XCPNG31 SM: [9747]     ret = cmd.run(sr)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return self._run_locked(sr)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 15 19:29:22 XCPNG31 SM: [9747]     rv = self._run(sr, target)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 612, in wrap
Jul 15 19:29:22 XCPNG31 SM: [9747]     return load(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 597, in load
Jul 15 19:29:22 XCPNG31 SM: [9747]     return wrapped_method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 443, in wrapped_method
Jul 15 19:29:22 XCPNG31 SM: [9747]     return method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 688, in create
Jul 15 19:29:22 XCPNG31 SM: [9747]     opterr='Not enough online hosts'

                                I have found the issue - the device-config:hosts list is case-sensitive, if the hosts are given in lower-case the above error occurs. Specifying the hosts in upper-case works.

                                Also using a fully-qualified name for the host fails - regardless of the case used.

                                ronan-aR 1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates šŸŖ Co-Founder CEO
                                  last edited by

                                  Pinging @ronan-a

                                  1 Reply Last reply Reply Quote 0
                                  • ronan-aR Offline
                                    ronan-a Vates šŸŖ XCP-ng Team @geoffbland
                                    last edited by

                                    @geoffbland

                                    Do the device-config:hosts need to be XCP-ng hosts - or can IP address of the "data-replication" network be provided here.

                                    This param must use the names of the XAPI host objects. The names are reused in the LINSTOR configuration by the smapi driver. I will try to remove this param later to simplify xe sr-create command.

                                    For example, my XCP-NG servers have dual NICs, I could use the second NIC on a private network/switch with a different subnet to the "main" hosts and use this solely for XOSTOR/Linstor disk replication. Is this possible?

                                    Yes, you can. Please take a look to: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-managing_network_interface_cards

                                    To get the storage pool name, execute this command in your pool:

                                    linstor --controllers=<comma-separated-list-of-ips> storage-pool list
                                    G 2 Replies Last reply Reply Quote 0
                                    • G Offline
                                      geoffbland @ronan-a
                                      last edited by

                                      @ronan-a said in XOSTOR hyperconvergence preview:

                                      Yes, you can. Please take a look to: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-managing_network_interface_cards
                                      To get the storage pool name, execute this command in your pool:

                                      OK - so to set up this XOSTOR SR I would first

                                      xe sr-create type=linstor name-label=XOSTOR01 host-uuid=xxx device-config:hosts=<XCPNG Host Names> ...etc...

                                      to create the Linstor storage-pool. Then

                                      linstor storage-pool list

                                      to get the name the pool. Then on each node found in device-config:hosts=<XCPNG Host Names> run the following command:

                                      linstor storage-pool set-property <host/node_name> <pool_name> PrefNic <nic_name>

                                      where nic_name is the name of the Linstor interface created for the specific NIC.

                                      1 Reply Last reply Reply Quote 0
                                      • G Offline
                                        geoffbland @ronan-a
                                        last edited by

                                        @ronan-a

                                        As promised I have done some more "organised" testing, with a brand new cluster set up to test XOSTOR. Early simple tests seemed to be OK and I can create, restart, snapshot, move and delete VMs with no problem.

                                        But then after putting a VM under load for an hour and then restarting it I am seeing the same weird behaviour I saw previously with XOSTOR.

                                        Firstly, the VM took far longer than expected to shutdown. Then trying to restart the VM fails.

                                        Let me know if you want me to provide logs or any specific testing.

                                        vm.start
{
  "id": "ade699f2-42f0-8629-35ea-6fcc69de99d7",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "SR_BACKEND_FAILURE_1200",
  "params": [
    "",
    "No such Tapdisk(minor=12)",
    ""
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8",
      false,
      false
    ]
  },
  "message": "SR_BACKEND_FAILURE_1200(, No such Tapdisk(minor=12), )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_1200(, No such Tapdisk(minor=12), )
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.callbackTrampoline (internal/async_hooks.js:130:17)"
}

                                        The linstore volume for this VM appears to be OK (by the way I hope eventually we have an easy way to match up VMs to Linstore volumes)

                                        [16:53 XCPNG31 ~]# linstor volume list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| XCPNG30 | xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | xcp-sr-linstor_group |     0 |    1001 | /dev/drbd1001 | 40.10 GiB | InUse  |   UpToDate |
| XCPNG31 | xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | xcp-sr-linstor_group |     0 |    1001 | /dev/drbd1001 | 40.10 GiB | Unused |   UpToDate |
| XCPNG32 | xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | DfltDisklessStorPool |     0 |    1001 | /dev/drbd1001 |           | Unused |   Diskless |

[16:59 XCPNG31 ~]# linstor node list
ā•­ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•®
ā”Š Node    ā”Š NodeType ā”Š Addresses                 ā”Š State  ā”Š
ā•žā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•”
ā”Š XCPNG30 ā”Š COMBINED ā”Š 192.168.1.30:3366 (PLAIN) ā”Š Online ā”Š
ā”Š XCPNG31 ā”Š COMBINED ā”Š 192.168.1.31:3366 (PLAIN) ā”Š Online ā”Š
ā”Š XCPNG32 ā”Š COMBINED ā”Š 192.168.1.32:3366 (PLAIN) ā”Š Online ā”Š
ā•°ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•Æ
[17:13 XCPNG31 ~]# linstor storage-pool list
ā•­ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•®
ā”Š StoragePool          ā”Š Node    ā”Š Driver   ā”Š PoolName      ā”Š FreeCapacity ā”Š TotalCapacity ā”Š CanSnapshots ā”Š State ā”Š SharedName ā”Š
ā•žā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•”
ā”Š DfltDisklessStorPool ā”Š XCPNG30 ā”Š DISKLESS ā”Š               ā”Š              ā”Š               ā”Š False        ā”Š Ok    ā”Š            ā”Š
ā”Š DfltDisklessStorPool ā”Š XCPNG31 ā”Š DISKLESS ā”Š               ā”Š              ā”Š               ā”Š False        ā”Š Ok    ā”Š            ā”Š
ā”Š DfltDisklessStorPool ā”Š XCPNG32 ā”Š DISKLESS ā”Š               ā”Š              ā”Š               ā”Š False        ā”Š Ok    ā”Š            ā”Š
ā”Š xcp-sr-linstor_group ā”Š XCPNG30 ā”Š LVM      ā”Š linstor_group ā”Š     3.50 TiB ā”Š      3.64 TiB ā”Š False        ā”Š Ok    ā”Š            ā”Š
ā”Š xcp-sr-linstor_group ā”Š XCPNG31 ā”Š LVM      ā”Š linstor_group ā”Š     3.49 TiB ā”Š      3.64 TiB ā”Š False        ā”Š Ok    ā”Š            ā”Š
ā”Š xcp-sr-linstor_group ā”Š XCPNG32 ā”Š LVM      ā”Š linstor_group ā”Š     3.50 TiB ā”Š      3.64 TiB ā”Š False        ā”Š Ok    ā”Š            ā”Š
ā•°ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•Æ

                                        I am seeing the following errors in the SMLog (bb027c9a-5655-4f93-9090-e76b34b2c90d is the disk for this VM)

                                        Jul 23 16:48:30 XCPNG30 SM: [22961] lock: opening lock file /var/lock/sm/bb027c9a-5655-4f93-9090-e76b34b2c90d/vdi
Jul 23 16:48:30 XCPNG30 SM: [22961] blktap2.deactivate
Jul 23 16:48:30 XCPNG30 SM: [22961] lock: acquired /var/lock/sm/bb027c9a-5655-4f93-9090-e76b34b2c90d/vdi
Jul 23 16:48:30 XCPNG30 SM: [22961] ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30']
Jul 23 16:49:00 XCPNG30 SM: [22961]  = 5
Jul 23 16:49:00 XCPNG30 SM: [22961] ***** BLKTAP2:<function _deactivate_locked at 0x7f6fb33208c0>: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30'] failed: status=5, pid=22983, errmsg=Input/output error
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 85, in wrapper
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = op(self, *args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1744, in _deactivate_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._deactivate(sr_uuid, vdi_uuid, caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1785, in _deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._tap_deactivate(minor)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1368, in _tap_deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapdisk.shutdown()
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 880, in shutdown
Jul 23 16:49:00 XCPNG30 SM: [22961]     TapCtl.close(self.pid, self.minor, force)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 433, in close
Jul 23 16:49:00 XCPNG30 SM: [22961]     cls._pread(args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 296, in _pread
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapctl._wait(quiet)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 285, in _wait
Jul 23 16:49:00 XCPNG30 SM: [22961]     raise self.CommandFailure(self.cmd, **info)
Jul 23 16:49:00 XCPNG30 SM: [22961]
Jul 23 16:49:00 XCPNG30 SM: [22961] lock: released /var/lock/sm/bb027c9a-5655-4f93-9090-e76b34b2c90d/vdi
Jul 23 16:49:00 XCPNG30 SM: [22961] call-plugin on 7aaaf4a5-0e43-442e-a9b1-38620c87fd69 (linstor-manager:lockVdi with {'groupName': 'linstor_group', 'srUuid': '141d63f6-d3ed-4a2f-588a-1835f0cea588', 'vdiUuid': 'bb027c9a-5655-4f93-9090-e76b34b2c90d', 'locked': 'False'}) returned: True
Jul 23 16:49:00 XCPNG30 SM: [22961] ***** generic exception: vdi_deactivate: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30'] failed: status=5, pid=22983, errmsg=Input/output error
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 23 16:49:00 XCPNG30 SM: [22961]     return self._run_locked(sr)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     rv = self._run(sr, target)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 274, in _run
Jul 23 16:49:00 XCPNG30 SM: [22961]     caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1729, in deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     if self._deactivate_locked(sr_uuid, vdi_uuid, caching_params):
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 85, in wrapper
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = op(self, *args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1744, in _deactivate_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._deactivate(sr_uuid, vdi_uuid, caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1785, in _deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._tap_deactivate(minor)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1368, in _tap_deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapdisk.shutdown()
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 880, in shutdown
Jul 23 16:49:00 XCPNG30 SM: [22961]     TapCtl.close(self.pid, self.minor, force)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 433, in close
Jul 23 16:49:00 XCPNG30 SM: [22961]     cls._pread(args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 296, in _pread
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapctl._wait(quiet)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 285, in _wait
Jul 23 16:49:00 XCPNG30 SM: [22961]     raise self.CommandFailure(self.cmd, **info)
Jul 23 16:49:00 XCPNG30 SM: [22961]
Jul 23 16:49:00 XCPNG30 SM: [22961] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30'] failed: status=5, pid=22983, errmsg=Input/output error
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = cmd.run(sr)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 23 16:49:00 XCPNG30 SM: [22961]     return self._run_locked(sr)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     rv = self._run(sr, target)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 274, in _run
Jul 23 16:49:00 XCPNG30 SM: [22961]     caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1729, in deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     if self._deactivate_locked(sr_uuid, vdi_uuid, caching_params):
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 85, in wrapper
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = op(self, *args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1744, in _deactivate_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._deactivate(sr_uuid, vdi_uuid, caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1785, in _deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._tap_deactivate(minor)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1368, in _tap_deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapdisk.shutdown()
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 880, in shutdown
Jul 23 16:49:00 XCPNG30 SM: [22961]     TapCtl.close(self.pid, self.minor, force)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 433, in close
Jul 23 16:49:00 XCPNG30 SM: [22961]     cls._pread(args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 296, in _pread
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapctl._wait(quiet)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 285, in _wait
Jul 23 16:49:00 XCPNG30 SM: [22961]     raise self.CommandFailure(self.cmd, **info)
Jul 23 16:49:00 XCPNG30 SM: [22961]

                                        If I try to start this VM after the above issue I get this error:

                                        vm.start
{
  "id": "ade699f2-42f0-8629-35ea-6fcc69de99d7",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "FAILED_TO_START_EMULATOR",
  "params": [
    "OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8",
    "domid 12",
    "QMP failure at File \"xc/device.ml\", line 3366, characters 71-78"
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8",
      false,
      false
    ]
  },
  "message": "FAILED_TO_START_EMULATOR(OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8, domid 12, QMP failure at File \"xc/device.ml\", line 3366, characters 71-78)",
  "name": "XapiError",
  "stack": "XapiError: FAILED_TO_START_EMULATOR(OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8, domid 12, QMP failure at File \"xc/device.ml\", line 3366, characters 71-78)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.callbackTrampoline (internal/async_hooks.js:130:17)"
}

                                        This VM is set up the same way as several other VMs I have running on another cluster but using SRs on NFS mounts.

                                        This is with latest XCP-ng version, all servers patched and up-to-date.

                                        yum update
...
No packages marked for update
                                        ronan-aR 1 Reply Last reply Reply Quote 0
                                        • ronan-aR Offline
                                          ronan-a Vates šŸŖ XCP-ng Team @geoffbland
                                          last edited by

                                          @geoffbland Thank you for your tests. šŸ™‚

                                          Could you send me the other logs (/var/log/SMlog + kern.log + drbd-kern.log please)? Also check if the LVM volumes are reachable with linstor resource list. Also, you can check with lvs command on each host. EIO error is not a nice error to observe.

                                          G 1 Reply Last reply Reply Quote 0
                                          • G Offline
                                            geoffbland @ronan-a
                                            last edited by

                                            @ronan-a The volumes are reachable on all nodes:

                                            [16:13 XCPNG30 ~]# linstor --controllers=192.168.1.30,192.168.1.31,192.168.1.32 resource list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG30 | 7001 | InUse  | Ok    |   UpToDate | 2022-07-15 20:03:53 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG31 | 7001 | Unused | Ok    |   UpToDate | 2022-07-15 20:03:59 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG32 | 7001 | Unused | Ok    |   Diskless | 2022-07-15 20:03:51 |

[16:12 XCPNG31 ~]# linstor --controllers=192.168.1.30,192.168.1.31,192.168.1.32 resource list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG30 | 7001 | InUse  | Ok    |   UpToDate | 2022-07-15 20:03:53 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG31 | 7001 | Unused | Ok    |   UpToDate | 2022-07-15 20:03:59 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG32 | 7001 | Unused | Ok    |   Diskless | 2022-07-15 20:03:51 |

[16:14 XCPNG32 ~]# linstor --controllers=192.168.1.30,192.168.1.31,192.168.1.32 resource list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG30 | 7001 | InUse  | Ok    |   UpToDate | 2022-07-15 20:03:53 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG31 | 7001 | Unused | Ok    |   UpToDate | 2022-07-15 20:03:59 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG32 | 7001 | Unused | Ok    |   Diskless | 2022-07-15 20:03:51 |

                                            Volumes appear to be OK on 2 hosts and not present on the third - although with rep set as 2 I think that is expected?

                                            [16:14 XCPNG30 ~]# lvs | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
  xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f_00000 linstor_group                                      -wi-ao---- <40.10g                   

[16:15 XCPNG31 ~]# lvs | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
  xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f_00000 linstor_group                                      -wi-ao---- <40.10g                                    
  
[16:19 XCPNG32 ~]# lvs | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
No lines

                                            Although when running the lvs command on each host I am getting a lot of warnings about DRDB volumes - these seem to be volumes that were previously deleted but not cleaned up fully:

                                              /dev/drbd1024: open failed: Wrong medium type
  /dev/drbd1026: open failed: Wrong medium type
  /dev/drbd1028: open failed: Wrong medium type
  /dev/drbd1000: open failed: Wrong medium type
  /dev/drbd1002: open failed: Wrong medium type
  /dev/drbd1012: open failed: Wrong medium type
  /dev/drbd1014: open failed: Wrong medium type
  /dev/drbd1016: open failed: Wrong medium type
  /dev/drbd1018: open failed: Wrong medium type
  /dev/drbd1020: open failed: Wrong medium type
  /dev/drbd1022: open failed: Wrong medium type
                                            ronan-aR 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post