Re: XOSTOR SR_BACKEND_FAILURE_78 VDI Creation failed

Our third host, which was the Supermicro host, was the XOSTOR controller. I tried migrating all VMs off that host. I could not get one VM off it that showed one disk on XOSTOR and one disk on local storage. After reboot, it seemed promising I could move the VM off the XOSTOR. The job started and was progressing percentage wise. It failed.

Now the VM has no VDIs attached to it so it's diskless.

I found the second VDI still on the 3rd host's local boot disk, which is where it said it was before the migration. The first 30gb disk that was showing on XOSTOR, is now 100% MIA. Just disappeared and missing. Luckily this VM was migrated from VMware and we still have a copy there.

Is this possibly a hardware compatibility issues? I'm not really sure how a vdi can one second be there and then not really seem to be there at all anymore. Makes me think it wasn't actually there to begin with. I'm starting to get a bit discouraged with the issues we have encountered. The logs I have dug through don't seem to help me, but at the same time I don't know exactly what I am looking for and where.

I'm not sure if I should just abandon this idea and move to testing NFS, even though I have never seen NFS be very performative even on VMware backed by the same exact storage array hosting ISCSI. We have always seen a latency increase for NFS, about double that of ISCSI.

vm.migrate
{
  "vm": "5ed47a71-74be-4df7-5982-a80f6cc71980",
  "migrationNetwork": "65a61b2c-d144-d75c-a3e4-e6b1151fdfeb",
  "sr": "176ab84d-368e-17b6-6107-d1dd8362da9d",
  "targetHost": "aef32e81-ee83-4ada-98a1-07ee3d201977"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Storage_error ([S(Does_not_exist);[S(mirror);S(2b2a0552-8ba0-3c17-5215-1e764460d88d/1cea5d10-ac1d-4e43-b582-433205fe391c)]])"
  ],
  "task": {
    "uuid": "14b6f743-9e98-74ff-bc5a-97729785d3fb",
    "name_label": "Async.VM.migrate_send",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20240605T20:46:10Z",
    "finished": "20240605T21:11:14Z",
    "status": "failure",
    "resident_on": "OpaqueRef:58de225e-1ef7-46a3-85d1-31012e174220",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Storage_error ([S(Does_not_exist);[S(mirror);S(2b2a0552-8ba0-3c17-5215-1e764460d88d/1cea5d10-ac1d-4e43-b582-433205fe391c)]])"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 6378))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 134))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1210))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2279))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "INTERNAL_ERROR(Storage_error ([S(Does_not_exist);[S(mirror);S(2b2a0552-8ba0-3c17-5215-1e764460d88d/1cea5d10-ac1d-4e43-b582-433205fe391c)]]))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Storage_error ([S(Does_not_exist);[S(mirror);S(2b2a0552-8ba0-3c17-5215-1e764460d88d/1cea5d10-ac1d-4e43-b582-433205fe391c)]]))
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1033:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1067:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1057:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1230:14)"
}

Thanks for any suggestions and help.

It took me a bit to setup XOSTOR without configuration issues. I am looking for advice on what to look into next. Thanks in advance! I am honestly thinking about a fresh install of everything now that we have ironed a ton of things out and understand better every step we need to take.

XOA 5.95.0
XCP-NG 8.2.1
We don't have 100% control over the hardware IBM Cloud provides us so we ended up with:
2x Lenovo, (,ThinkSystem SR630 -[7X02]-,)
1x Supermicro, (,SYS-6019U-TN4R4T,)
First SSD is boot only, next 3 960gb SSDs were configured in XOSTOR.
Does not appear they have the same MEGARAID card in all these, even though that shouldn't matter a whole lot since they should be just running in passthrough with no actual raid configured. I guess there could be some firmware issues berried here.

Before I setup XOSTO, I assigned an IP to use as a dedicated storage network. The UI would not let me set this, I believe because it was throwing a error because IPv6 was null. Weirdly enough after I got XOSTOR configured I went back and removed the gateway off these interfaces with the UI no problem.
#xe pif-list host-name-label=hostname
#xe pif-reconfigure-ip mode=static IP=ip netmask=nm gateway=gw uuid=uuid
#xe pif-param-set disallow-unplug=true uuid=uuid
#xe pif-param-set other-config:ManagementPurpose="Storage" uuid=uuid

After deploying XOSTOR with the UI, node 002 in our cluster was displaying an SR-Patch failure message. Tried applying patches multiple times witch resulted in same error. I don't have this message anymore. I ignored this error an went straight to trying to migrate VMs. Then was getting these Backend Failures. Went back to host002 and rebooted it. Patch error went away from this host. And after reboot, looked like I could actually migrate my first disk. Then the second disk failed. Now all I am getting is this same SR_BACKENG_FAILURE_78 VDI Creation failed error. Along the way there has been many toolstack restarts lol.

vdi.migrate
{
  "id": "84e50307-6d27-4b53-a886-756a470d343e",
  "sr_id": "3f544ac8-0e9b-3010-2298-0f69c660d759"
}
{
  "code": "SR_BACKEND_FAILURE_78",
  "params": [
    "",
    "VDI Creation failed [opterr=error cmd: `['/usr/bin/vhd-util', 'create', '--debug', '-n', '/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0', '-s', '92160', '-S', '2097152']`, code: `30`, reason: `/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0: failed to create: -30` (openers: {'xhyper002*': {}, 'xhyper003': {u'2020': {u'open-duration': u'3857', u'process-name': u'multipathd'}}, 'xhyper001': {}})]",
    ""
  ],
  "task": {
    "uuid": "3dbf2cb7-e3d2-5063-f3e1-9c158e4b13f7",
    "name_label": "Async.VDI.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20240604T20:49:32Z",
    "finished": "20240604T20:49:48Z",
    "status": "failure",
    "resident_on": "OpaqueRef:58de225e-1ef7-46a3-85d1-31012e174220",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "SR_BACKEND_FAILURE_78",
      "",
      "VDI Creation failed [opterr=error cmd: `['/usr/bin/vhd-util', 'create', '--debug', '-n', '/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0', '-s', '92160', '-S', '2097152']`, code: `30`, reason: `/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0: failed to create: -30` (openers: {'xhyper002': {}, 'xhyper003': {u'2020': {u'open-duration': u'3857', u'process-name': u'multipathd'}}, 'xhyper001': {}})]",
      ""
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 12359))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 134))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "SR_BACKEND_FAILURE_78(, VDI Creation failed [opterr=error cmd: `['/usr/bin/vhd-util', 'create', '--debug', '-n', '/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0', '-s', '92160', '-S', '2097152']`, code: `30`, reason: `/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0: failed to create: -30` (openers: {'xhyper002': {}, 'xhyper003': {u'2020': {u'open-duration': u'3857', u'process-name': u'multipathd'}}, 'xhyper001: {}})], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_78(, VDI Creation failed [opterr=error cmd: `['/usr/bin/vhd-util', 'create', '--debug', '-n', '/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0', '-s', '92160', '-S', '2097152']`, code: `30`, reason: `/dev/drbd/by-res/xcp-volume-ae71e4c5-7741-434e-97da-775eac185294/0: failed to create: -30` (openers: {'xhyper002': {}, 'xhyper003': {u'2020': {u'open-duration': u'3857', u'process-name': u'multipathd'}}, 'xhyper001': {}})], )
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1033:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1067:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1057:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1230:14)"
}