geoffbland

geoffbland

ronan-a said in XOSTOR hyperconvergence preview:

The outdated flag is removed automatically after a short delay if there is no issue with the network.
See: https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-outdate
Do you still have this flag?

Sorry about the long delay in this response - unfortunately I have been busy with work and so not able to spend much time looking at this. But two weeks later after the Outdated volume is still present. As far as I can tell there was no issue with the network.

I wiped the install again and could get DRDB in the same state again by creating a few VMs each with several disks and then deleting the VMs - eventually the issue occurs again.

I have since wiped again and done a fresh XCPNG install - this time with a dedicated network (separate NICs and private switch) for data and I'll see how that goes.

geoffbland

ronan-a said in [XOSTOR hyperconvergence preview]

Like I said in my e-mail, maybe there are more details in another log file. I hope.

In the end I realised I was more trying to "use" XOSTOR whilst testing rather than properly test it. So I decided to rip it all down and start again and retest it again - this time properly recording each step so any issues can be replicated. I will let you know how this goes.

geoffbland

Just a note to say how well-designed the XO user interface is. No UI is perfect but XO does seem very well designed, things kind of just work the way you expect them to - that sometimes seems to be hard to achieve.

I just noticed a nice touch today; if you restore a VM from backup it automatically gets given a "restored from backup" tag. To me, things like this show that someone is thinking carefully about how the UI is used.

geoffbland

stevezemlicka Thanks for mentioning this and reporting your "mistake" on the forums. I had been banging my head against the wall for a day before finding this post and realising the obvious.

geoffbland

ronan-a said in XOSTOR hyperconvergence preview:

The outdated flag is removed automatically after a short delay if there is no issue with the network.
See: https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-outdate
Do you still have this flag?

Sorry about the long delay in this response - unfortunately I have been busy with work and so not able to spend much time looking at this. But two weeks later after the Outdated volume is still present. As far as I can tell there was no issue with the network.

I wiped the install again and could get DRDB in the same state again by creating a few VMs each with several disks and then deleting the VMs - eventually the issue occurs again.

I have since wiped again and done a fresh XCPNG install - this time with a dedicated network (separate NICs and private switch) for data and I'll see how that goes.

geoffbland

ronan-a said in XOSTOR hyperconvergence preview:

You can forget the VDI to remove the VM

I couldn't forget it as the VM needs to be started to forget it and the VM is stuck in a "paused" state.

I was eventually able to get the VM in a stopped state by force rebooting all the hosts in the pool. Once the VM was stopped by this I was then able to delete the VM and all XOSTOR disks were then also removed.

Do you always have this issue when you create new VMs?
Yes, I got this error anytime I try to create a new VM on the XOSTOR SR. However after rebooting all the hosts in the pool I am able to recreate VMs again.

I will continue with more testing as and when I get time. Currently I have a VM up and running and seemingly healthy yet linstor reports the volume as outdated, what would cause this and how do I fix it?

┊ XCPNG30 ┊ xcp-volume-9163fab8-a449-439d-a599-05b8b2fa27bf ┊ DfltDisklessStorPool ┊     0 ┊    1002 ┊ /dev/drbd1002 ┊           ┊ InUse  ┊ Diskless ┊
┊ XCPNG31 ┊ xcp-volume-9163fab8-a449-439d-a599-05b8b2fa27bf ┊ xcp-sr-linstor_group ┊     0 ┊    1002 ┊ /dev/drbd1002 ┊ 20.05 GiB ┊ Unused ┊ UpToDate ┊
┊ XCPNG32 ┊ xcp-volume-9163fab8-a449-439d-a599-05b8b2fa27bf ┊ xcp-sr-linstor_group ┊     0 ┊    1002 ┊ /dev/drbd1002 ┊ 20.05 GiB ┊ Unused ┊ Outdated ┊

┊ XCPNG30 ┊ COMBINED ┊ 192.168.1.30:3366 (PLAIN) ┊ Online ┊
┊ XCPNG31 ┊ COMBINED ┊ 192.168.1.31:3366 (PLAIN) ┊ Online ┊
┊ XCPNG32 ┊ COMBINED ┊ 192.168.1.32:3366 (PLAIN) ┊ Online ┊

geoffbland

ronan-a In my latest test I created a new VM with multiple disks on XOSTOR. This worked OK and I was able to run and access all the disks.

However I then tried to remove this VM. After a long period of nothing happening (other than the spinning icon on the remove button) I get a "operation timed out" error and the VM is now shown as paused again.

vm.delete
{
  "id": "90613dbb-bd40-8082-c227-a318cbdbd01d"
}
{
  "call": {
    "method": "VM.hard_shutdown",
    "params": [
      "OpaqueRef:8aa8abb0-d204-43fd-897f-04425b790e68"
    ]
  },
  "message": "operation timed out",
  "name": "TimeoutError",
  "stack": "TimeoutError: operation timed out
    at Promise.call (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/timeout.js:11:16)
    at Xapi.apply (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/index.js:693:37)
    at Xapi._call (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/limit-concurrency-decorator/src/index.js:85:24)
    at /opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/index.js:771:21
    at loopResolver (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:83:46)
    at Promise._execute (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/debuggability.js:384:9)
    at Promise._resolveFromExecutor (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:518:18)
    at new Promise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:103:10)
    at loop (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:85:22)
    at retry (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:87:10)
    at Xapi._sessionCall (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/index.js:762:20)
    at Xapi.call (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/index.js:273:14)
    at loopResolver (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:83:46)
    at Promise._execute (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/debuggability.js:384:9)
    at Promise._resolveFromExecutor (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:518:18)
    at new Promise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:103:10)
    at loop (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:85:22)
    at Xapi.retry (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:87:10)
    at Xapi.call (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/promise-toolbox/retry.js:103:18)
    at Xapi.destroy (/opt/xo/xo-builds/xen-orchestra-202206111352/@xen-orchestra/xapi/vm.js:361:18)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202206111352/packages/xo-server/src/xo-mixins/api.mjs:310:20)"
}

If I try to delete again, the same thing happens.

All the volumes used by this VM still exist on linstor and linstor shows no errors.

Now when I try to create any new VM, this now also fails with the following error:

vm.create
{
  "clone": true,
  "existingDisks": {},
  "installation": {
    "method": "cdrom",
    "repository": "16ead07f-2f23-438f-9010-6f1e6c847e2c"
  },
  "name_label": "testx",
  "template": "d276dc0c-3870-2b7e-64c2-b612bb856227-2cf37285-57bc-4633-a24f-0c6c825dda66",
  "VDIs": [
    {
      "bootable": true,
      "device": "0",
      "size": 23622320128,
      "type": "system",
      "SR": "141d63f6-d3ed-4a2f-588a-1835f0cea588",
      "name_description": "testx_vdi",
      "name_label": "testx_xostor_vdi"
    }
  ],
  "VIFs": [
    {
      "network": "965db545-28a2-5daf-1c90-0ae9a7882bc1",
      "allowedIpv4Addresses": [],
      "allowedIpv6Addresses": []
    }
  ],
  "CPUs": "4",
  "cpusMax": 4,
  "cpuWeight": null,
  "cpuCap": null,
  "name_description": "testx",
  "memory": 4294967296,
  "bootAfterCreate": true,
  "copyHostBiosStrings": false,
  "secureBoot": false,
  "share": false,
  "coreOs": false,
  "tags": [],
  "hvmBootFirmware": "bios"
}
{
  "code": "SR_BACKEND_FAILURE_78",
  "params": [
    "",
    "VDI Creation failed [opterr=error Invalid path, current=/dev/drbd1031, expected=/dev/drbd/by-res/xcp-volume-cc55faf8-84a0-431c-a2dc-a618d70e2c49/0 (realpath=/dev/drbd/by-res/xcp-volume-cc55faf8-84a0-431c-a2dc-a618d70e2c49/0)]",
    ""
  ],
  "call": {
    "method": "VDI.create",
    "params": [
      {
        "name_description": "testx_vdi",
        "name_label": "testx_xostor_vdi",
        "other_config": {},
        "read_only": false,
        "sharable": false,
        "SR": "OpaqueRef:7709e595-7889-4cf1-8980-c04bd145d296",
        "type": "user",
        "virtual_size": 23622320128
      }
    ]
  },
  "message": "SR_BACKEND_FAILURE_78(, VDI Creation failed [opterr=error Invalid path, current=/dev/drbd1031, expected=/dev/drbd/by-res/xcp-volume-cc55faf8-84a0-431c-a2dc-a618d70e2c49/0 (realpath=/dev/drbd/by-res/xcp-volume-cc55faf8-84a0-431c-a2dc-a618d70e2c49/0)], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_78(, VDI Creation failed [opterr=error Invalid path, current=/dev/drbd1031, expected=/dev/drbd/by-res/xcp-volume-cc55faf8-84a0-431c-a2dc-a618d70e2c49/0 (realpath=/dev/drbd/by-res/xcp-volume-cc55faf8-84a0-431c-a2dc-a618d70e2c49/0)], )
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.callbackTrampoline (internal/async_hooks.js:130:17)"
}

Note /dev/drbd1031 does not exist in /dev/drdb or as a volume.

How do I remove the test VM? How to fix the issue with creating new VMs?

geoffbland

ronan-a The volumes are reachable on all nodes:

[16:13 XCPNG30 ~]# linstor --controllers=192.168.1.30,192.168.1.31,192.168.1.32 resource list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG30 | 7001 | InUse  | Ok    |   UpToDate | 2022-07-15 20:03:53 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG31 | 7001 | Unused | Ok    |   UpToDate | 2022-07-15 20:03:59 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG32 | 7001 | Unused | Ok    |   Diskless | 2022-07-15 20:03:51 |

[16:12 XCPNG31 ~]# linstor --controllers=192.168.1.30,192.168.1.31,192.168.1.32 resource list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG30 | 7001 | InUse  | Ok    |   UpToDate | 2022-07-15 20:03:53 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG31 | 7001 | Unused | Ok    |   UpToDate | 2022-07-15 20:03:59 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG32 | 7001 | Unused | Ok    |   Diskless | 2022-07-15 20:03:51 |

[16:14 XCPNG32 ~]# linstor --controllers=192.168.1.30,192.168.1.31,192.168.1.32 resource list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG30 | 7001 | InUse  | Ok    |   UpToDate | 2022-07-15 20:03:53 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG31 | 7001 | Unused | Ok    |   UpToDate | 2022-07-15 20:03:59 |
| xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | XCPNG32 | 7001 | Unused | Ok    |   Diskless | 2022-07-15 20:03:51 |

Volumes appear to be OK on 2 hosts and not present on the third - although with rep set as 2 I think that is expected?

[16:14 XCPNG30 ~]# lvs | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
  xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f_00000 linstor_group                                      -wi-ao---- <40.10g                   

[16:15 XCPNG31 ~]# lvs | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
  xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f_00000 linstor_group                                      -wi-ao---- <40.10g                                    
  
[16:19 XCPNG32 ~]# lvs | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
No lines

Although when running the lvs command on each host I am getting a lot of warnings about DRDB volumes - these seem to be volumes that were previously deleted but not cleaned up fully:

  /dev/drbd1024: open failed: Wrong medium type
  /dev/drbd1026: open failed: Wrong medium type
  /dev/drbd1028: open failed: Wrong medium type
  /dev/drbd1000: open failed: Wrong medium type
  /dev/drbd1002: open failed: Wrong medium type
  /dev/drbd1012: open failed: Wrong medium type
  /dev/drbd1014: open failed: Wrong medium type
  /dev/drbd1016: open failed: Wrong medium type
  /dev/drbd1018: open failed: Wrong medium type
  /dev/drbd1020: open failed: Wrong medium type
  /dev/drbd1022: open failed: Wrong medium type

geoffbland

ronan-a

As promised I have done some more "organised" testing, with a brand new cluster set up to test XOSTOR. Early simple tests seemed to be OK and I can create, restart, snapshot, move and delete VMs with no problem.

But then after putting a VM under load for an hour and then restarting it I am seeing the same weird behaviour I saw previously with XOSTOR.

Firstly, the VM took far longer than expected to shutdown. Then trying to restart the VM fails.

Let me know if you want me to provide logs or any specific testing.

vm.start
{
  "id": "ade699f2-42f0-8629-35ea-6fcc69de99d7",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "SR_BACKEND_FAILURE_1200",
  "params": [
    "",
    "No such Tapdisk(minor=12)",
    ""
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8",
      false,
      false
    ]
  },
  "message": "SR_BACKEND_FAILURE_1200(, No such Tapdisk(minor=12), )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_1200(, No such Tapdisk(minor=12), )
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.callbackTrampoline (internal/async_hooks.js:130:17)"
}

The linstore volume for this VM appears to be OK (by the way I hope eventually we have an easy way to match up VMs to Linstore volumes)

[16:53 XCPNG31 ~]# linstor volume list | grep xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f
| XCPNG30 | xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | xcp-sr-linstor_group |     0 |    1001 | /dev/drbd1001 | 40.10 GiB | InUse  |   UpToDate |
| XCPNG31 | xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | xcp-sr-linstor_group |     0 |    1001 | /dev/drbd1001 | 40.10 GiB | Unused |   UpToDate |
| XCPNG32 | xcp-volume-00b34ae3-2ad3-44ea-aa13-d5de1fbf756f | DfltDisklessStorPool |     0 |    1001 | /dev/drbd1001 |           | Unused |   Diskless |

[16:59 XCPNG31 ~]# linstor node list
╭─────────────────────────────────────────────────────────╮
┊ Node    ┊ NodeType ┊ Addresses                 ┊ State  ┊
╞═════════════════════════════════════════════════════════╡
┊ XCPNG30 ┊ COMBINED ┊ 192.168.1.30:3366 (PLAIN) ┊ Online ┊
┊ XCPNG31 ┊ COMBINED ┊ 192.168.1.31:3366 (PLAIN) ┊ Online ┊
┊ XCPNG32 ┊ COMBINED ┊ 192.168.1.32:3366 (PLAIN) ┊ Online ┊
╰─────────────────────────────────────────────────────────╯
[17:13 XCPNG31 ~]# linstor storage-pool list
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node    ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ XCPNG30 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ XCPNG31 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ XCPNG32 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ xcp-sr-linstor_group ┊ XCPNG30 ┊ LVM      ┊ linstor_group ┊     3.50 TiB ┊      3.64 TiB ┊ False        ┊ Ok    ┊            ┊
┊ xcp-sr-linstor_group ┊ XCPNG31 ┊ LVM      ┊ linstor_group ┊     3.49 TiB ┊      3.64 TiB ┊ False        ┊ Ok    ┊            ┊
┊ xcp-sr-linstor_group ┊ XCPNG32 ┊ LVM      ┊ linstor_group ┊     3.50 TiB ┊      3.64 TiB ┊ False        ┊ Ok    ┊            ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I am seeing the following errors in the SMLog (bb027c9a-5655-4f93-9090-e76b34b2c90d is the disk for this VM)

Jul 23 16:48:30 XCPNG30 SM: [22961] lock: opening lock file /var/lock/sm/bb027c9a-5655-4f93-9090-e76b34b2c90d/vdi
Jul 23 16:48:30 XCPNG30 SM: [22961] blktap2.deactivate
Jul 23 16:48:30 XCPNG30 SM: [22961] lock: acquired /var/lock/sm/bb027c9a-5655-4f93-9090-e76b34b2c90d/vdi
Jul 23 16:48:30 XCPNG30 SM: [22961] ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30']
Jul 23 16:49:00 XCPNG30 SM: [22961]  = 5
Jul 23 16:49:00 XCPNG30 SM: [22961] ***** BLKTAP2:<function _deactivate_locked at 0x7f6fb33208c0>: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30'] failed: status=5, pid=22983, errmsg=Input/output error
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 85, in wrapper
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = op(self, *args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1744, in _deactivate_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._deactivate(sr_uuid, vdi_uuid, caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1785, in _deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._tap_deactivate(minor)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1368, in _tap_deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapdisk.shutdown()
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 880, in shutdown
Jul 23 16:49:00 XCPNG30 SM: [22961]     TapCtl.close(self.pid, self.minor, force)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 433, in close
Jul 23 16:49:00 XCPNG30 SM: [22961]     cls._pread(args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 296, in _pread
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapctl._wait(quiet)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 285, in _wait
Jul 23 16:49:00 XCPNG30 SM: [22961]     raise self.CommandFailure(self.cmd, **info)
Jul 23 16:49:00 XCPNG30 SM: [22961]
Jul 23 16:49:00 XCPNG30 SM: [22961] lock: released /var/lock/sm/bb027c9a-5655-4f93-9090-e76b34b2c90d/vdi
Jul 23 16:49:00 XCPNG30 SM: [22961] call-plugin on 7aaaf4a5-0e43-442e-a9b1-38620c87fd69 (linstor-manager:lockVdi with {'groupName': 'linstor_group', 'srUuid': '141d63f6-d3ed-4a2f-588a-1835f0cea588', 'vdiUuid': 'bb027c9a-5655-4f93-9090-e76b34b2c90d', 'locked': 'False'}) returned: True
Jul 23 16:49:00 XCPNG30 SM: [22961] ***** generic exception: vdi_deactivate: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30'] failed: status=5, pid=22983, errmsg=Input/output error
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 23 16:49:00 XCPNG30 SM: [22961]     return self._run_locked(sr)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     rv = self._run(sr, target)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 274, in _run
Jul 23 16:49:00 XCPNG30 SM: [22961]     caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1729, in deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     if self._deactivate_locked(sr_uuid, vdi_uuid, caching_params):
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 85, in wrapper
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = op(self, *args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1744, in _deactivate_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._deactivate(sr_uuid, vdi_uuid, caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1785, in _deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._tap_deactivate(minor)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1368, in _tap_deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapdisk.shutdown()
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 880, in shutdown
Jul 23 16:49:00 XCPNG30 SM: [22961]     TapCtl.close(self.pid, self.minor, force)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 433, in close
Jul 23 16:49:00 XCPNG30 SM: [22961]     cls._pread(args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 296, in _pread
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapctl._wait(quiet)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 285, in _wait
Jul 23 16:49:00 XCPNG30 SM: [22961]     raise self.CommandFailure(self.cmd, **info)
Jul 23 16:49:00 XCPNG30 SM: [22961]
Jul 23 16:49:00 XCPNG30 SM: [22961] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'close', '-p', '19527', '-m', '2', '-t', '30'] failed: status=5, pid=22983, errmsg=Input/output error
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = cmd.run(sr)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 23 16:49:00 XCPNG30 SM: [22961]     return self._run_locked(sr)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     rv = self._run(sr, target)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/SRCommand.py", line 274, in _run
Jul 23 16:49:00 XCPNG30 SM: [22961]     caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1729, in deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     if self._deactivate_locked(sr_uuid, vdi_uuid, caching_params):
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 85, in wrapper
Jul 23 16:49:00 XCPNG30 SM: [22961]     ret = op(self, *args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1744, in _deactivate_locked
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._deactivate(sr_uuid, vdi_uuid, caching_params)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1785, in _deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     self._tap_deactivate(minor)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 1368, in _tap_deactivate
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapdisk.shutdown()
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 880, in shutdown
Jul 23 16:49:00 XCPNG30 SM: [22961]     TapCtl.close(self.pid, self.minor, force)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 433, in close
Jul 23 16:49:00 XCPNG30 SM: [22961]     cls._pread(args)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 296, in _pread
Jul 23 16:49:00 XCPNG30 SM: [22961]     tapctl._wait(quiet)
Jul 23 16:49:00 XCPNG30 SM: [22961]   File "/opt/xensource/sm/blktap2.py", line 285, in _wait
Jul 23 16:49:00 XCPNG30 SM: [22961]     raise self.CommandFailure(self.cmd, **info)
Jul 23 16:49:00 XCPNG30 SM: [22961]

If I try to start this VM after the above issue I get this error:

vm.start
{
  "id": "ade699f2-42f0-8629-35ea-6fcc69de99d7",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "FAILED_TO_START_EMULATOR",
  "params": [
    "OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8",
    "domid 12",
    "QMP failure at File \"xc/device.ml\", line 3366, characters 71-78"
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8",
      false,
      false
    ]
  },
  "message": "FAILED_TO_START_EMULATOR(OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8, domid 12, QMP failure at File \"xc/device.ml\", line 3366, characters 71-78)",
  "name": "XapiError",
  "stack": "XapiError: FAILED_TO_START_EMULATOR(OpaqueRef:a60d0553-a2f2-41e6-9df4-fad745fbacc8, domid 12, QMP failure at File \"xc/device.ml\", line 3366, characters 71-78)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202206111352/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202206111352/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.callbackTrampoline (internal/async_hooks.js:130:17)"
}

This VM is set up the same way as several other VMs I have running on another cluster but using SRs on NFS mounts.

This is with latest XCP-ng version, all servers patched and up-to-date.

yum update
...
No packages marked for update

geoffbland

ronan-a said in XOSTOR hyperconvergence preview:

Yes, you can. Please take a look to: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-managing_network_interface_cards
To get the storage pool name, execute this command in your pool:

OK - so to set up this XOSTOR SR I would first

xe sr-create type=linstor name-label=XOSTOR01 host-uuid=xxx device-config:hosts=<XCPNG Host Names> ...etc...

to create the Linstor storage-pool. Then

linstor storage-pool list

to get the name the pool. Then on each node found in device-config:hosts=<XCPNG Host Names> run the following command:

linstor storage-pool set-property <host/node_name> <pool_name> PrefNic <nic_name>

where nic_name is the name of the Linstor interface created for the specific NIC.

geoffbland

Doing some more testing on XOSTOR and starting from scratch again. A brand-new XCP-ng installation made onto a new 3 server pool, each server with a blank 4GB disk.
All servers have xcpng patched up to date.
Then I installed XOSTOR onto each of these servers, Linstor installed OK and I have the linstor group on each.

[19:22 XCPNG30 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize    VFree
  VG_XenStorage-a776b6b1-9a96-e179-ea12-f2419ae512b6   1   1   0 wz--n- <405.62g 405.61g
  linstor_group                                        1   0   0 wz--n-   <3.64t  <3.64t
[19:22 XCPNG30 ~]# rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
xha-10.1.0-2.2.0.linstor.2.xcpng8.2.x86_64
sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64
sm-rawhba-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64

[19:21 XCPNG31 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize    VFree
  VG_XenStorage-f75785ef-df30-b54c-2af4-84d19c966453   1   1   0 wz--n- <405.62g 405.61g
  linstor_group                                        1   0   0 wz--n-   <3.64t  <3.64t
[19:21 XCPNG31 ~]# rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
xha-10.1.0-2.2.0.linstor.2.xcpng8.2.x86_64
sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64
sm-rawhba-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64

[19:23 XCPNG32 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize    VFree
  VG_XenStorage-abaf8356-fc58-9124-a23b-c29e7e67c983   1   1   0 wz--n- <405.62g 405.61g
  linstor_group                                        1   0   0 wz--n-   <3.64t  <3.64t
[19:23 XCPNG32 ~]# rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
xha-10.1.0-2.2.0.linstor.2.xcpng8.2.x86_64
sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64
sm-rawhba-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64

[19:26 XCPNG31 ~]# xe host-list
uuid ( RO)                : 7c3f2fae-0456-4155-a9ad-43790fcb4155
          name-label ( RW): XCPNG32
    name-description ( RW): Default install

uuid ( RO)                : 2e48b46a-c420-4957-9233-3e029ea39305
          name-label ( RW): XCPNG30
    name-description ( RW): Default install

uuid ( RO)                : 7aaaf4a5-0e43-442e-a9b1-38620c87fd69
          name-label ( RW): XCPNG31
    name-description ( RW): Default install

But I am not able to create the SR.

xe sr-create type=linstor name-label=XOSTOR01 host-uuid=7aaaf4a5-0e43-442e-a9b1-38620c87fd69 device-config:hosts=xcpng30,xcpng31,xcpng32 device-config:group-name=linstor_group device-config:redundancy=2 shared=true device-config:provisioning=thick

This gives the following error:

Error code: SR_BACKEND_FAILURE_5006
Error parameters: , LINSTOR SR creation error [opterr=Not enough online hosts],

Here's the error in the SMLog

Jul 15 19:29:22 XCPNG31 SM: [9747] sr_create {'sr_uuid': '14aa2b8b-430f-34e5-fb74-c37667cb18ec', 'subtask_of': 'DummyRef:|d39839f1-ee3a-4bfe-8a41-7a077f4f2640|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:196f738d-24fa-4598-8e96-4a13390abc87', 'session_ref': 'OpaqueRef:e806b347-1e5f-4644-842f-26a7b06b2561', 'device_config': {'group-name': 'linstor_group', 'redundancy': '2', 'hosts': 'xcpng30,xcpng31,xcpng32', 'SRmaster': 'true', 'provisioning': 'thick'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:7ded7feb-729f-47c3-9893-1b62db0b7e17'}
Jul 15 19:29:22 XCPNG31 SM: [9747] LinstorSR.create for 14aa2b8b-430f-34e5-fb74-c37667cb18ec
Jul 15 19:29:22 XCPNG31 SM: [9747] Raising exception [5006, LINSTOR SR creation error [opterr=Not enough online hosts]]
Jul 15 19:29:22 XCPNG31 SM: [9747] lock: released /var/lock/sm/14aa2b8b-430f-34e5-fb74-c37667cb18ec/sr
Jul 15 19:29:22 XCPNG31 SM: [9747] ***** generic exception: sr_create: EXCEPTION <class 'SR.SROSError'>, LINSTOR SR creation error [opterr=Not enough online hosts]
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return self._run_locked(sr)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 15 19:29:22 XCPNG31 SM: [9747]     rv = self._run(sr, target)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 612, in wrap
Jul 15 19:29:22 XCPNG31 SM: [9747]     return load(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 597, in load
Jul 15 19:29:22 XCPNG31 SM: [9747]     return wrapped_method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 443, in wrapped_method
Jul 15 19:29:22 XCPNG31 SM: [9747]     return method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 688, in create
Jul 15 19:29:22 XCPNG31 SM: [9747]     opterr='Not enough online hosts'
Jul 15 19:29:22 XCPNG31 SM: [9747]
Jul 15 19:29:22 XCPNG31 SM: [9747] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'SR.SROSError'>, LINSTOR SR creation error [opterr=Not enough online hosts]
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Jul 15 19:29:22 XCPNG31 SM: [9747]     ret = cmd.run(sr)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return self._run_locked(sr)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jul 15 19:29:22 XCPNG31 SM: [9747]     rv = self._run(sr, target)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
Jul 15 19:29:22 XCPNG31 SM: [9747]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 612, in wrap
Jul 15 19:29:22 XCPNG31 SM: [9747]     return load(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 597, in load
Jul 15 19:29:22 XCPNG31 SM: [9747]     return wrapped_method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 443, in wrapped_method
Jul 15 19:29:22 XCPNG31 SM: [9747]     return method(self, *args, **kwargs)
Jul 15 19:29:22 XCPNG31 SM: [9747]   File "/opt/xensource/sm/LinstorSR", line 688, in create
Jul 15 19:29:22 XCPNG31 SM: [9747]     opterr='Not enough online hosts'

I have found the issue - the device-config:hosts list is case-sensitive, if the hosts are given in lower-case the above error occurs. Specifying the hosts in upper-case works.

Also using a fully-qualified name for the host fails - regardless of the case used.

geoffbland

Is it possible to use a separate network for the XOSTOR/Linstor disk replication from the "main" network used for XCP-ng servers?

If so when the SR is created with this command:

xe sr-create type=linstor name-label=XOSTOR host-uuid=bc3cd3af-3f09-48cf-ae55-515ba21930f5 device-config:hosts=host-a,host-b,host-c,host-d device-config:group-name=linstor_group/thin_device device-config:redundancy=4 shared=true device-config:provisioning=thin

Do the device-config:hosts need to be XCP-ng hosts - or can IP address of the "data-replication" network be provided here.

For example, my XCP-NG servers have dual NICs, I could use the second NIC on a private network/switch with a different subnet to the "main" hosts and use this solely for XOSTOR/Linstor disk replication. Is this possible?

geoffbland

Could you install your XO VM on a completely different host/hypervisor - for example, on Hyper-V on you PC?

I have one XO VM running on my XCP-NG pool and another XO VM running on an UNRAID server - just for these kind of situations.

geoffbland

mauzilla said in XCP / XO and Truenas Scale or Core:

I opted to install Scale today but as murphy would have it, I did not know of the caveat of raid controller support only to find this just as I was about to setup the NFS mount. The servers I am using is Dell R720DX with H710 raid controller, and my understanding is that it's a no-go due ZFS and performance impact / possibility of data loss when using a raid controller. It's a bit of a logistical nightmare as I also don't feel comfortable in flashing vendor hardware with non-vendor firmware (just doesn't feel right for something meant to be in production) and I am 800km away from the cabinet so i've opted to take a chance until I find a suitable HBA card

FWIW I too have a R720 with a H710P RAID card. I was able to follow this guide https://fohdeesha.com/docs/perc.html to flash the card into "IT mode" and access the disks as HBA. It worked flawlessly and was fairly easy and quick to do. You can flash it back to RAID if needed.

My tests showed TrueNAS Scale worked well with the H710P in IT mode, seeing all the individual disks with no issue.

Eventually I decided to drop back to TrueNAS Core due to issues unrelated to the controller or disks.

geoffbland

@geoffbland

Best posts made by geoffbland

Latest posts made by geoffbland