XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XOSTOR hyperconvergence preview

    Scheduled Pinned Locked Moved XOSTOR
    446 Posts 47 Posters 479.8k Views 48 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ronan-aR Offline
      ronan-a Vates 🪐 XCP-ng Team @geoffbland
      last edited by

      @geoffbland The entry for issues is this repo: https://github.com/xcp-ng/xcp
      The sm repo is used for the pull requests.

      1 Reply Last reply Reply Quote 0
      • G Offline
        geoffbland
        last edited by geoffbland

        First tests with XOSAN with newly created VMs have been good.

        I'm now trying to migrate some existing VMs from NFS (TrueNAS) to XOSAN to test "active" VMs.

        With the VM running - pressing the Migrate VDI button on the Disks tab, pauses the VM as expected but when the VM restarts the VDI is still on the original disk. The VDI has not been migrated to XOSAN.

        If I first stop the VM and then press the Migrate VDI button on the Disks tab, I then do get an error.

        vdi.migrate
        {
          "id": "8a3520ad-328f-4515-b547-2fb283edbd91",
          "sr_id": "cf896912-cd71-d2b2-488a-5792b7147c87"
        }
        {
          "code": "SR_BACKEND_FAILURE_46",
          "params": [
            "",
            "The VDI is not available [opterr=Could not load f1ca0b16-ce23-408a-b80e-xxxxxxxxxxxx because: No such file or directory]",
            ""
          ],
          "task": {
            "uuid": "8b3b47ee-4135-fea7-5f30-xxxxxxxxxxxx",
            "name_label": "Async.VDI.pool_migrate",
            "name_description": "",
            "allowed_operations": [],
            "current_operations": {},
            "created": "20220522T12:20:12Z",
            "finished": "20220522T12:20:54Z",
            "status": "failure",
            "resident_on": "OpaqueRef:a1e9a8f3-0a79-4824-b29f-d81b3246d190",
            "progress": 1,
            "type": "<none/>",
            "result": "",
            "error_info": [
              "SR_BACKEND_FAILURE_46",
              "",
              "The VDI is not available [opterr=Could not load f1ca0b16-ce23-408a-b80e-xxxxxxxxxxxx because: No such file or directory]",
              ""
            ],
            "other_config": {},
            "subtask_of": "OpaqueRef:NULL",
            "subtasks": [],
            "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 12325))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
          },
          "message": "SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Could not load f1ca0b16-ce23-408a-b80e-xxxxxxxxxxxx because: No such file or directory], )",
          "name": "XapiError",
          "stack": "XapiError: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Could not load f1ca0b16-ce23-408a-b80e-xxxxxxxxxxxx because: No such file or directory], )
            at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_XapiError.js:16:12)
            at _default (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_getTaskResult.js:11:29)
            at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:949:24)
            at forEach (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:983:14)
            at Array.forEach (<anonymous>)
            at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:973:12)
            at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:1139:14)"
        }
        

        Exporting the VDI from NFS and (re)importing as a VM on XOSTOR does work.

        I'm guessing this is not a problem with XOSTOR specifically but with XO or NFS, still I would like to work out what is causing migration and how to fix it?

        Also I have noticed that the underlying volume on XOSTOR/linstor that was created and started to be populated does get cleaned up when the migrate fails.

        This is using XO from the sources - updated fairly recently (commit 8ed84) and XO server 5.92.0.

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Online
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          It could be interesting to understand why the migration failed the first time. Is there absolutely no error during this first migration?

          G 1 Reply Last reply Reply Quote 0
          • G Offline
            geoffbland @olivierlambert
            last edited by

            @olivierlambert Thanks for the prompt response.

            I am pretty sure there was no error reported but as I cleared down the logs when I retried from on export/import I can't be 100% sure.

            So I tested migration on another VM to try and replicate this and this time migration worked OK.

            The only difference I can think of is that the failure occurred on a VM created quite a time ago - whilst the working VM had been created recently.

            I will do a few more tests and see if I can replicate this again.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Online
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Okay great šŸ™‚ If you can reproduce, that would be even better to try to do the migration with xe CLI, this way we remove more moving pieces in the middle šŸ™‚

              G 1 Reply Last reply Reply Quote 0
              • G Offline
                geoffbland @olivierlambert
                last edited by

                @olivierlambert Sorry, took some time to get around to this. But trying to migrate a VDI from an NFS store to XOSTOR is still failing most of the time. This is a VM that was created some time ago - it I do the same with the VDI of a a recently created VM the migration seems to work OK.

                >xe vm-disk-list vm=lb01
                Disk 0 VBD:
                uuid ( RO)             : d9d06048-6f91-1913-714d-xxxxxxxxaece
                    vm-name-label ( RO): lb01
                       userdevice ( RW): 0
                
                
                Disk 0 VDI:
                uuid ( RO)             : a38f27e8-c6a0-49d3-9fd3-xxxxxxxx10e3
                       name-label ( RW): lb01_tnc01_hdd
                    sr-name-label ( RO): XCPNG_VMs_TrueNAS
                     virtual-size ( RO): 10737418240
                
                >xe sr-list name-label=XOSTOR01
                uuid ( RO)                : cf896912-cd71-d2b2-488a-xxxxxxxx7c87
                          name-label ( RW): XOSTOR01
                    name-description ( RW):
                                host ( RO): <shared>
                                type ( RO): linstor
                        content-type ( RO):
                
                >xe vdi-pool-migrate uuid=a38f27e8-c6a0-49d3-9fd3-xxxxxxxx10e3 sr-uuid=cf896912-cd71-d2b2-488a-xxxxxxxx7c87
                
                Error code: SR_BACKEND_FAILURE_46
                Error parameters: , The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-xxxxxxxxec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']],
                

                Running this I see the VM pause as expected for a few minutes and then it just starts up again. VM is still running with no issues - it just did not move the VDI.

                What is the resource with UUID 735fc2d7-f1f0-4cc6-9d35-xxxxxxxxec6c that it is trying to find? That UUID does not match the VDI.

                The VDI must be OK as the VM is still up and running with no errors.

                As this is probably not an XOSTOR issue - should I raise a new topic for this?

                ronan-aR 1 Reply Last reply Reply Quote 0
                • olivierlambertO Online
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  It's hard to tell. If you can migrate between non-XOSTOR SRs and see if you reproduce, then it's another issue. If it's only happening when using XOSTOR in the loop, then it's relevant here šŸ™‚

                  1 Reply Last reply Reply Quote 0
                  • ronan-aR Offline
                    ronan-a Vates 🪐 XCP-ng Team @geoffbland
                    last edited by

                    @geoffbland I can't reproduce your problem, can you send me the SMlog of your hosts please? šŸ™‚

                    G 1 Reply Last reply Reply Quote 0
                    • G Offline
                      geoffbland @ronan-a
                      last edited by

                      @ronan-a said in XOSTOR hyperconvergence preview:

                      @geoffbland I can't reproduce your problem, can you send me the SMlog of your hosts please? šŸ™‚

                      As requested,

                      May 24 09:13:22 XCPNG01 SM: [18127] ['/usr/bin/vhd-util', 'query', '--debug', '-vsfp', '-n', '/dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-xxxxxxxx58c0/0']
                      May 24 09:13:23 XCPNG01 SM: [18127] FAILED in util.pread: (rc 2) stdout: '40960
                      May 24 09:13:23 XCPNG01 SM: [18127] 2048840192
                      May 24 09:13:23 XCPNG01 SM: [18127] query failed
                      May 24 09:13:23 XCPNG01 SM: [18127] hidden: 0
                      May 24 09:13:23 XCPNG01 SM: [18127] ', stderr: ''
                      May 24 09:13:23 XCPNG01 SM: [18127] linstor-manager:get_vhd_info error: No such file or directory
                      May 24 09:13:26 XCPNG01 SM: [18158] ['/usr/bin/vhd-util', 'query', '--debug', '-vsfp', '-n', '/dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-xxxxxxxx58c0/0']
                      May 24 09:13:26 XCPNG01 SM: [18158] FAILED in util.pread: (rc 2) stdout: '40960
                      May 24 09:13:26 XCPNG01 SM: [18158] 2048840192
                      May 24 09:13:26 XCPNG01 SM: [18158] query failed
                      May 24 09:13:26 XCPNG01 SM: [18158] hidden: 0
                      May 24 09:13:26 XCPNG01 SM: [18158] ', stderr: ''
                      May 24 09:13:26 XCPNG01 SM: [18158] linstor-manager:get_vhd_info error: No such file or directory
                      May 24 09:13:29 XCPNG01 SM: [18200] ['/usr/bin/vhd-util', 'query', '--debug', '-vsfp', '-n', '/dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-xxxxxxxx58c0/0']
                      May 24 09:13:29 XCPNG01 SM: [18200] FAILED in util.pread: (rc 2) stdout: '40960
                      May 24 09:13:29 XCPNG01 SM: [18200] 2048840192
                      May 24 09:13:29 XCPNG01 SM: [18200] query failed
                      May 24 09:13:29 XCPNG01 SM: [18200] hidden: 0
                      May 24 09:13:29 XCPNG01 SM: [18200] ', stderr: ''
                      May 24 09:13:29 XCPNG01 SM: [18200] linstor-manager:get_vhd_info error: No such file or directory
                      May 24 09:13:32 XCPNG01 SM: [18212] ['/usr/bin/vhd-util', 'query', '--debug', '-vsfp', '-n', '/dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-xxxxxxxx58c0/0']
                      May 24 09:13:32 XCPNG01 SM: [18212] FAILED in util.pread: (rc 2) stdout: '40960
                      May 24 09:13:32 XCPNG01 SM: [18212] 2048840192
                      May 24 09:13:32 XCPNG01 SM: [18212] query failed
                      May 24 09:13:32 XCPNG01 SM: [18212] hidden: 0
                      May 24 09:13:32 XCPNG01 SM: [18212] ', stderr: ''
                      May 24 09:13:32 XCPNG01 SM: [18212] linstor-manager:get_vhd_info error: No such file or directory
                      May 24 09:13:35 XCPNG01 SM: [18247] ['/usr/bin/vhd-util', 'query', '--debug', '-vsfp', '-n', '/dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-xxxxxxxx58c0/0']
                      May 24 09:13:35 XCPNG01 SM: [18247] FAILED in util.pread: (rc 2) stdout: '40960
                      May 24 09:13:35 XCPNG01 SM: [18247] 2048840192
                      May 24 09:13:35 XCPNG01 SM: [18247] query failed
                      May 24 09:13:35 XCPNG01 SM: [18247] hidden: 0
                      May 24 09:13:35 XCPNG01 SM: [18247] ', stderr: ''
                      May 24 09:13:35 XCPNG01 SM: [18247] linstor-manager:get_vhd_info error: No such file or directory
                      May 24 09:13:36 XCPNG01 SM: [18259] ['/usr/bin/vhd-util', 'query', '--debug', '-vsfp', '-n', '/dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-xxxxxxxx58c0/0']
                      May 24 09:13:36 XCPNG01 SM: [18259] FAILED in util.pread: (rc 2) stdout: '40960
                      May 24 09:13:36 XCPNG01 SM: [18259] 2048840192
                      May 24 09:13:36 XCPNG01 SM: [18259] query failed
                      May 24 09:13:36 XCPNG01 SM: [18259] hidden: 0
                      May 24 09:13:36 XCPNG01 SM: [18259] ', stderr: ''
                      May 24 09:13:36 XCPNG01 SM: [18259] linstor-manager:get_vhd_info error: No such file or directory
                      
                      1 Reply Last reply Reply Quote 0
                      • G Offline
                        geoffbland
                        last edited by geoffbland

                        I found one of the VMs I had been using to test XOSTOR was locked up this morning. I restarted it but it will not start up and gives an error about the VDI being missing.

                        >xe vm-list name-label=test04
                        uuid ( RO)           : 8ec952b4-7229-7a30-81b6-1564a58f6343
                             name-label ( RW): test04
                            power-state ( RO): halted
                        
                        >xe vm-disk-list vm=test04
                        Disk 0 VBD:
                        uuid ( RO)             : e3c465b8-17a4-d147-6383-527bd9341a16
                            vm-name-label ( RO): test04
                               userdevice ( RW): 0
                        
                        
                        Disk 0 VDI:
                        uuid ( RO)             : 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c
                               name-label ( RW): test04_xostor01_vdi
                            sr-name-label ( RO): XOSTOR01
                             virtual-size ( RO): 42949672960
                        
                        >xe vm-start vm=test04
                        Error code: SR_BACKEND_FAILURE_46
                        Error parameters: , The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']],
                        

                        The logs for this are attached as file xostor issue 1.txt

                        ronan-aR 1 Reply Last reply Reply Quote 0
                        • ronan-aR Offline
                          ronan-a Vates 🪐 XCP-ng Team @geoffbland
                          last edited by

                          @geoffbland Can you execute this command on the other hosts please?

                          ls -l /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                          

                          Also I don't have all the info in your previous log, can you send me the previous SMlog files? (Using private message if you want. šŸ™‚ )

                          G 1 Reply Last reply Reply Quote 0
                          • G Offline
                            geoffbland @ronan-a
                            last edited by

                            @ronan-a said in XOSTOR hyperconvergence preview:

                            Can you execute this command on the other hosts please?

                            As requested

                            XCPNG01 - Current linstor master

                            [10:59 XCPNG01 ~]# ls -l /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                            lrwxrwxrwx 1 root root 17 May 22 19:24 /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0 -> ../../../drbd1004
                            

                            XCPNG02

                            [11:00 XCPNG02 ~]# ls -l /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                            lrwxrwxrwx 1 root root 17 May 22 19:25 /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0 -> ../../../drbd1004
                            

                            XCPNG03

                            [07:31 XCPNG03 ~]# ls -l /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                            lrwxrwxrwx 1 root root 17 May 22 19:24 /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0 -> ../../../drbd1004
                            

                            XCPNG04

                            [07:35 XCPNG04 ~]# ls -l /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                            lrwxrwxrwx 1 root root 17 May 22 19:24 /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0 -> ../../../drbd1004
                            

                            XCPNG05

                            [10:49 XCPNG05 ~]# ls -l /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                            lrwxrwxrwx 1 root root 17 May 22 19:24 /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0 -> ../../../drbd1004
                            
                            ronan-aR 1 Reply Last reply Reply Quote 0
                            • G Offline
                              geoffbland
                              last edited by

                              It seems like xe may be getting mixed up between where a host is running and where the XOSTOR storage is held.
                              Apologies if I have misunderstood and done something wrong here - but I think this migration should have worked.
                              I created a new VM on one of my hosts XCPNG05 using XOSTOR as the VDI RS. I can see that the linstore volumes are on hosts XCPNG01, XCPNG03 and XCPNG05.
                              XCPNG05 is an Intel server, XCPNG01 and XCPNG03 are AMD. The VM is running on XCPNG05.
                              Now when I try to migrate the VM's VDI from XOSTOR onto a local VDI on the same host the VM is currently running on I get a warning about incompatible CPUs.

                              To replicate the issue:

                              Create new VM test05 on XOSTOR.
                              VM is created on host XCPNG05.

                              >xe vm-list name-label=test05
                              uuid ( RO)           : d3f8c52d-be3c-3712-0ccc-a526dcc241a5
                                   name-label ( RW): test05
                                  power-state ( RO): running
                              
                              >xe vm-disk-list vm=test05
                              Disk 0 VBD:
                              uuid ( RO)             : a337cd1f-04cc-ce46-fbfb-d5d8e290dc03
                                  vm-name-label ( RO): test05
                                     userdevice ( RW): 0
                              
                              Disk 0 VDI:
                              uuid ( RO)             : f856680c-c00d-44af-ba3f-16d9952ccb2f
                                     name-label ( RW): test05_vdi
                                  sr-name-label ( RO): XOSTOR01
                                   virtual-size ( RO): 34359738368
                              
                              >xe sr-list name-label=XOSTOR01
                              uuid ( RO)                : cf896912-cd71-d2b2-488a-xxxxxxxx7c87
                                        name-label ( RW): XOSTOR01
                                  name-description ( RW):
                                              host ( RO): <shared>
                                              type ( RO): linstor
                                      content-type ( RO):
                              

                              Migrate to local disk (SSD1) on same host (XCPNG05) - this fails migrating to the same host the VM is currently running on due to incompatible CPU.

                              >xe sr-list name-label=XCPNG05SSD1
                              uuid ( RO)                : c0851501-3a1b-c661-70b9-54373e0d9847
                                        name-label ( RW): XCPNG05SSD1
                                  name-description ( RW):
                                              host ( RO): XCPNG05
                                              type ( RO): lvm
                                      content-type ( RO): user
                              
                              >xe vdi-pool-migrate uuid=f856680c-c00d-44af-ba3f-16d9952ccb2f sr-uuid=c0851501-3a1b-c661-70b9-54373e0d9847
                              The VM is incompatible with the CPU features of this host.
                              vm: d3f8c52d-be3c-3712-0ccc-a526dcc241a5 (test05)
                              host: 7bd62a77-71d6-4b51-9a86-850dd4ff4b60 (XCPNG05)
                              reason: VM last booted on a host which had a CPU from a different vendor.
                              
                              
                              1 Reply Last reply Reply Quote 0
                              • ronan-aR Offline
                                ronan-a Vates 🪐 XCP-ng Team @geoffbland
                                last edited by

                                @geoffbland Thank you, so ok the VDI is still here on all hosts.

                                You can try to check the status of the VDH like the smapi using:

                                /usr/bin/vhd-util query --debug -vsfp -n /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                                

                                If you have this problem on many hosts, I suspect a problem with DRBD, so maybe there is a useful info in daemon.log and/or kern.log.

                                G 1 Reply Last reply Reply Quote 0
                                • G Offline
                                  geoffbland @ronan-a
                                  last edited by geoffbland

                                  @ronan-a said in XOSTOR hyperconvergence preview:

                                  You can try to check the status of the VDH like the smapi using:
                                  /usr/bin/vhd-util query --debug -vsfp -n /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                                  useful info in daemon.log and/or kern.log.

                                  This gives the following:

                                  [10:59 XCPNG01 ~]# /usr/bin/vhd-util query --debug -vsfp -n /dev/drbd/by-res/xcp-volume-75c11231-0fb8-4b40-9e2e-a0665bb758c0/0
                                  40960
                                  2061447680
                                  query failed
                                  hidden: 0
                                  

                                  I will send logs by direct mail.

                                  ronan-aR 1 Reply Last reply Reply Quote 0
                                  • ronan-aR Offline
                                    ronan-a Vates 🪐 XCP-ng Team @geoffbland
                                    last edited by

                                    @geoffbland Okay so it's probably not related to the driver itself, I will take a look to the logs after reception. šŸ™‚

                                    G 1 Reply Last reply Reply Quote 0
                                    • G Offline
                                      geoffbland
                                      last edited by

                                      Failure trying to revert a VM to a snapshot with XOSTOR.

                                      Created a VM with main VDI on XOSTOR (24GB) and with 6 disks each also on XOSTOR (2GB each).
                                      All is running OK.
                                      Now create a snapshot of the VM - this takes quite a while but does eventually succeed.
                                      Now using XO (from sources) click the "Revert VM to this snapshot". This errors and the VM stops.

                                      vm.revert
                                      {
                                        "snapshot": "6032fc73-eb7f-cf64-2481-4346b7b57204"
                                      }
                                      {
                                        "code": "VM_REVERT_FAILED",
                                        "params": [
                                          "OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378",
                                          "OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add"
                                        ],
                                        "task": {
                                          "uuid": "4804fefd-0037-d7dd-9a7c-769230728483",
                                          "name_label": "Async.VM.revert",
                                          "name_description": "",
                                          "allowed_operations": [],
                                          "current_operations": {},
                                          "created": "20220527T15:01:42Z",
                                          "finished": "20220527T15:01:46Z",
                                          "status": "failure",
                                          "resident_on": "OpaqueRef:a1e9a8f3-0a79-4824-b29f-d81b3246d190",
                                          "progress": 1,
                                          "type": "<none/>",
                                          "result": "",
                                          "error_info": [
                                            "VM_REVERT_FAILED",
                                            "OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378",
                                            "OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add"
                                          ],
                                          "other_config": {},
                                          "subtask_of": "OpaqueRef:NULL",
                                          "subtasks": [],
                                          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 492))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
                                        },
                                        "message": "VM_REVERT_FAILED(OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378, OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add)",
                                        "name": "XapiError",
                                        "stack": "XapiError: VM_REVERT_FAILED(OpaqueRef:1439fd0f-4e66-44c9-99af-1f8536e59378, OpaqueRef:5ad4c51e-473e-4ab0-877d-2d0dbdb90add)
                                          at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_XapiError.js:16:12)
                                          at _default (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_getTaskResult.js:11:29)
                                          at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:949:24)
                                          at forEach (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:983:14)
                                          at Array.forEach (<anonymous>)
                                          at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:973:12)
                                          at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:1139:14)"
                                      }
                                      

                                      Now viewing the VM with XO on the disks tab shows no attached disks - disk tab is blank.

                                      XOSTOR01 Storage No Disks.png

                                      But linstor appears to still have the disks and the snapshot disks too.

                                      ā”Š XCPNG01 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-142cb89f-2850-4ac8-a47c-10bb2cfc4692 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1010 ā”Š /dev/drbd1010 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š           ā”Š InUse  ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-18fa145a-d36b-44bd-b1b5-af1e9424ea00 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1018 ā”Š /dev/drbd1018 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-1a6c7272-f718-4c4d-a8b0-ca8419eab314 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1024 ā”Š /dev/drbd1024 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-2cab6c2d-abf6-42c7-9094-d75351ed8ebb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1016 ā”Š /dev/drbd1016 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-30bf014b-025d-4f3f-a068-f9a9bf34fab2 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1013 ā”Š /dev/drbd1013 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-30bf014b-025d-4f3f-a068-f9a9bf34fab2 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1013 ā”Š /dev/drbd1013 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-30bf014b-025d-4f3f-a068-f9a9bf34fab2 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1013 ā”Š /dev/drbd1013 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-3bdb2b25-706c-4309-ab8f-df3190f57c43 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1021 ā”Š /dev/drbd1021 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-450f65f7-7fcc-4ffd-893e-761a2f6ac366 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1020 ā”Š /dev/drbd1020 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-466938db-11f1-4b59-8a90-ad08fa20e085 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1015 ā”Š /dev/drbd1015 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-470dcf6f-d916-403d-8258-e012c065b8ec ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1009 ā”Š /dev/drbd1009 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-551db5b5-7772-407a-9e8c-e549db3a0e5f ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1008 ā”Š /dev/drbd1008 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-699871db-2319-4ddd-9a44-0514d2e7aee3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1025 ā”Š /dev/drbd1025 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-6c96822b-7ded-41dd-b4ff-690dc4795ee7 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1023 ā”Š /dev/drbd1023 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-70004559-a2c4-480f-b7bc-b26dcb95bfba ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1027 ā”Š /dev/drbd1027 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-707a0158-ad31-4b4b-af2b-20d89e5717de ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1026 ā”Š /dev/drbd1026 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-7aaa7a6e-98c4-4a57-a4f1-4fea0a36b17a ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1011 ā”Š /dev/drbd1011 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-7aaa7a6e-98c4-4a57-a4f1-4fea0a36b17a ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1011 ā”Š /dev/drbd1011 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-7aaa7a6e-98c4-4a57-a4f1-4fea0a36b17a ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1011 ā”Š /dev/drbd1011 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-9320c158-489e-49e7-92b8-85c93c9e3eeb ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1022 ā”Š /dev/drbd1022 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-b341848b-01d1-4019-a62f-85c6108a53e3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1006 ā”Š /dev/drbd1006 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-b341848b-01d1-4019-a62f-85c6108a53e3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1006 ā”Š /dev/drbd1006 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-b341848b-01d1-4019-a62f-85c6108a53e3 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1006 ā”Š /dev/drbd1006 ā”Š 24.06 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-bccefe12-9ff5-4317-b05c-515cb44a5710 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1014 ā”Š /dev/drbd1014 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-cdc051ae-bc39-4012-9ce0-6e4f855a5063 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1012 ā”Š /dev/drbd1012 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-d5a744ec-d1a1-4116-a576-38608b9dd790 ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1019 ā”Š /dev/drbd1019 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG01 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      ā”Š XCPNG02 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG03 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG04 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š xcp-sr-linstor_group ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š  2.02 GiB ā”Š Unused ā”Š UpToDate ā”Š
                                      ā”Š XCPNG05 ā”Š xcp-volume-f9cf9143-829d-4246-9051-9102f2c4709c ā”Š DfltDisklessStorPool ā”Š     0 ā”Š    1017 ā”Š /dev/drbd1017 ā”Š           ā”Š Unused ā”Š Diskless ā”Š
                                      
                                      

                                      From the VM disks tab if I try to Attach the disks, two of the disks created on XOSTOR are missing (data1 and data4).

                                      XOSTOR01 Storage Disks Missing.png

                                      Finally if I go to storage and bring up the XOSTOR storage and then press "Rescan all disks" I get this error:

                                      sr.scan
                                      {
                                        "id": "cf896912-cd71-d2b2-488a-5792b7147c87"
                                      }
                                      {
                                        "code": "SR_BACKEND_FAILURE_46",
                                        "params": [
                                          "",
                                          "The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']]",
                                          ""
                                        ],
                                        "task": {
                                          "uuid": "4dcac885-dfaa-784a-eb2d-02335efde0fb",
                                          "name_label": "Async.SR.scan",
                                          "name_description": "",
                                          "allowed_operations": [],
                                          "current_operations": {},
                                          "created": "20220527T16:27:36Z",
                                          "finished": "20220527T16:27:50Z",
                                          "status": "failure",
                                          "resident_on": "OpaqueRef:a1e9a8f3-0a79-4824-b29f-d81b3246d190",
                                          "progress": 1,
                                          "type": "<none/>",
                                          "result": "",
                                          "error_info": [
                                            "SR_BACKEND_FAILURE_46",
                                            "",
                                            "The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']]",
                                            ""
                                          ],
                                          "other_config": {},
                                          "subtask_of": "OpaqueRef:NULL",
                                          "subtasks": [],
                                          "backtrace": "(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_access.ml)(line 32))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 128))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
                                        },
                                        "message": "SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']], )",
                                        "name": "XapiError",
                                        "stack": "XapiError: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Could not load 735fc2d7-f1f0-4cc6-9d35-42a049d8ec6c because: ['XENAPI_PLUGIN_FAILURE', 'getVHDInfo', 'CommandException', 'No such file or directory']], )
                                          at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_XapiError.js:16:12)
                                          at _default (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/_getTaskResult.js:11:29)
                                          at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:949:24)
                                          at forEach (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:983:14)
                                          at Array.forEach (<anonymous>)
                                          at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:973:12)
                                          at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202204291839/packages/xen-api/src/index.js:1139:14)"
                                      }
                                      
                                      1 Reply Last reply Reply Quote 0
                                      • G Offline
                                        geoffbland @ronan-a
                                        last edited by

                                        @ronan-a said in XOSTOR hyperconvergence preview:

                                        Okay so it's probably not related to the driver itself, I will take a look to the logs after reception. šŸ™‚

                                        Did you get chance to look at the logs I sent?

                                        ronan-aR 1 Reply Last reply Reply Quote 0
                                        • ronan-aR Offline
                                          ronan-a Vates 🪐 XCP-ng Team @geoffbland
                                          last edited by ronan-a

                                          @geoffbland So, I didn't notice useful info outside of:

                                          FIXME drbd_a_xcp-volu[24302] op clear, bitmap locked for 'set_n_write sync_handshake' by drbd_r_xcp-volu[24231]
                                          ...
                                          FIXME drbd_a_xcp-volu[24328] op clear, bitmap locked for 'demote' by drbd_w_xcp-volu[24188]
                                          

                                          Like I said in my e-mail, maybe there are more details in another log file. I hope. šŸ™‚

                                          G 2 Replies Last reply Reply Quote 0
                                          • G Offline
                                            geoffbland @ronan-a
                                            last edited by

                                            @ronan-a said in [XOSTOR hyperconvergence preview]

                                            Like I said in my e-mail, maybe there are more details in another log file. I hope. šŸ™‚

                                            In the end I realised I was more trying to "use" XOSTOR whilst testing rather than properly test it. So I decided to rip it all down and start again and retest it again - this time properly recording each step so any issues can be replicated. I will let you know how this goes.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post