XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    xoce INTERNAL_ERROR while trying to create VM

    Scheduled Pinned Locked Moved Xen Orchestra
    37 Posts 5 Posters 8.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • burbilogB
      burbilog
      last edited by

      Something happened with my xoce installation. May be some updates, may be something else, but I did not do much there for some months except upgrading xoce. Any attempts to create a new VM now results in pink box saying "unknown error from the peer" in top right corner. Clicking on view log shows this message:

      INTERNAL_ERROR((Failure "Expected string, got 'N'"))

      If I create NFS SR and then immediately attempt to delete it, I get another strange message:

      SR_HAS_NO_PBDS(OpaqueRef:790a7cdc-b5c1-46d5-9c91-f7789e70aa2d)

      And there is no way to delete that SR anymore. Any ideas how to fix it? Can't create VMs, can't do anything. Xen Orchestra, commit ecae5, xo-server 5.93.1, xo-web 5.96.0, updated by this script: https://github.com/Jarli01/xenorchestra_updater. This happened before today's upgrade, I upgraded to latest version, but it did not fix the problem.

      Hardware servers are like this:

      NAME="XCP-ng"
VERSION="8.2.1"
ID="xenenterprise"
ID_LIKE="centos rhel fedora"
VERSION_ID="8.2.1"
PRETTY_NAME="XCP-ng 8.2.1"
ANSI_COLOR="0;31"
HOME_URL="http://xcp-ng.org/"
BUG_REPORT_URL="https://github.com/xcp-ng/xcp"
      1 Reply Last reply Reply Quote 0
      • olivierlambertO
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hi,

        I think it's not related to Xen Orchestra but XCP-ng. Can you display the SR object via xe sr-param-list uuid=<SR UUID>?

        burbilogB 1 Reply Last reply Reply Quote 0
        • burbilogB
          burbilog @olivierlambert
          last edited by

          @olivierlambert said in xoce INTERNAL_ERROR while trying to create VM:

          sr-param-list uuid=

          Yes, I can. Here it is:

          [10:11 xen ~]# xe sr-param-list uuid=861221a3-24a6-a43c-176d-2158059cd64e
uuid ( RO)                    : 861221a3-24a6-a43c-176d-2158059cd64e
              name-label ( RW): Fiend ISO
        name-description ( RW): Fiend ISO
                    host ( RO): <not in database>
      allowed-operations (SRO): unplug; plug; PBD.create; PBD.destroy; forget; VDI.clone; scan; VDI.create; VDI.destroy
      current-operations (SRO): 
                    VDIs (SRO): ed51798b-742d-4e38-bb40-acb1f9e3114e; fe8b400f-35f0-4f90-8f26-e5b850f2a8cd; a89f4d6f-20c9-418d-b565-32b36a1ff7bd; 1d654467-f612-4ed4-8f18-9f67a3115072; a6412615-f7e8-4397-bdad-618612f3ad96
                    PBDs (SRO): 
      virtual-allocation ( RO): 11692546048
    physical-utilisation ( RO): 17673748480
           physical-size ( RO): 5856807616512
                    type ( RO): iso
            content-type ( RO): iso
                  shared ( RW): true
           introduced-by ( RO): <not in database>
             is-tools-sr ( RO): false
            other-config (MRW): 
               sm-config (MRO): 
                   blobs ( RO): 
     local-cache-enabled ( RO): false
                    tags (SRW): 
               clustered ( RO): false

          I just reinstalled xoce completely, added servers and the problem persists.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            There's no PBDs in there. Just forget the SR and recreate it.

            xe sr-forget uuid=861221a3-24a6-a43c-176d-2158059cd64e

            burbilogB 1 Reply Last reply Reply Quote 0
            • burbilogB
              burbilog @olivierlambert
              last edited by

              @olivierlambert said in xoce INTERNAL_ERROR while trying to create VM:

              There's no PBDs in there. Just forget the SR and recreate it.

              xe sr-forget uuid=861221a3-24a6-a43c-176d-2158059cd64e

              Well, it helped with SR, but not with VM creation. I created new SR. It created ok. I see ISOs there. It's mounted, I can read files from NFS:

              10.1.0.222:/nas/xoce/iso 5719538688 17259520 5702279168   1% /run/sr-mount/b9af5c9d-0baf-8fe1-f325-3fed4351097b

              But if I try to create VM pink box with "unknown error from the peer" happenes and new entry to the log is added:

              INTERNAL_ERROR((Failure "Expected string, got 'N'"))

This is a XenServer/XCP-ng error

              BTW, I can't remove newly created SR. Any attempts to delete it immediately results in "unknown error" and SR_NOT_EMPTY() in log.

              1 Reply Last reply Reply Quote 0
              • olivierlambertO
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Please display the SR record for the one you freshly created.

                You can't remove a SR if there's disks in it, you need to forget it instead.

                burbilogB 1 Reply Last reply Reply Quote 0
                • burbilogB
                  burbilog @olivierlambert
                  last edited by

                  @olivierlambert here:

                  [10:48 xen ~]# xe sr-param-list uuid=11b39dd4-c011-79b4-2f50-1c9299d8d1a5
uuid ( RO)                    : 11b39dd4-c011-79b4-2f50-1c9299d8d1a5
              name-label ( RW): Fiend ISO N3
        name-description ( RW): Fiend ISO N3
                    host ( RO): xen.legion.domain
      allowed-operations (SRO): unplug; plug; PBD.create; PBD.destroy; VDI.clone; scan; VDI.create; VDI.destroy
      current-operations (SRO): 
                    VDIs (SRO): 90a7264c-b21e-4f24-a723-9f9469feaf97; 00e36a4f-084a-491c-8a02-ad43f57d843e; e10aa9eb-a193-4560-8189-3a4c6f6b14db; 34c97f95-adbe-4a2c-803b-621614c6423b; 310af2b4-18e5-4fcc-8dfc-86f827c138a3
                    PBDs (SRO): 6c906b4d-32ab-ab2a-ccfb-45301f0ce6b4
      virtual-allocation ( RO): 11692546048
    physical-utilisation ( RO): 17673748480
           physical-size ( RO): 5856807616512
                    type ( RO): iso
            content-type ( RO): iso
                  shared ( RW): true
           introduced-by ( RO): <not in database>
             is-tools-sr ( RO): false
            other-config (MRW): 
               sm-config (MRO): 
                   blobs ( RO): 
     local-cache-enabled ( RO): false
                    tags (SRW): 
               clustered ( RO): false
                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    This sounds OK to me. I think your issue is related to something else, please display the VM record (xe vm-param-list uuid=<VM UUID>

                    burbilogB 1 Reply Last reply Reply Quote 0
                    • burbilogB
                      burbilog @olivierlambert
                      last edited by

                      @olivierlambert How can I show that param if I'm getting error during my attempts to create a VM?

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        So the VM isn't created at all? You really need to give us more info. What template are you using? Is it working if you don't add an ISO?

                        burbilogB 1 Reply Last reply Reply Quote 0
                        • burbilogB
                          burbilog @olivierlambert
                          last edited by

                          @olivierlambert vates.png

                          burbilogB 1 Reply Last reply Reply Quote 0
                          • burbilogB
                            burbilog @burbilog
                            last edited by

                            Yes, this error:

                            INTERNAL_ERROR((Failure "Expected string, got 'N'"))

This is a XenServer/XCP-ng error

                            happens when I click create.

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Please provide the detailed error log (eye icon)

                              burbilogB 1 Reply Last reply Reply Quote 0
                              • burbilogB
                                burbilog @olivierlambert
                                last edited by

                                @olivierlambert

                                vm.create
{
  "clone": true,
  "existingDisks": {},
  "installation": {
    "method": "cdrom",
    "repository": "44b7c853-a994-4563-9981-b31e0437ae5c"
  },
  "name_label": "x",
  "template": "a961ebe2-d3fc-bc79-7eab-e0eef25f0025-4b6425be-728a-4a2c-8333-0aef209a348b",
  "VDIs": [
    {
      "bootable": true,
      "device": "0",
      "size": 10737418240,
      "type": "system",
      "SR": "47a51cb0-8113-9b8f-07d6-6c61ac99f479",
      "name_description": "Created by XO",
      "name_label": "CentOS 8_ezaro"
    }
  ],
  "VIFs": [
    {
      "network": "82a075b9-03c3-e4fc-8592-09da06b879ab",
      "allowedIpv4Addresses": [],
      "allowedIpv6Addresses": []
    }
  ],
  "CPUs": 1,
  "cpusMax": 1,
  "cpuWeight": null,
  "cpuCap": null,
  "name_description": "x",
  "memory": 4294967296,
  "bootAfterCreate": true,
  "copyHostBiosStrings": false,
  "secureBoot": false,
  "share": false,
  "coreOs": false,
  "tags": [],
  "hvmBootFirmware": "bios"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "(Failure \"Expected string, got 'N'\")"
  ],
  "call": {
    "method": "VM.get_power_state",
    "params": [
      null
    ]
  },
  "message": "INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
    at cb (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}
                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  @burbilog said in xoce INTERNAL_ERROR while trying to create VM:

                                  INTERNAL_ERROR((Failure "Expected string, got 'N'")

                                  You have to investigate on your XCP-ng host, there's obviously a problem in there.

                                  See https://xcp-ng.org/docs/troubleshooting.html

                                  I would suggest to restart the toolstack too.

                                  burbilogB 1 Reply Last reply Reply Quote 0
                                  • burbilogB
                                    burbilog @olivierlambert
                                    last edited by

                                    @olivierlambert This problem is common for all 3 servers, i did not touch other two hosts at all for a while. I see some strange error appear in xensource.log after VM creation attempt:

                                    May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] VM.add_to_platform D:87deedffed4a failed with exception Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "device-model")
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "device-model")
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] VM.add_to_platform D:6b3c2fce951b failed with exception Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "secureboot")
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "secureboot")
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace]
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace]
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace]

                                    This problem is common for all servers.

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      You have duplicates in XAPI DB, that's not normal 🤔

                                      It's coming from a template I assume.

                                      burbilogB 1 Reply Last reply Reply Quote 0
                                      • burbilogB
                                        burbilog @olivierlambert
                                        last edited by

                                        @olivierlambert But templates are stock templates, no? It does not matter what template I choose, it fails anyway.

                                        How to fix them?

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          I have no context on what you did on those hosts previously, so it's hard to answer what to do to assist.

                                          I don't know what could trigger that in the first place 🤔

                                          burbilogB 1 Reply Last reply Reply Quote 0
                                          • burbilogB
                                            burbilog @olivierlambert
                                            last edited by

                                            @olivierlambert I did not do anything on these hosts, except occasional yum update. Frankly, i did not touch this setup for a while and it worked a month or two ago ok.

                                            I'm kinda surprised that all 3 servers do the same. One of them wasn't touched at all except yum update since it worked ok.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post