XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    xoce INTERNAL_ERROR while trying to create VM

    Scheduled Pinned Locked Moved Xen Orchestra
    37 Posts 5 Posters 10.4k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • burbilogB Offline
      burbilog @olivierlambert
      last edited by

      @olivierlambert here:

      [10:48 xen ~]# xe sr-param-list uuid=11b39dd4-c011-79b4-2f50-1c9299d8d1a5
      uuid ( RO)                    : 11b39dd4-c011-79b4-2f50-1c9299d8d1a5
                    name-label ( RW): Fiend ISO N3
              name-description ( RW): Fiend ISO N3
                          host ( RO): xen.legion.domain
            allowed-operations (SRO): unplug; plug; PBD.create; PBD.destroy; VDI.clone; scan; VDI.create; VDI.destroy
            current-operations (SRO): 
                          VDIs (SRO): 90a7264c-b21e-4f24-a723-9f9469feaf97; 00e36a4f-084a-491c-8a02-ad43f57d843e; e10aa9eb-a193-4560-8189-3a4c6f6b14db; 34c97f95-adbe-4a2c-803b-621614c6423b; 310af2b4-18e5-4fcc-8dfc-86f827c138a3
                          PBDs (SRO): 6c906b4d-32ab-ab2a-ccfb-45301f0ce6b4
            virtual-allocation ( RO): 11692546048
          physical-utilisation ( RO): 17673748480
                 physical-size ( RO): 5856807616512
                          type ( RO): iso
                  content-type ( RO): iso
                        shared ( RW): true
                 introduced-by ( RO): <not in database>
                   is-tools-sr ( RO): false
                  other-config (MRW): 
                     sm-config (MRO): 
                         blobs ( RO): 
           local-cache-enabled ( RO): false
                          tags (SRW): 
                     clustered ( RO): false
      
      
      1 Reply Last reply Reply Quote 0
      • olivierlambertO Online
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        This sounds OK to me. I think your issue is related to something else, please display the VM record (xe vm-param-list uuid=<VM UUID>

        burbilogB 1 Reply Last reply Reply Quote 0
        • burbilogB Offline
          burbilog @olivierlambert
          last edited by

          @olivierlambert How can I show that param if I'm getting error during my attempts to create a VM?

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Online
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            So the VM isn't created at all? You really need to give us more info. What template are you using? Is it working if you don't add an ISO?

            burbilogB 1 Reply Last reply Reply Quote 0
            • burbilogB Offline
              burbilog @olivierlambert
              last edited by

              @olivierlambert vates.png

              burbilogB 1 Reply Last reply Reply Quote 0
              • burbilogB Offline
                burbilog @burbilog
                last edited by

                Yes, this error:

                INTERNAL_ERROR((Failure "Expected string, got 'N'"))
                
                This is a XenServer/XCP-ng error
                

                happens when I click create.

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Online
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  Please provide the detailed error log (eye icon)

                  burbilogB 1 Reply Last reply Reply Quote 0
                  • burbilogB Offline
                    burbilog @olivierlambert
                    last edited by

                    @olivierlambert

                    vm.create
                    {
                      "clone": true,
                      "existingDisks": {},
                      "installation": {
                        "method": "cdrom",
                        "repository": "44b7c853-a994-4563-9981-b31e0437ae5c"
                      },
                      "name_label": "x",
                      "template": "a961ebe2-d3fc-bc79-7eab-e0eef25f0025-4b6425be-728a-4a2c-8333-0aef209a348b",
                      "VDIs": [
                        {
                          "bootable": true,
                          "device": "0",
                          "size": 10737418240,
                          "type": "system",
                          "SR": "47a51cb0-8113-9b8f-07d6-6c61ac99f479",
                          "name_description": "Created by XO",
                          "name_label": "CentOS 8_ezaro"
                        }
                      ],
                      "VIFs": [
                        {
                          "network": "82a075b9-03c3-e4fc-8592-09da06b879ab",
                          "allowedIpv4Addresses": [],
                          "allowedIpv6Addresses": []
                        }
                      ],
                      "CPUs": 1,
                      "cpusMax": 1,
                      "cpuWeight": null,
                      "cpuCap": null,
                      "name_description": "x",
                      "memory": 4294967296,
                      "bootAfterCreate": true,
                      "copyHostBiosStrings": false,
                      "secureBoot": false,
                      "share": false,
                      "coreOs": false,
                      "tags": [],
                      "hvmBootFirmware": "bios"
                    }
                    {
                      "code": "INTERNAL_ERROR",
                      "params": [
                        "(Failure \"Expected string, got 'N'\")"
                      ],
                      "call": {
                        "method": "VM.get_power_state",
                        "params": [
                          null
                        ]
                      },
                      "message": "INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))",
                      "name": "XapiError",
                      "stack": "XapiError: INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))
                        at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
                        at /opt/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:37:27
                        at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
                        at cb (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42)
                        at tryCatcher (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23)
                        at Promise._settlePromiseFromHandler (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31)
                        at Promise._settlePromise (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18)
                        at Promise._settlePromise0 (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10)
                        at Promise._settlePromises (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18)
                        at _drainQueueStep (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12)
                        at _drainQueue (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9)
                        at Async._drainQueues (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5)
                        at Immediate.Async.drainQueues [as _onImmediate] (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14)
                        at processImmediate (node:internal/timers:466:21)
                        at process.callbackTrampoline (node:internal/async_hooks:130:17)"
                    }
                    
                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Online
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      @burbilog said in xoce INTERNAL_ERROR while trying to create VM:

                      INTERNAL_ERROR((Failure "Expected string, got 'N'")

                      You have to investigate on your XCP-ng host, there's obviously a problem in there.

                      See https://xcp-ng.org/docs/troubleshooting.html

                      I would suggest to restart the toolstack too.

                      burbilogB 1 Reply Last reply Reply Quote 0
                      • burbilogB Offline
                        burbilog @olivierlambert
                        last edited by

                        @olivierlambert This problem is common for all 3 servers, i did not touch other two hosts at all for a while. I see some strange error appear in xensource.log after VM creation attempt:

                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] VM.add_to_platform D:87deedffed4a failed with exception Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "device-model")
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "device-model")
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] VM.add_to_platform D:6b3c2fce951b failed with exception Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "secureboot")
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:cae2f8cd-6d33-4ca8-81a1-0eb76df149bd", "secureboot")
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                        May 13 11:59:07 xen xapi: [error||13930 :::80||backtrace]
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                        May 13 11:59:07 xen xapi: [error||13931 :::80||backtrace]
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
                        May 13 11:59:07 xen xapi: [error||13929 :::80||backtrace]
                        

                        This problem is common for all servers.

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Online
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          You have duplicates in XAPI DB, that's not normal 🤔

                          It's coming from a template I assume.

                          burbilogB 1 Reply Last reply Reply Quote 0
                          • burbilogB Offline
                            burbilog @olivierlambert
                            last edited by

                            @olivierlambert But templates are stock templates, no? It does not matter what template I choose, it fails anyway.

                            How to fix them?

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Online
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              I have no context on what you did on those hosts previously, so it's hard to answer what to do to assist.

                              I don't know what could trigger that in the first place 🤔

                              burbilogB 1 Reply Last reply Reply Quote 0
                              • burbilogB Offline
                                burbilog @olivierlambert
                                last edited by

                                @olivierlambert I did not do anything on these hosts, except occasional yum update. Frankly, i did not touch this setup for a while and it worked a month or two ago ok.

                                I'm kinda surprised that all 3 servers do the same. One of them wasn't touched at all except yum update since it worked ok.

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Online
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  It's normal you have the problem on the whole pool/all hosts, because the XAPI database is shared with all hosts.

                                  I would search for duplicate keys for platform record on all your existing VMs.

                                  burbilogB 1 Reply Last reply Reply Quote 0
                                  • burbilogB Offline
                                    burbilog @olivierlambert
                                    last edited by

                                    @olivierlambert So, how to look up these keys? I ran xe vm-list and xe sr-list on all hosts, it seems that all UUIDs are different.

                                    I even reinstalled one of the hosts, set up 8.2.1 anew and the problem here persists.

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Online
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      Again, reinstalling won't change anything since the info is within the main DB, which is copied on all pool members from the master 🙂

                                      You need to find duplicates UUIDs in all your VMs objects.

                                      burbilogB 2 Replies Last reply Reply Quote 0
                                      • burbilogB Offline
                                        burbilog @olivierlambert
                                        last edited by

                                        @olivierlambert I ran xe sr-list and xe-vm-list on all hosts, compiled a list of uuids, sorted them and all of them are unique. How to search for duplicate keys then?

                                        1 Reply Last reply Reply Quote 0
                                        • burbilogB Offline
                                          burbilog @olivierlambert
                                          last edited by

                                          @olivierlambert installed xo-cli, connected and ran xo-cli --list-objects , then grepped all uuids, stirpped everything else and tried to find dupes -- none. all uuids are unique there.

                                          burbilogB 1 Reply Last reply Reply Quote 0
                                          • burbilogB Offline
                                            burbilog @burbilog
                                            last edited by

                                            @burbilog ah, no, there are some repeated, i had to sort it first. but there is a lot of uuids that are repeated.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post