XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Cannot use cloudConfig while creating VM with RPC call

    Scheduled Pinned Locked Moved Solved Xen Orchestra
    23 Posts 5 Posters 3.4k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • sMteXS Offline
      sMteX
      last edited by

      Have you had a chance to take a look please? @ddelnano

      D 1 Reply Last reply Reply Quote 0
      • D Offline
        ddelnano Terraform Team @sMteX
        last edited by

        @sMteX sorry for the late reply. I responded to your GitHub issue with some questions and comments.

        Let's keep the discussion there, and we can summarize the conclusion here once we get to the bottom of it.

        TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 1
        • TheiLLeniumStudiosT Offline
          TheiLLeniumStudios @ddelnano
          last edited by

          @ddelnano I'm running into a very similar issue. Didn't want to create a separate issue on github so added my comments on the old one here: https://github.com/terra-farm/terraform-provider-xenorchestra/issues/227#issuecomment-1536727385

          Would really appreciate it if you could help me out investigating the root cause and fixing it. I'll be happy to try out things and provide more details if required

          sMteX created this issue in terra-farm/terraform-provider-xenorchestra

          closed Cannot use cloud_config while creating VM #227

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Online
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Is your VM created on a slave host in a pool?

            TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
            • TheiLLeniumStudiosT Offline
              TheiLLeniumStudios @olivierlambert
              last edited by TheiLLeniumStudios

              @olivierlambert Yes, the problematic VMs are on a slave host in the pool. I'm explicitly assigning affinity host + local SR for every new VM based on even odd numbering. I create a total of 6 VMs, 4 of which are on the master host which boot with the cloud-config drive just fine. 2 gets scheduled on the slave and their cloud config gets corrupted and Talos cannot read it

              TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
              • TheiLLeniumStudiosT Offline
                TheiLLeniumStudios @TheiLLeniumStudios
                last edited by

                @olivierlambert after tracing through the xen-orchestra code, I found where the problem originates from. I see this error in XO whenever a cloud config drive is being created:

                2023-05-05T22:51:22.141Z xo:xapi WARN importVdiContent:  {
                  error: Error: 404 Not Found
                      at Object.assertSuccess (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:138:19)
                      at httpRequestPlus (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:205:22)
                      at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
                      at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
                      at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
                      at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:211:11)
                      at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
                    originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3A696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef%3A1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef%3Acabc6a41-7f36-2ae5-d03b-5515d77f8c60',
                    url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef:1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef:cabc6a41-7f36-2ae5-d03b-5515d77f8c60',
                    pool_master: host {
                      uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
                      name_label: 'minisforum-hm80-01.servers.illenium.gg',
                      name_description: 'Default install',
                      memory_overhead: 1161101312,
                      allowed_operations: [Array],
                      current_operations: [Object],
                      API_version_major: 2,
                      API_version_minor: 20,
                      API_version_vendor: 'XenSource',
                      API_version_vendor_implementation: {},
                      enabled: true,
                      software_version: [Object],
                      other_config: [Object],
                      capabilities: [Array],
                      cpu_configuration: {},
                      sched_policy: 'credit',
                      supported_bootloaders: [Array],
                      resident_VMs: [Array],
                      logging: {},
                      PIFs: [Array],
                      suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
                      crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
                      crashdumps: [],
                      patches: [],
                      updates: [],
                      PBDs: [Array],
                      host_CPUs: [Array],
                      cpu_info: [Object],
                      hostname: 'minisforum-hm80-01.servers.illenium.gg',
                      address: '10.0.0.12',
                      metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
                      license_params: [Object],
                      ha_statefiles: [],
                      ha_network_peers: [],
                      blobs: {},
                      tags: [],
                      external_auth_type: '',
                      external_auth_service_name: '',
                      external_auth_configuration: {},
                      edition: 'xcp-ng',
                      license_server: [Object],
                      bios_strings: [Object],
                      power_on_mode: '',
                      power_on_config: {},
                      local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
                      chipset_info: [Object],
                      PCIs: [Array],
                      PGPUs: [Array],
                      PUSBs: [Array],
                      ssl_legacy: false,
                      guest_VCPUs_params: {},
                      display: 'enabled',
                      virtual_hardware_platform_versions: [Array],
                      control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
                      updates_requiring_reboot: [],
                      features: [],
                      iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
                      multipathing: false,
                      uefi_certificates: '',
                      certificates: [Array],
                      editions: [Array],
                      pending_guidances: [],
                      tls_verification_enabled: true,
                      last_software_update: '19700101T00:00:00Z',
                      https_only: false
                    },
                    SR: SR {
                      uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
                      name_label: 'minisforum-hm80-02 SSD',
                      name_description: '',
                      allowed_operations: [Array],
                      current_operations: {},
                      VDIs: [Array],
                      PBDs: [Array],
                      virtual_allocation: 25404899328,
                      physical_utilisation: 289996800,
                      physical_size: 901115478016,
                      type: 'ext',
                      content_type: 'user',
                      shared: false,
                      other_config: [Object],
                      tags: [],
                      sm_config: [Object],
                      blobs: {},
                      local_cache_enabled: true,
                      introduced_by: 'OpaqueRef:NULL',
                      clustered: false,
                      is_tools_sr: false
                    },
                    VDI: VDI {
                      uuid: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
                      name_label: 'XO CloudConfigDrive',
                      name_description: '',
                      allowed_operations: [Array],
                      current_operations: {},
                      SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
                      VBDs: [],
                      crash_dumps: [],
                      virtual_size: 10485760,
                      physical_utilisation: 3584,
                      type: 'user',
                      sharable: false,
                      read_only: false,
                      other_config: {},
                      storage_lock: false,
                      location: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
                      managed: true,
                      missing: false,
                      parent: 'OpaqueRef:NULL',
                      xenstore_data: {},
                      sm_config: {},
                      is_a_snapshot: false,
                      snapshot_of: 'OpaqueRef:NULL',
                      snapshots: [],
                      snapshot_time: '19700101T00:00:00Z',
                      tags: [],
                      allow_caching: false,
                      on_boot: 'persist',
                      metadata_of_pool: '',
                      metadata_latest: false,
                      is_tools_iso: false,
                      cbt_enabled: false
                    }
                  }
                }
                

                This seems to come from this specific line of code: https://github.com/vatesfr/xen-orchestra/blob/master/packages/xo-server/src/xapi/index.mjs#L1332-L1334

                And there's a comment there as well by 1 of the devs which suggests that it could be a potential bug?

                TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                • TheiLLeniumStudiosT Offline
                  TheiLLeniumStudios @TheiLLeniumStudios
                  last edited by

                  Basically what happens is that as a result of ignoring the warning, an empty cloudinitdrive gets created and doesn't have the correct data inside and the VM creation succeeds

                  TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                  • TheiLLeniumStudiosT Offline
                    TheiLLeniumStudios @TheiLLeniumStudios
                    last edited by TheiLLeniumStudios

                    I think importVdiContent is ignoring the SR id and is trying to clone the VDI on the master instead of the slave, hence the 404

                    TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                    • TheiLLeniumStudiosT Offline
                      TheiLLeniumStudios @TheiLLeniumStudios
                      last edited by

                      await this.putResource(cancelToken, stream, '/import_raw_vdi/', {
                              query: {
                                format,
                                vdi: ref,
                              },
                              task: await this.task_create(`Importing content into VDI ${await this.getField('VDI', ref, 'name_label')}`),
                            })
                      

                      This returns 404. I'm trying to understand what does /import_raw_vdi/ do? Also, this task gets created on the master host instead of the slave, even for the VMs that are scheduled on the slave host. Is that why it returns a 404? If it does, then how does the filesystem for the cloudconfigdrive get created on the slave? Maybe that needs to be moved over to the master as well? I don't know if I'm understanding currently or not so feel free to correct me if this is not how it works and my assumptions are wrong

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Online
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        The problem is already detected by @julien-f

                        Expect a fix in the next days or in our next XOA patch release next week

                        TheiLLeniumStudiosT 2 Replies Last reply Reply Quote 0
                        • TheiLLeniumStudiosT Offline
                          TheiLLeniumStudios @olivierlambert
                          last edited by

                          @olivierlambert Awesome! I use XO built from git source so will monitor the commits then. Thanks!

                          1 Reply Last reply Reply Quote 0
                          • TheiLLeniumStudiosT Offline
                            TheiLLeniumStudios @olivierlambert
                            last edited by

                            @olivierlambert Just tried the latest changes from xen-orchestra and the problem persists. I get the exact same error produced in XO when importing CloudConfigDrive on a slave:

                            2023-05-13T09:03:41.719Z xo:xapi WARN importVdiContent:  {
                              error: Error: 404 Not Found
                                  at Object.assertSuccess (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:140:19)
                                  at httpRequestPlus (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:207:22)
                                  at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
                                  at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
                                  at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
                                  at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:217:11)
                                  at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
                                originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3Aa4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef%3A828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef%3A1c14118b-d2e4-f178-3484-4b9c13c50bc7',
                                url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:a4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef:828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef:1c14118b-d2e4-f178-3484-4b9c13c50bc7',
                                pool_master: host {
                                  uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
                                  name_label: 'minisforum-hm80-01.servers.illenium.gg',
                                  name_description: 'Default install',
                                  memory_overhead: 1161101312,
                                  allowed_operations: [Array],
                                  current_operations: [Object],
                                  API_version_major: 2,
                                  API_version_minor: 20,
                                  API_version_vendor: 'XenSource',
                                  API_version_vendor_implementation: {},
                                  enabled: true,
                                  software_version: [Object],
                                  other_config: [Object],
                                  capabilities: [Array],
                                  cpu_configuration: {},
                                  sched_policy: 'credit',
                                  supported_bootloaders: [Array],
                                  resident_VMs: [Array],
                                  logging: {},
                                  PIFs: [Array],
                                  suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
                                  crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
                                  crashdumps: [],
                                  patches: [],
                                  updates: [],
                                  PBDs: [Array],
                                  host_CPUs: [Array],
                                  cpu_info: [Object],
                                  hostname: 'minisforum-hm80-01.servers.illenium.gg',
                                  address: '10.0.0.12',
                                  metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
                                  license_params: [Object],
                                  ha_statefiles: [],
                                  ha_network_peers: [],
                                  blobs: {},
                                  tags: [],
                                  external_auth_type: '',
                                  external_auth_service_name: '',
                                  external_auth_configuration: {},
                                  edition: 'xcp-ng',
                                  license_server: [Object],
                                  bios_strings: [Object],
                                  power_on_mode: '',
                                  power_on_config: {},
                                  local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
                                  chipset_info: [Object],
                                  PCIs: [Array],
                                  PGPUs: [Array],
                                  PUSBs: [Array],
                                  ssl_legacy: false,
                                  guest_VCPUs_params: {},
                                  display: 'enabled',
                                  virtual_hardware_platform_versions: [Array],
                                  control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
                                  updates_requiring_reboot: [],
                                  features: [],
                                  iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
                                  multipathing: false,
                                  uefi_certificates: '',
                                  certificates: [Array],
                                  editions: [Array],
                                  pending_guidances: [],
                                  tls_verification_enabled: true,
                                  last_software_update: '19700101T00:00:00Z',
                                  https_only: false
                                },
                                SR: SR {
                                  uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
                                  name_label: 'minisforum-hm80-02 SSD',
                                  name_description: '',
                                  allowed_operations: [Array],
                                  current_operations: {},
                                  VDIs: [Array],
                                  PBDs: [Array],
                                  virtual_allocation: 45583695872,
                                  physical_utilisation: 290037760,
                                  physical_size: 901115478016,
                                  type: 'ext',
                                  content_type: 'user',
                                  shared: false,
                                  other_config: [Object],
                                  tags: [],
                                  sm_config: [Object],
                                  blobs: {},
                                  local_cache_enabled: true,
                                  introduced_by: 'OpaqueRef:NULL',
                                  clustered: false,
                                  is_tools_sr: false
                                },
                                VDI: VDI {
                                  uuid: '24d348fa-a8db-45b9-a623-724e64883a55',
                                  name_label: 'XO CloudConfigDrive',
                                  name_description: '',
                                  allowed_operations: [Array],
                                  current_operations: {},
                                  SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
                                  VBDs: [],
                                  crash_dumps: [],
                                  virtual_size: 10485760,
                                  physical_utilisation: 3584,
                                  type: 'user',
                                  sharable: false,
                                  read_only: false,
                                  other_config: {},
                                  storage_lock: false,
                                  location: '24d348fa-a8db-45b9-a623-724e64883a55',
                                  managed: true,
                                  missing: false,
                                  parent: 'OpaqueRef:NULL',
                                  xenstore_data: {},
                                  sm_config: {},
                                  is_a_snapshot: false,
                                  snapshot_of: 'OpaqueRef:NULL',
                                  snapshots: [],
                                  snapshot_time: '19700101T00:00:00Z',
                                  tags: [],
                                  allow_caching: false,
                                  on_boot: 'persist',
                                  metadata_of_pool: '',
                                  metadata_latest: false,
                                  is_tools_iso: false,
                                  cbt_enabled: false
                                }
                              }
                            }
                            

                            It stills tries to use the master and results in a 404
                            I'm running XO with the following version:
                            f4b81fc6-bd36-4142-8727-97fa6e61a5df-image.png

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Online
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Please provide the commit you are using, not the version 🙂

                              TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                              • TheiLLeniumStudiosT Offline
                                TheiLLeniumStudios @olivierlambert
                                last edited by

                                @olivierlambert sure. Using commit 17b275629109160ea3840de0fc70a1faf41bd392

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Online
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Okay that's good. Are you sure you did rebuild everything?

                                  TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                                  • TheiLLeniumStudiosT Offline
                                    TheiLLeniumStudios @olivierlambert
                                    last edited by

                                    @olivierlambert said in Cannot use cloudConfig while creating VM with RPC call:

                                    Reply

                                    yes I did rebuild everything properly. I even built a new container using dockerfiles from here using the commit above: https://github.com/Ezka77/xen-orchestra-ce
                                    and then pushed and tested it and got the same result

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Online
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      Pinging @julien-f

                                      TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                                      • TheiLLeniumStudiosT Offline
                                        TheiLLeniumStudios @olivierlambert
                                        last edited by

                                        @julien-f what's weird is that the url points to the correct Host (Slave) but the originalUrl points to the Master. I'm not sure which one receives the request since it is unclear from the warning. My suspicion is that it is 1 of the 2 (assuming that the affinity host is the Slave):

                                        1. The VDI content i.e., Cloud Config gets built on the Master but when calling /import_raw_vdi it triggers the Slave endpoint i.e., url
                                        2. (Most likely) The Cloud Config gets built on the Slave but when calling /import_raw_vdi it triggers the Master endpoint i.e., originalUrl

                                        I think #2 is most likely what's happening because I see an Import VDI hanging task on the Master host for every VM that was scheduled on the Slave. I saw it through XOA in the tasks section

                                        julien-fJ 1 Reply Last reply Reply Quote 0
                                        • julien-fJ Offline
                                          julien-f Vates 🪐 Co-Founder XO Team @TheiLLeniumStudios
                                          last edited by

                                          @TheiLLeniumStudios I messed up my previous fix 🤦 , a new one has been merged yesterday: https://github.com/vatesfr/xen-orchestra/commit/01ba10fedb3c47720391c0bcd15751ac557ac3c1

                                          0 julien-f committed to vatesfr/xen-orchestra
                                          fix(xen-api/putResource): really fix (302) redirection with non-stream body
                                          
                                          Replaces the incorrect fix in 87e6f7fde
                                          
                                          Introduced by ab96c549a
                                          
                                          Fixes zammad#13375
                                          Fixes zammad#13952
                                          Fixes zammad#14001
                                          TheiLLeniumStudiosT 2 Replies Last reply Reply Quote 1
                                          • TheiLLeniumStudiosT Offline
                                            TheiLLeniumStudios @julien-f
                                            last edited by

                                            @julien-f awesome! Let me give it a try right now

                                            1 Reply Last reply Reply Quote 0
                                            • olivierlambertO olivierlambert marked this topic as a question on
                                            • First post
                                              Last post