XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Cannot use cloudConfig while creating VM with RPC call

    Scheduled Pinned Locked Moved Solved Xen Orchestra
    23 Posts 5 Posters 2.0k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • TheiLLeniumStudiosT Offline
      TheiLLeniumStudios @ddelnano
      last edited by

      @ddelnano I'm running into a very similar issue. Didn't want to create a separate issue on github so added my comments on the old one here: https://github.com/terra-farm/terraform-provider-xenorchestra/issues/227#issuecomment-1536727385

      Would really appreciate it if you could help me out investigating the root cause and fixing it. I'll be happy to try out things and provide more details if required

      sMteX created this issue in terra-farm/terraform-provider-xenorchestra

      closed Cannot use cloud_config while creating VM #227

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Is your VM created on a slave host in a pool?

        TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
        • TheiLLeniumStudiosT Offline
          TheiLLeniumStudios @olivierlambert
          last edited by TheiLLeniumStudios

          @olivierlambert Yes, the problematic VMs are on a slave host in the pool. I'm explicitly assigning affinity host + local SR for every new VM based on even odd numbering. I create a total of 6 VMs, 4 of which are on the master host which boot with the cloud-config drive just fine. 2 gets scheduled on the slave and their cloud config gets corrupted and Talos cannot read it

          TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
          • TheiLLeniumStudiosT Offline
            TheiLLeniumStudios @TheiLLeniumStudios
            last edited by

            @olivierlambert after tracing through the xen-orchestra code, I found where the problem originates from. I see this error in XO whenever a cloud config drive is being created:

            2023-05-05T22:51:22.141Z xo:xapi WARN importVdiContent:  {
  error: Error: 404 Not Found
      at Object.assertSuccess (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:138:19)
      at httpRequestPlus (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:205:22)
      at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
      at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
      at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
      at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:211:11)
      at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
    originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3A696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef%3A1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef%3Acabc6a41-7f36-2ae5-d03b-5515d77f8c60',
    url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef:1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef:cabc6a41-7f36-2ae5-d03b-5515d77f8c60',
    pool_master: host {
      uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
      name_label: 'minisforum-hm80-01.servers.illenium.gg',
      name_description: 'Default install',
      memory_overhead: 1161101312,
      allowed_operations: [Array],
      current_operations: [Object],
      API_version_major: 2,
      API_version_minor: 20,
      API_version_vendor: 'XenSource',
      API_version_vendor_implementation: {},
      enabled: true,
      software_version: [Object],
      other_config: [Object],
      capabilities: [Array],
      cpu_configuration: {},
      sched_policy: 'credit',
      supported_bootloaders: [Array],
      resident_VMs: [Array],
      logging: {},
      PIFs: [Array],
      suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crashdumps: [],
      patches: [],
      updates: [],
      PBDs: [Array],
      host_CPUs: [Array],
      cpu_info: [Object],
      hostname: 'minisforum-hm80-01.servers.illenium.gg',
      address: '10.0.0.12',
      metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
      license_params: [Object],
      ha_statefiles: [],
      ha_network_peers: [],
      blobs: {},
      tags: [],
      external_auth_type: '',
      external_auth_service_name: '',
      external_auth_configuration: {},
      edition: 'xcp-ng',
      license_server: [Object],
      bios_strings: [Object],
      power_on_mode: '',
      power_on_config: {},
      local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      chipset_info: [Object],
      PCIs: [Array],
      PGPUs: [Array],
      PUSBs: [Array],
      ssl_legacy: false,
      guest_VCPUs_params: {},
      display: 'enabled',
      virtual_hardware_platform_versions: [Array],
      control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
      updates_requiring_reboot: [],
      features: [],
      iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
      multipathing: false,
      uefi_certificates: '',
      certificates: [Array],
      editions: [Array],
      pending_guidances: [],
      tls_verification_enabled: true,
      last_software_update: '19700101T00:00:00Z',
      https_only: false
    },
    SR: SR {
      uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
      name_label: 'minisforum-hm80-02 SSD',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      VDIs: [Array],
      PBDs: [Array],
      virtual_allocation: 25404899328,
      physical_utilisation: 289996800,
      physical_size: 901115478016,
      type: 'ext',
      content_type: 'user',
      shared: false,
      other_config: [Object],
      tags: [],
      sm_config: [Object],
      blobs: {},
      local_cache_enabled: true,
      introduced_by: 'OpaqueRef:NULL',
      clustered: false,
      is_tools_sr: false
    },
    VDI: VDI {
      uuid: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
      name_label: 'XO CloudConfigDrive',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
      VBDs: [],
      crash_dumps: [],
      virtual_size: 10485760,
      physical_utilisation: 3584,
      type: 'user',
      sharable: false,
      read_only: false,
      other_config: {},
      storage_lock: false,
      location: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
      managed: true,
      missing: false,
      parent: 'OpaqueRef:NULL',
      xenstore_data: {},
      sm_config: {},
      is_a_snapshot: false,
      snapshot_of: 'OpaqueRef:NULL',
      snapshots: [],
      snapshot_time: '19700101T00:00:00Z',
      tags: [],
      allow_caching: false,
      on_boot: 'persist',
      metadata_of_pool: '',
      metadata_latest: false,
      is_tools_iso: false,
      cbt_enabled: false
    }
  }
}

            This seems to come from this specific line of code: https://github.com/vatesfr/xen-orchestra/blob/master/packages/xo-server/src/xapi/index.mjs#L1332-L1334

            And there's a comment there as well by 1 of the devs which suggests that it could be a potential bug?

            TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
            • TheiLLeniumStudiosT Offline
              TheiLLeniumStudios @TheiLLeniumStudios
              last edited by

              Basically what happens is that as a result of ignoring the warning, an empty cloudinitdrive gets created and doesn't have the correct data inside and the VM creation succeeds

              TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
              • TheiLLeniumStudiosT Offline
                TheiLLeniumStudios @TheiLLeniumStudios
                last edited by TheiLLeniumStudios

                I think importVdiContent is ignoring the SR id and is trying to clone the VDI on the master instead of the slave, hence the 404

                TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                • TheiLLeniumStudiosT Offline
                  TheiLLeniumStudios @TheiLLeniumStudios
                  last edited by 

                  await this.putResource(cancelToken, stream, '/import_raw_vdi/', {
        query: {
          format,
          vdi: ref,
        },
        task: await this.task_create(`Importing content into VDI ${await this.getField('VDI', ref, 'name_label')}`),
      })

                  This returns 404. I'm trying to understand what does /import_raw_vdi/ do? Also, this task gets created on the master host instead of the slave, even for the VMs that are scheduled on the slave host. Is that why it returns a 404? If it does, then how does the filesystem for the cloudconfigdrive get created on the slave? Maybe that needs to be moved over to the master as well? I don't know if I'm understanding currently or not so feel free to correct me if this is not how it works and my assumptions are wrong

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    The problem is already detected by @julien-f

                    Expect a fix in the next days or in our next XOA patch release next week

                    TheiLLeniumStudiosT 2 Replies Last reply Reply Quote 0
                    • TheiLLeniumStudiosT Offline
                      TheiLLeniumStudios @olivierlambert
                      last edited by

                      @olivierlambert Awesome! I use XO built from git source so will monitor the commits then. Thanks!

                      1 Reply Last reply Reply Quote 0
                      • TheiLLeniumStudiosT Offline
                        TheiLLeniumStudios @olivierlambert
                        last edited by

                        @olivierlambert Just tried the latest changes from xen-orchestra and the problem persists. I get the exact same error produced in XO when importing CloudConfigDrive on a slave:

                        2023-05-13T09:03:41.719Z xo:xapi WARN importVdiContent:  {
  error: Error: 404 Not Found
      at Object.assertSuccess (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:140:19)
      at httpRequestPlus (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:207:22)
      at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
      at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
      at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
      at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:217:11)
      at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
    originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3Aa4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef%3A828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef%3A1c14118b-d2e4-f178-3484-4b9c13c50bc7',
    url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:a4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef:828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef:1c14118b-d2e4-f178-3484-4b9c13c50bc7',
    pool_master: host {
      uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
      name_label: 'minisforum-hm80-01.servers.illenium.gg',
      name_description: 'Default install',
      memory_overhead: 1161101312,
      allowed_operations: [Array],
      current_operations: [Object],
      API_version_major: 2,
      API_version_minor: 20,
      API_version_vendor: 'XenSource',
      API_version_vendor_implementation: {},
      enabled: true,
      software_version: [Object],
      other_config: [Object],
      capabilities: [Array],
      cpu_configuration: {},
      sched_policy: 'credit',
      supported_bootloaders: [Array],
      resident_VMs: [Array],
      logging: {},
      PIFs: [Array],
      suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crashdumps: [],
      patches: [],
      updates: [],
      PBDs: [Array],
      host_CPUs: [Array],
      cpu_info: [Object],
      hostname: 'minisforum-hm80-01.servers.illenium.gg',
      address: '10.0.0.12',
      metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
      license_params: [Object],
      ha_statefiles: [],
      ha_network_peers: [],
      blobs: {},
      tags: [],
      external_auth_type: '',
      external_auth_service_name: '',
      external_auth_configuration: {},
      edition: 'xcp-ng',
      license_server: [Object],
      bios_strings: [Object],
      power_on_mode: '',
      power_on_config: {},
      local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      chipset_info: [Object],
      PCIs: [Array],
      PGPUs: [Array],
      PUSBs: [Array],
      ssl_legacy: false,
      guest_VCPUs_params: {},
      display: 'enabled',
      virtual_hardware_platform_versions: [Array],
      control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
      updates_requiring_reboot: [],
      features: [],
      iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
      multipathing: false,
      uefi_certificates: '',
      certificates: [Array],
      editions: [Array],
      pending_guidances: [],
      tls_verification_enabled: true,
      last_software_update: '19700101T00:00:00Z',
      https_only: false
    },
    SR: SR {
      uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
      name_label: 'minisforum-hm80-02 SSD',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      VDIs: [Array],
      PBDs: [Array],
      virtual_allocation: 45583695872,
      physical_utilisation: 290037760,
      physical_size: 901115478016,
      type: 'ext',
      content_type: 'user',
      shared: false,
      other_config: [Object],
      tags: [],
      sm_config: [Object],
      blobs: {},
      local_cache_enabled: true,
      introduced_by: 'OpaqueRef:NULL',
      clustered: false,
      is_tools_sr: false
    },
    VDI: VDI {
      uuid: '24d348fa-a8db-45b9-a623-724e64883a55',
      name_label: 'XO CloudConfigDrive',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
      VBDs: [],
      crash_dumps: [],
      virtual_size: 10485760,
      physical_utilisation: 3584,
      type: 'user',
      sharable: false,
      read_only: false,
      other_config: {},
      storage_lock: false,
      location: '24d348fa-a8db-45b9-a623-724e64883a55',
      managed: true,
      missing: false,
      parent: 'OpaqueRef:NULL',
      xenstore_data: {},
      sm_config: {},
      is_a_snapshot: false,
      snapshot_of: 'OpaqueRef:NULL',
      snapshots: [],
      snapshot_time: '19700101T00:00:00Z',
      tags: [],
      allow_caching: false,
      on_boot: 'persist',
      metadata_of_pool: '',
      metadata_latest: false,
      is_tools_iso: false,
      cbt_enabled: false
    }
  }
}

                        It stills tries to use the master and results in a 404
                        I'm running XO with the following version:
                        f4b81fc6-bd36-4142-8727-97fa6e61a5df-image.png

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          Please provide the commit you are using, not the version 🙂

                          TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                          • TheiLLeniumStudiosT Offline
                            TheiLLeniumStudios @olivierlambert
                            last edited by

                            @olivierlambert sure. Using commit 17b275629109160ea3840de0fc70a1faf41bd392

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Okay that's good. Are you sure you did rebuild everything?

                              TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                              • TheiLLeniumStudiosT Offline
                                TheiLLeniumStudios @olivierlambert
                                last edited by

                                @olivierlambert said in Cannot use cloudConfig while creating VM with RPC call:

                                Reply

                                yes I did rebuild everything properly. I even built a new container using dockerfiles from here using the commit above: https://github.com/Ezka77/xen-orchestra-ce
                                and then pushed and tested it and got the same result

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Pinging @julien-f

                                  TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                                  • TheiLLeniumStudiosT Offline
                                    TheiLLeniumStudios @olivierlambert
                                    last edited by

                                    @julien-f what's weird is that the url points to the correct Host (Slave) but the originalUrl points to the Master. I'm not sure which one receives the request since it is unclear from the warning. My suspicion is that it is 1 of the 2 (assuming that the affinity host is the Slave):

                                    1. The VDI content i.e., Cloud Config gets built on the Master but when calling /import_raw_vdi it triggers the Slave endpoint i.e., url
                                    2. (Most likely) The Cloud Config gets built on the Slave but when calling /import_raw_vdi it triggers the Master endpoint i.e., originalUrl

                                    I think #2 is most likely what's happening because I see an Import VDI hanging task on the Master host for every VM that was scheduled on the Slave. I saw it through XOA in the tasks section

                                    julien-fJ 1 Reply Last reply Reply Quote 0
                                    • julien-fJ Offline
                                      julien-f Vates 🪐 Co-Founder XO Team @TheiLLeniumStudios
                                      last edited by

                                      @TheiLLeniumStudios I messed up my previous fix 🤦 , a new one has been merged yesterday: https://github.com/vatesfr/xen-orchestra/commit/01ba10fedb3c47720391c0bcd15751ac557ac3c1

                                      0 julien-f committed to vatesfr/xen-orchestra 
                                      fix(xen-api/putResource): really fix (302) redirection with non-stream body

Replaces the incorrect fix in 87e6f7fde

Introduced by ab96c549a

Fixes zammad#13375
Fixes zammad#13952
Fixes zammad#14001
                                      TheiLLeniumStudiosT 2 Replies Last reply Reply Quote 1
                                      • TheiLLeniumStudiosT Offline
                                        TheiLLeniumStudios @julien-f
                                        last edited by

                                        @julien-f awesome! Let me give it a try right now

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO olivierlambert marked this topic as a question on
                                        • TheiLLeniumStudiosT Offline
                                          TheiLLeniumStudios @julien-f
                                          last edited by

                                          @julien-f The problem seems to be fixed now! Thank you for the fix 😄

                                          1 Reply Last reply Reply Quote 1
                                          • olivierlambertO olivierlambert has marked this topic as solved on
                                          • First post
                                            Last post