XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Cannot use cloudConfig while creating VM with RPC call

    Scheduled Pinned Locked Moved Solved Xen Orchestra
    23 Posts 5 Posters 2.1k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      That's a question of for @ddelnano

      1 Reply Last reply Reply Quote 0
      • sMteXS Offline
        sMteX
        last edited by

        Have you had a chance to take a look please? @ddelnano

        D 1 Reply Last reply Reply Quote 0
        • D Offline
          ddelnano Terraform Team @sMteX
          last edited by

          @sMteX sorry for the late reply. I responded to your GitHub issue with some questions and comments.

          Let's keep the discussion there, and we can summarize the conclusion here once we get to the bottom of it.

          TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 1
          • TheiLLeniumStudiosT Offline
            TheiLLeniumStudios @ddelnano
            last edited by

            @ddelnano I'm running into a very similar issue. Didn't want to create a separate issue on github so added my comments on the old one here: https://github.com/terra-farm/terraform-provider-xenorchestra/issues/227#issuecomment-1536727385

            Would really appreciate it if you could help me out investigating the root cause and fixing it. I'll be happy to try out things and provide more details if required

            sMteX created this issue in terra-farm/terraform-provider-xenorchestra

            closed Cannot use cloud_config while creating VM #227

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Is your VM created on a slave host in a pool?

              TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
              • TheiLLeniumStudiosT Offline
                TheiLLeniumStudios @olivierlambert
                last edited by TheiLLeniumStudios

                @olivierlambert Yes, the problematic VMs are on a slave host in the pool. I'm explicitly assigning affinity host + local SR for every new VM based on even odd numbering. I create a total of 6 VMs, 4 of which are on the master host which boot with the cloud-config drive just fine. 2 gets scheduled on the slave and their cloud config gets corrupted and Talos cannot read it

                TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                • TheiLLeniumStudiosT Offline
                  TheiLLeniumStudios @TheiLLeniumStudios
                  last edited by

                  @olivierlambert after tracing through the xen-orchestra code, I found where the problem originates from. I see this error in XO whenever a cloud config drive is being created:

                  2023-05-05T22:51:22.141Z xo:xapi WARN importVdiContent:  {
  error: Error: 404 Not Found
      at Object.assertSuccess (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:138:19)
      at httpRequestPlus (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:205:22)
      at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
      at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
      at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
      at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:211:11)
      at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
    originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3A696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef%3A1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef%3Acabc6a41-7f36-2ae5-d03b-5515d77f8c60',
    url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef:1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef:cabc6a41-7f36-2ae5-d03b-5515d77f8c60',
    pool_master: host {
      uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
      name_label: 'minisforum-hm80-01.servers.illenium.gg',
      name_description: 'Default install',
      memory_overhead: 1161101312,
      allowed_operations: [Array],
      current_operations: [Object],
      API_version_major: 2,
      API_version_minor: 20,
      API_version_vendor: 'XenSource',
      API_version_vendor_implementation: {},
      enabled: true,
      software_version: [Object],
      other_config: [Object],
      capabilities: [Array],
      cpu_configuration: {},
      sched_policy: 'credit',
      supported_bootloaders: [Array],
      resident_VMs: [Array],
      logging: {},
      PIFs: [Array],
      suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crashdumps: [],
      patches: [],
      updates: [],
      PBDs: [Array],
      host_CPUs: [Array],
      cpu_info: [Object],
      hostname: 'minisforum-hm80-01.servers.illenium.gg',
      address: '10.0.0.12',
      metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
      license_params: [Object],
      ha_statefiles: [],
      ha_network_peers: [],
      blobs: {},
      tags: [],
      external_auth_type: '',
      external_auth_service_name: '',
      external_auth_configuration: {},
      edition: 'xcp-ng',
      license_server: [Object],
      bios_strings: [Object],
      power_on_mode: '',
      power_on_config: {},
      local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      chipset_info: [Object],
      PCIs: [Array],
      PGPUs: [Array],
      PUSBs: [Array],
      ssl_legacy: false,
      guest_VCPUs_params: {},
      display: 'enabled',
      virtual_hardware_platform_versions: [Array],
      control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
      updates_requiring_reboot: [],
      features: [],
      iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
      multipathing: false,
      uefi_certificates: '',
      certificates: [Array],
      editions: [Array],
      pending_guidances: [],
      tls_verification_enabled: true,
      last_software_update: '19700101T00:00:00Z',
      https_only: false
    },
    SR: SR {
      uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
      name_label: 'minisforum-hm80-02 SSD',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      VDIs: [Array],
      PBDs: [Array],
      virtual_allocation: 25404899328,
      physical_utilisation: 289996800,
      physical_size: 901115478016,
      type: 'ext',
      content_type: 'user',
      shared: false,
      other_config: [Object],
      tags: [],
      sm_config: [Object],
      blobs: {},
      local_cache_enabled: true,
      introduced_by: 'OpaqueRef:NULL',
      clustered: false,
      is_tools_sr: false
    },
    VDI: VDI {
      uuid: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
      name_label: 'XO CloudConfigDrive',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
      VBDs: [],
      crash_dumps: [],
      virtual_size: 10485760,
      physical_utilisation: 3584,
      type: 'user',
      sharable: false,
      read_only: false,
      other_config: {},
      storage_lock: false,
      location: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
      managed: true,
      missing: false,
      parent: 'OpaqueRef:NULL',
      xenstore_data: {},
      sm_config: {},
      is_a_snapshot: false,
      snapshot_of: 'OpaqueRef:NULL',
      snapshots: [],
      snapshot_time: '19700101T00:00:00Z',
      tags: [],
      allow_caching: false,
      on_boot: 'persist',
      metadata_of_pool: '',
      metadata_latest: false,
      is_tools_iso: false,
      cbt_enabled: false
    }
  }
}

                  This seems to come from this specific line of code: https://github.com/vatesfr/xen-orchestra/blob/master/packages/xo-server/src/xapi/index.mjs#L1332-L1334

                  And there's a comment there as well by 1 of the devs which suggests that it could be a potential bug?

                  TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                  • TheiLLeniumStudiosT Offline
                    TheiLLeniumStudios @TheiLLeniumStudios
                    last edited by

                    Basically what happens is that as a result of ignoring the warning, an empty cloudinitdrive gets created and doesn't have the correct data inside and the VM creation succeeds

                    TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                    • TheiLLeniumStudiosT Offline
                      TheiLLeniumStudios @TheiLLeniumStudios
                      last edited by TheiLLeniumStudios

                      I think importVdiContent is ignoring the SR id and is trying to clone the VDI on the master instead of the slave, hence the 404

                      TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                      • TheiLLeniumStudiosT Offline
                        TheiLLeniumStudios @TheiLLeniumStudios
                        last edited by 

                        await this.putResource(cancelToken, stream, '/import_raw_vdi/', {
        query: {
          format,
          vdi: ref,
        },
        task: await this.task_create(`Importing content into VDI ${await this.getField('VDI', ref, 'name_label')}`),
      })

                        This returns 404. I'm trying to understand what does /import_raw_vdi/ do? Also, this task gets created on the master host instead of the slave, even for the VMs that are scheduled on the slave host. Is that why it returns a 404? If it does, then how does the filesystem for the cloudconfigdrive get created on the slave? Maybe that needs to be moved over to the master as well? I don't know if I'm understanding currently or not so feel free to correct me if this is not how it works and my assumptions are wrong

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          The problem is already detected by @julien-f

                          Expect a fix in the next days or in our next XOA patch release next week

                          TheiLLeniumStudiosT 2 Replies Last reply Reply Quote 0
                          • TheiLLeniumStudiosT Offline
                            TheiLLeniumStudios @olivierlambert
                            last edited by

                            @olivierlambert Awesome! I use XO built from git source so will monitor the commits then. Thanks!

                            1 Reply Last reply Reply Quote 0
                            • TheiLLeniumStudiosT Offline
                              TheiLLeniumStudios @olivierlambert
                              last edited by

                              @olivierlambert Just tried the latest changes from xen-orchestra and the problem persists. I get the exact same error produced in XO when importing CloudConfigDrive on a slave:

                              2023-05-13T09:03:41.719Z xo:xapi WARN importVdiContent:  {
  error: Error: 404 Not Found
      at Object.assertSuccess (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:140:19)
      at httpRequestPlus (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:207:22)
      at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
      at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
      at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
      at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:217:11)
      at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
    originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3Aa4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef%3A828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef%3A1c14118b-d2e4-f178-3484-4b9c13c50bc7',
    url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:a4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef:828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef:1c14118b-d2e4-f178-3484-4b9c13c50bc7',
    pool_master: host {
      uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
      name_label: 'minisforum-hm80-01.servers.illenium.gg',
      name_description: 'Default install',
      memory_overhead: 1161101312,
      allowed_operations: [Array],
      current_operations: [Object],
      API_version_major: 2,
      API_version_minor: 20,
      API_version_vendor: 'XenSource',
      API_version_vendor_implementation: {},
      enabled: true,
      software_version: [Object],
      other_config: [Object],
      capabilities: [Array],
      cpu_configuration: {},
      sched_policy: 'credit',
      supported_bootloaders: [Array],
      resident_VMs: [Array],
      logging: {},
      PIFs: [Array],
      suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      crashdumps: [],
      patches: [],
      updates: [],
      PBDs: [Array],
      host_CPUs: [Array],
      cpu_info: [Object],
      hostname: 'minisforum-hm80-01.servers.illenium.gg',
      address: '10.0.0.12',
      metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
      license_params: [Object],
      ha_statefiles: [],
      ha_network_peers: [],
      blobs: {},
      tags: [],
      external_auth_type: '',
      external_auth_service_name: '',
      external_auth_configuration: {},
      edition: 'xcp-ng',
      license_server: [Object],
      bios_strings: [Object],
      power_on_mode: '',
      power_on_config: {},
      local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
      chipset_info: [Object],
      PCIs: [Array],
      PGPUs: [Array],
      PUSBs: [Array],
      ssl_legacy: false,
      guest_VCPUs_params: {},
      display: 'enabled',
      virtual_hardware_platform_versions: [Array],
      control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
      updates_requiring_reboot: [],
      features: [],
      iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
      multipathing: false,
      uefi_certificates: '',
      certificates: [Array],
      editions: [Array],
      pending_guidances: [],
      tls_verification_enabled: true,
      last_software_update: '19700101T00:00:00Z',
      https_only: false
    },
    SR: SR {
      uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
      name_label: 'minisforum-hm80-02 SSD',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      VDIs: [Array],
      PBDs: [Array],
      virtual_allocation: 45583695872,
      physical_utilisation: 290037760,
      physical_size: 901115478016,
      type: 'ext',
      content_type: 'user',
      shared: false,
      other_config: [Object],
      tags: [],
      sm_config: [Object],
      blobs: {},
      local_cache_enabled: true,
      introduced_by: 'OpaqueRef:NULL',
      clustered: false,
      is_tools_sr: false
    },
    VDI: VDI {
      uuid: '24d348fa-a8db-45b9-a623-724e64883a55',
      name_label: 'XO CloudConfigDrive',
      name_description: '',
      allowed_operations: [Array],
      current_operations: {},
      SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
      VBDs: [],
      crash_dumps: [],
      virtual_size: 10485760,
      physical_utilisation: 3584,
      type: 'user',
      sharable: false,
      read_only: false,
      other_config: {},
      storage_lock: false,
      location: '24d348fa-a8db-45b9-a623-724e64883a55',
      managed: true,
      missing: false,
      parent: 'OpaqueRef:NULL',
      xenstore_data: {},
      sm_config: {},
      is_a_snapshot: false,
      snapshot_of: 'OpaqueRef:NULL',
      snapshots: [],
      snapshot_time: '19700101T00:00:00Z',
      tags: [],
      allow_caching: false,
      on_boot: 'persist',
      metadata_of_pool: '',
      metadata_latest: false,
      is_tools_iso: false,
      cbt_enabled: false
    }
  }
}

                              It stills tries to use the master and results in a 404
                              I'm running XO with the following version:
                              f4b81fc6-bd36-4142-8727-97fa6e61a5df-image.png

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Please provide the commit you are using, not the version 🙂

                                TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                                • TheiLLeniumStudiosT Offline
                                  TheiLLeniumStudios @olivierlambert
                                  last edited by

                                  @olivierlambert sure. Using commit 17b275629109160ea3840de0fc70a1faf41bd392

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    Okay that's good. Are you sure you did rebuild everything?

                                    TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                                    • TheiLLeniumStudiosT Offline
                                      TheiLLeniumStudios @olivierlambert
                                      last edited by

                                      @olivierlambert said in Cannot use cloudConfig while creating VM with RPC call:

                                      Reply

                                      yes I did rebuild everything properly. I even built a new container using dockerfiles from here using the commit above: https://github.com/Ezka77/xen-orchestra-ce
                                      and then pushed and tested it and got the same result

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        Pinging @julien-f

                                        TheiLLeniumStudiosT 1 Reply Last reply Reply Quote 0
                                        • TheiLLeniumStudiosT Offline
                                          TheiLLeniumStudios @olivierlambert
                                          last edited by

                                          @julien-f what's weird is that the url points to the correct Host (Slave) but the originalUrl points to the Master. I'm not sure which one receives the request since it is unclear from the warning. My suspicion is that it is 1 of the 2 (assuming that the affinity host is the Slave):

                                          1. The VDI content i.e., Cloud Config gets built on the Master but when calling /import_raw_vdi it triggers the Slave endpoint i.e., url
                                          2. (Most likely) The Cloud Config gets built on the Slave but when calling /import_raw_vdi it triggers the Master endpoint i.e., originalUrl

                                          I think #2 is most likely what's happening because I see an Import VDI hanging task on the Master host for every VM that was scheduled on the Slave. I saw it through XOA in the tasks section

                                          julien-fJ 1 Reply Last reply Reply Quote 0
                                          • julien-fJ Offline
                                            julien-f Vates 🪐 Co-Founder XO Team @TheiLLeniumStudios
                                            last edited by

                                            @TheiLLeniumStudios I messed up my previous fix 🤦 , a new one has been merged yesterday: https://github.com/vatesfr/xen-orchestra/commit/01ba10fedb3c47720391c0bcd15751ac557ac3c1

                                            0 julien-f committed to vatesfr/xen-orchestra 
                                            fix(xen-api/putResource): really fix (302) redirection with non-stream body

Replaces the incorrect fix in 87e6f7fde

Introduced by ab96c549a

Fixes zammad#13375
Fixes zammad#13952
Fixes zammad#14001
                                            TheiLLeniumStudiosT 2 Replies Last reply Reply Quote 1
                                            • First post
                                              Last post