Offline
@yann Hi yann. I tried it out and it works perfectly fine on a normal VM. Are there any tips to run it as a container instead? With Talos I cannot really run it as a process as the OS is immutable so I need to run it as a container. I used to run this containerized version but it seems to be very old: https://github.com/GeoMSK/ros-xe-guest-utilities
Any tips on what would I need to expose to the container itself from the host, to be able to get it to work would be appreciated
@julien-f The problem seems to be fixed now! Thank you for the fix 
@julien-f awesome! Let me give it a try right now
@julien-f what's weird is that the url points to the correct Host (Slave) but the originalUrl points to the Master. I'm not sure which one receives the request since it is unclear from the warning. My suspicion is that it is 1 of the 2 (assuming that the affinity host is the Slave):
urloriginalUrlI think #2 is most likely what's happening because I see an Import VDI hanging task on the Master host for every VM that was scheduled on the Slave. I saw it through XOA in the tasks section
@olivierlambert said in Cannot use cloudConfig while creating VM with RPC call:
Reply
yes I did rebuild everything properly. I even built a new container using dockerfiles from here using the commit above: https://github.com/Ezka77/xen-orchestra-ce
and then pushed and tested it and got the same result
@olivierlambert sure. Using commit 17b275629109160ea3840de0fc70a1faf41bd392
@olivierlambert Just tried the latest changes from xen-orchestra and the problem persists. I get the exact same error produced in XO when importing CloudConfigDrive on a slave:
2023-05-13T09:03:41.719Z xo:xapi WARN importVdiContent: {
error: Error: 404 Not Found
at Object.assertSuccess (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:140:19)
at httpRequestPlus (/home/node/xen-orchestra/packages/xen-api/node_modules/http-request-plus/index.js:207:22)
at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:217:11)
at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3Aa4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef%3A828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef%3A1c14118b-d2e4-f178-3484-4b9c13c50bc7',
url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:a4c7cb1a-8e78-4da2-b35a-f3b113ebbb6c&session_id=OpaqueRef:828e4212-69aa-2987-bbc4-4e52365ce42a&task_id=OpaqueRef:1c14118b-d2e4-f178-3484-4b9c13c50bc7',
pool_master: host {
uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
name_label: 'minisforum-hm80-01.servers.illenium.gg',
name_description: 'Default install',
memory_overhead: 1161101312,
allowed_operations: [Array],
current_operations: [Object],
API_version_major: 2,
API_version_minor: 20,
API_version_vendor: 'XenSource',
API_version_vendor_implementation: {},
enabled: true,
software_version: [Object],
other_config: [Object],
capabilities: [Array],
cpu_configuration: {},
sched_policy: 'credit',
supported_bootloaders: [Array],
resident_VMs: [Array],
logging: {},
PIFs: [Array],
suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
crashdumps: [],
patches: [],
updates: [],
PBDs: [Array],
host_CPUs: [Array],
cpu_info: [Object],
hostname: 'minisforum-hm80-01.servers.illenium.gg',
address: '10.0.0.12',
metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
license_params: [Object],
ha_statefiles: [],
ha_network_peers: [],
blobs: {},
tags: [],
external_auth_type: '',
external_auth_service_name: '',
external_auth_configuration: {},
edition: 'xcp-ng',
license_server: [Object],
bios_strings: [Object],
power_on_mode: '',
power_on_config: {},
local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
chipset_info: [Object],
PCIs: [Array],
PGPUs: [Array],
PUSBs: [Array],
ssl_legacy: false,
guest_VCPUs_params: {},
display: 'enabled',
virtual_hardware_platform_versions: [Array],
control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
updates_requiring_reboot: [],
features: [],
iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
multipathing: false,
uefi_certificates: '',
certificates: [Array],
editions: [Array],
pending_guidances: [],
tls_verification_enabled: true,
last_software_update: '19700101T00:00:00Z',
https_only: false
},
SR: SR {
uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
name_label: 'minisforum-hm80-02 SSD',
name_description: '',
allowed_operations: [Array],
current_operations: {},
VDIs: [Array],
PBDs: [Array],
virtual_allocation: 45583695872,
physical_utilisation: 290037760,
physical_size: 901115478016,
type: 'ext',
content_type: 'user',
shared: false,
other_config: [Object],
tags: [],
sm_config: [Object],
blobs: {},
local_cache_enabled: true,
introduced_by: 'OpaqueRef:NULL',
clustered: false,
is_tools_sr: false
},
VDI: VDI {
uuid: '24d348fa-a8db-45b9-a623-724e64883a55',
name_label: 'XO CloudConfigDrive',
name_description: '',
allowed_operations: [Array],
current_operations: {},
SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
VBDs: [],
crash_dumps: [],
virtual_size: 10485760,
physical_utilisation: 3584,
type: 'user',
sharable: false,
read_only: false,
other_config: {},
storage_lock: false,
location: '24d348fa-a8db-45b9-a623-724e64883a55',
managed: true,
missing: false,
parent: 'OpaqueRef:NULL',
xenstore_data: {},
sm_config: {},
is_a_snapshot: false,
snapshot_of: 'OpaqueRef:NULL',
snapshots: [],
snapshot_time: '19700101T00:00:00Z',
tags: [],
allow_caching: false,
on_boot: 'persist',
metadata_of_pool: '',
metadata_latest: false,
is_tools_iso: false,
cbt_enabled: false
}
}
}
It stills tries to use the master and results in a 404
I'm running XO with the following version:
@olivierlambert Awesome! I use XO built from git source so will monitor the commits then. Thanks!
await this.putResource(cancelToken, stream, '/import_raw_vdi/', {
query: {
format,
vdi: ref,
},
task: await this.task_create(`Importing content into VDI ${await this.getField('VDI', ref, 'name_label')}`),
})
This returns 404. I'm trying to understand what does /import_raw_vdi/ do? Also, this task gets created on the master host instead of the slave, even for the VMs that are scheduled on the slave host. Is that why it returns a 404? If it does, then how does the filesystem for the cloudconfigdrive get created on the slave? Maybe that needs to be moved over to the master as well? I don't know if I'm understanding currently or not so feel free to correct me if this is not how it works and my assumptions are wrong
I think importVdiContent is ignoring the SR id and is trying to clone the VDI on the master instead of the slave, hence the 404
Basically what happens is that as a result of ignoring the warning, an empty cloudinitdrive gets created and doesn't have the correct data inside and the VM creation succeeds
@olivierlambert after tracing through the xen-orchestra code, I found where the problem originates from. I see this error in XO whenever a cloud config drive is being created:
2023-05-05T22:51:22.141Z xo:xapi WARN importVdiContent: {
error: Error: 404 Not Found
at Object.assertSuccess (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:138:19)
at httpRequestPlus (/home/node/xen-orchestra/node_modules/http-request-plus/index.js:205:22)
at Xapi.putResource (/home/node/xen-orchestra/packages/xen-api/src/index.js:508:22)
at Xapi.importContent (/home/node/xen-orchestra/@xen-orchestra/xapi/vdi.js:138:7)
at Xapi.createCloudInitConfigDrive (file:///home/node/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1332:5)
at Xo.<anonymous> (file:///home/node/xen-orchestra/packages/xo-server/src/api/vm.mjs:211:11)
at Api.#callApiMethod (file:///home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:417:20) {
originalUrl: 'https://10.0.0.12/import_raw_vdi/?format=raw&vdi=OpaqueRef%3A696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef%3A1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef%3Acabc6a41-7f36-2ae5-d03b-5515d77f8c60',
url: 'https://10.0.0.13/import_raw_vdi/?format=raw&vdi=OpaqueRef:696fd7c8-2560-5bca-3047-fcd53b905725&session_id=OpaqueRef:1d0c35c4-1107-aea1-02c9-352d7f6a0a5c&task_id=OpaqueRef:cabc6a41-7f36-2ae5-d03b-5515d77f8c60',
pool_master: host {
uuid: 'd1e916dc-8f8a-4aec-98c1-206c4c8144b0',
name_label: 'minisforum-hm80-01.servers.illenium.gg',
name_description: 'Default install',
memory_overhead: 1161101312,
allowed_operations: [Array],
current_operations: [Object],
API_version_major: 2,
API_version_minor: 20,
API_version_vendor: 'XenSource',
API_version_vendor_implementation: {},
enabled: true,
software_version: [Object],
other_config: [Object],
capabilities: [Array],
cpu_configuration: {},
sched_policy: 'credit',
supported_bootloaders: [Array],
resident_VMs: [Array],
logging: {},
PIFs: [Array],
suspend_image_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
crash_dump_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
crashdumps: [],
patches: [],
updates: [],
PBDs: [Array],
host_CPUs: [Array],
cpu_info: [Object],
hostname: 'minisforum-hm80-01.servers.illenium.gg',
address: '10.0.0.12',
metrics: 'OpaqueRef:655a27bd-de06-75cb-d231-8460d949d70a',
license_params: [Object],
ha_statefiles: [],
ha_network_peers: [],
blobs: {},
tags: [],
external_auth_type: '',
external_auth_service_name: '',
external_auth_configuration: {},
edition: 'xcp-ng',
license_server: [Object],
bios_strings: [Object],
power_on_mode: '',
power_on_config: {},
local_cache_sr: 'OpaqueRef:b90d54ba-44e7-8181-a4a2-6593276e73cd',
chipset_info: [Object],
PCIs: [Array],
PGPUs: [Array],
PUSBs: [Array],
ssl_legacy: false,
guest_VCPUs_params: {},
display: 'enabled',
virtual_hardware_platform_versions: [Array],
control_domain: 'OpaqueRef:58fb5c9b-18c4-6578-803a-d9b8db989ec9',
updates_requiring_reboot: [],
features: [],
iscsi_iqn: 'iqn.2023-03.gg.illenium.servers:6e94cf19',
multipathing: false,
uefi_certificates: '',
certificates: [Array],
editions: [Array],
pending_guidances: [],
tls_verification_enabled: true,
last_software_update: '19700101T00:00:00Z',
https_only: false
},
SR: SR {
uuid: '8c7e88d7-25b4-02e2-5a4b-6e1e3ead70cf',
name_label: 'minisforum-hm80-02 SSD',
name_description: '',
allowed_operations: [Array],
current_operations: {},
VDIs: [Array],
PBDs: [Array],
virtual_allocation: 25404899328,
physical_utilisation: 289996800,
physical_size: 901115478016,
type: 'ext',
content_type: 'user',
shared: false,
other_config: [Object],
tags: [],
sm_config: [Object],
blobs: {},
local_cache_enabled: true,
introduced_by: 'OpaqueRef:NULL',
clustered: false,
is_tools_sr: false
},
VDI: VDI {
uuid: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
name_label: 'XO CloudConfigDrive',
name_description: '',
allowed_operations: [Array],
current_operations: {},
SR: 'OpaqueRef:13a829ed-3408-f42e-695e-f14adf469fb3',
VBDs: [],
crash_dumps: [],
virtual_size: 10485760,
physical_utilisation: 3584,
type: 'user',
sharable: false,
read_only: false,
other_config: {},
storage_lock: false,
location: 'c8ccacc8-38bc-4d46-9649-ecfb65e24da6',
managed: true,
missing: false,
parent: 'OpaqueRef:NULL',
xenstore_data: {},
sm_config: {},
is_a_snapshot: false,
snapshot_of: 'OpaqueRef:NULL',
snapshots: [],
snapshot_time: '19700101T00:00:00Z',
tags: [],
allow_caching: false,
on_boot: 'persist',
metadata_of_pool: '',
metadata_latest: false,
is_tools_iso: false,
cbt_enabled: false
}
}
}
This seems to come from this specific line of code: https://github.com/vatesfr/xen-orchestra/blob/master/packages/xo-server/src/xapi/index.mjs#L1332-L1334
And there's a comment there as well by 1 of the devs which suggests that it could be a potential bug?
@olivierlambert Yes, the problematic VMs are on a slave host in the pool. I'm explicitly assigning affinity host + local SR for every new VM based on even odd numbering. I create a total of 6 VMs, 4 of which are on the master host which boot with the cloud-config drive just fine. 2 gets scheduled on the slave and their cloud config gets corrupted and Talos cannot read it
@ddelnano I'm running into a very similar issue. Didn't want to create a separate issue on github so added my comments on the old one here: https://github.com/terra-farm/terraform-provider-xenorchestra/issues/227#issuecomment-1536727385
Would really appreciate it if you could help me out investigating the root cause and fixing it. I'll be happy to try out things and provide more details if required
Is it possible to only use 2 hosts for XOSTOR?
@olivierlambert No worries 
I'll wait for a response from the team
Inspecting the xensource.log file, I can see this in the logs:
May 1 13:05:02 minisforum-hm80-01 xapi: [debug||23091959 HTTPS 10.0.0.10->:::80|[XO] Importing content into VDI XO CloudConfigDrive R:ca2ba12b1631|taskhelper] the status of R:ca2ba12b1631 is: success; cannot set it to `success
But there's no error or anything that indicates why it cannot set it to success
Maybe it is related to the provider not gracefully performing all the operations? Not really sure though
I'm also running into a weird issue with using a VM template that has the disks on a local SR of 1 of the nodes of a pool and weirdly enough, whenever I create VMs on a different node, the cloudconfig works fine, but not for the VMs created on the same node as the VM template is on. To give some context, this is how I'm creating the VMs:
data "xenorchestra_pool" "pool" {
name_label = var.xo_pool
}
data "xenorchestra_hosts" "hosts" {
pool_id = data.xenorchestra_pool.pool.id
sort_by = "name_label"
sort_order = "asc"
}
data "xenorchestra_sr" "local_storage" {
count = length(data.xenorchestra_hosts.hosts.hosts)
name_label = format("%s %s", split(".", data.xenorchestra_hosts.hosts.hosts[count.index].name_label)[0], var.xo_storage_tier)
pool_id = data.xenorchestra_pool.pool.id
}
data "xenorchestra_template" "template" {
name_label = var.xo_vm_template
pool_id = data.xenorchestra_pool.pool.id
}
data "xenorchestra_network" "net" {
name_label = var.xo_vm_network
pool_id = data.xenorchestra_pool.pool.id
}
resource "xenorchestra_vm" "controlplane" {
count = var.master_count
memory_max = var.vm_memory * 1024 * 1024 * 1024
cpus = var.vm_cpu
name_label = "${var.talos_cluster_name}-master-${count.index + 1}"
template = data.xenorchestra_template.template.id
cloud_config = data.talos_machine_configuration.controlplane[count.index].machine_configuration
affinity_host = data.xenorchestra_hosts.hosts.hosts[count.index % length(data.xenorchestra_hosts.hosts.hosts)].id
network {
network_id = data.xenorchestra_network.net.id
#mac_address = var.master_macs[count.index]
}
disk {
sr_id = data.xenorchestra_sr.local_storage[count.index % length(data.xenorchestra_sr.local_storage)].id
name_label = "${var.talos_cluster_name}-master-${count.index + 1}-disk1"
size = var.vm_disk * 1024 * 1024 * 1024
}
tags = [
var.talos_cluster_name,
"controlplane"
]
}
I'm using the pool to get all the available hosts and then based on the vm count, assigning a specific host (Doing this because I don't have centralized storage at the moment and XOSTOR seems to not work well with 2 hosts)
After the VMs are created, I see that all of them get the cloud config drive attached to them but only the ones on host1 use the cloud config, host2 just ignores it entirely. Could it be related to the VM template being on different host SR?
@olivierlambert Awesome! Can't wait to take a look.
Also as a side-note. I've been playing around with cloud configs and creating VMs via terraform and ran into some weird hanging tasks:
Any idea how do I debug this and see what's causing these lockups?