Any updates on the new management agent? (Talos compatible)

TheiLLeniumStudios

Hi,

I've been working on creating terraform modules for HA Kubernetes via Talos. I'm wondering if it is possible to install the management agent on Talos somehow? Because I need the IPs to be reported so that I can rely on DHCP and bootstrap the cluster. Right now I have to provide static MAC addresses to the VMs and have static IPs via DHCP which is not ideal since automating the DNS is a bit tricky in my setup. Any idea how I can achieve this? I looked the recent post that mentioned a new agent being worked on that could support Talos but couldn't find the repo for it

TheiLLeniumStudios

There's also this project that supports Talos but is for ESXi: https://github.com/mologie/talos-vmtoolsd

It seems like an easy thing to port it to Xen but I'm not sure what API's I'd have to use in place of the govmomi. Any pointers would be appreciated

olivierlambert

Adding @yann in the convo

TheiLLeniumStudios

@olivierlambert Another thing that I tried as an alternative was to use Talos's nocloud images. I was able to create a template out of it but the network-config passed at VM creation wasn't recognized and failed with this error:

The network-config that I used was:

network:
  version: 1
  config:
    - type: physical
      name: eth0
      subnets:
        - type: static
          address: 10.2.0.5/24
          gateway: 10.2.0.1
          dns_nameservers:
            - 192.168.10.222
            - 1.1.1.1
            - 8.8.8.8

Talos also provides documentation around this and seems like the format for network-config is correct. I was wondering if the cloud-init cdrom image needs to be built in a different way to be recognized properly? Here is what their document mentions about building the ISO image:

Does xcp-ng / XO build the image in a different way than above?

TheiLLeniumStudios

Okay so it seems like Talos doesn't like network-config in the cloudconfig ISO. I was able to just use user-data to pass in the Talos machine config with hardcoded network configuration per machine and that worked

olivierlambert

Thanks for the feedback. @yann will give you access to the Rust agent so you can play with it. Frankly, it works pretty well and I'm not really concerned about it

TheiLLeniumStudios

@olivierlambert Awesome! Can't wait to take a look.

Also as a side-note. I've been playing around with cloud configs and creating VMs via terraform and ran into some weird hanging tasks:

Any idea how do I debug this and see what's causing these lockups?

TheiLLeniumStudios

I'm also running into a weird issue with using a VM template that has the disks on a local SR of 1 of the nodes of a pool and weirdly enough, whenever I create VMs on a different node, the cloudconfig works fine, but not for the VMs created on the same node as the VM template is on. To give some context, this is how I'm creating the VMs:

data "xenorchestra_pool" "pool" {
  name_label = var.xo_pool
}

data "xenorchestra_hosts" "hosts" {
  pool_id = data.xenorchestra_pool.pool.id

  sort_by = "name_label"
  sort_order = "asc"
}

data "xenorchestra_sr" "local_storage" {
  count = length(data.xenorchestra_hosts.hosts.hosts)
  name_label = format("%s %s", split(".", data.xenorchestra_hosts.hosts.hosts[count.index].name_label)[0], var.xo_storage_tier)
  pool_id = data.xenorchestra_pool.pool.id
}

data "xenorchestra_template" "template" {
    name_label = var.xo_vm_template
    pool_id = data.xenorchestra_pool.pool.id
}

data "xenorchestra_network" "net" {
  name_label = var.xo_vm_network
  pool_id = data.xenorchestra_pool.pool.id
}

resource "xenorchestra_vm" "controlplane" {
    count = var.master_count
    memory_max = var.vm_memory * 1024 * 1024 * 1024
    cpus  = var.vm_cpu
    name_label = "${var.talos_cluster_name}-master-${count.index + 1}"
    template = data.xenorchestra_template.template.id

    cloud_config = data.talos_machine_configuration.controlplane[count.index].machine_configuration

    affinity_host = data.xenorchestra_hosts.hosts.hosts[count.index % length(data.xenorchestra_hosts.hosts.hosts)].id
    network {
      network_id = data.xenorchestra_network.net.id
      #mac_address = var.master_macs[count.index]
    }

    disk {
      sr_id = data.xenorchestra_sr.local_storage[count.index % length(data.xenorchestra_sr.local_storage)].id
      name_label = "${var.talos_cluster_name}-master-${count.index + 1}-disk1"
      size = var.vm_disk * 1024 * 1024 * 1024 
    }

    tags = [
      var.talos_cluster_name,
      "controlplane"
    ]
}

I'm using the pool to get all the available hosts and then based on the vm count, assigning a specific host (Doing this because I don't have centralized storage at the moment and XOSTOR seems to not work well with 2 hosts)

After the VMs are created, I see that all of them get the cloud config drive attached to them but only the ones on host1 use the cloud config, host2 just ignores it entirely. Could it be related to the VM template being on different host SR?

TheiLLeniumStudios

Maybe it is related to the provider not gracefully performing all the operations? Not really sure though

TheiLLeniumStudios

Inspecting the xensource.log file, I can see this in the logs:

May  1 13:05:02 minisforum-hm80-01 xapi: [debug||23091959 HTTPS 10.0.0.10->:::80|[XO] Importing content into VDI XO CloudConfigDrive R:ca2ba12b1631|taskhelper] the status of R:ca2ba12b1631 is: success; cannot set it to `success

But there's no error or anything that indicates why it cannot set it to success

olivierlambert

It's labor day today, therefore it's pretty calm. I'm sure more people will take it from here from tomorrow

TheiLLeniumStudios

@olivierlambert No worries I'll wait for a response from the team

yann

@TheiLLeniumStudios you can get a prebuild Linux binary as linked from the blog post, so not from the latest code, but should be good for a first test. Please let us know how it fares!

TheiLLeniumStudios

@yann Hi yann. I tried it out and it works perfectly fine on a normal VM. Are there any tips to run it as a container instead? With Talos I cannot really run it as a process as the OS is immutable so I need to run it as a container. I used to run this containerized version but it seems to be very old: https://github.com/GeoMSK/ros-xe-guest-utilities

Any tips on what would I need to expose to the container itself from the host, to be able to get it to work would be appreciated

yann

@TheiLLeniumStudios from what I understood it could be run by generating a custom Talos image. I don't have any detailed knowledge of Talos, so I can't say if there is any easier way.
Everything named "xe-guest-utilities" refers to the current guest tools or an older version thereof, all of which require many external utilities not present in Talos.

fitzgeraldtech

@TheiLLeniumStudios I'm trying to do something similar with Talos, were you able to find a workflow that worked for you since this post?