Posts made by dj423 | XCP-ng and XO forum

dj423

Would that be something like this for a dynamic network config as an example?

network:
  version: 1
  config:
    - type: physical
      name: enX0
      subnets:
        - type: static
          address: 192.168.99.{index}/24
          gateway: 192.168.99.1
    - type: nameserver
      address:
        - 172.31.31.1
        - 192.168.45.11
      search:
        - example.tld

I plan to test this at some point, but I don't do a whole lot of bulk provisioning myself. Just curious.

dj423

@encryptblockr

FWIW: I know I have done this from cli using xo-cli in a bash script.

I don't have all that code handy at the moment, but here is a hack from memory as an example:

#! /bin/bash

while IFS=',' read -r name_c1 ip_c2
do
    file=v1-$ip_c2.yaml

    cat >> "$file" << EOF
network:
  version: 1
  config:
    - type: physical
      name: eth0
      subnets:
        - type: static
          address: $ip_c2/24
          gateway: 192.168.0.1
    - type: nameserver
      address:
        - 192.168.0.10
        - 192.168.23.50
      search:
        - example.tld
EOF

xo-cli vm.create bootAfterCreate=true cloudConfig="$(cat /root/user.yaml)" networkConfig="$(cat $file)" clone=true name_label="$name_c1" template=0856a8d6-7761-f39d-f968-290b18a1bd42 VIFs='json:[{"network":"ca96456f-3843-26f5-7075-1e54xxxx121xxf97"}]' hvmBootFirmware=bios copyHostBiosStrings=true

rm $file
done <  prov-list
exit

prov-list example:

Bookworm,192.168.0.217
Noble1,192.168.0.218
Jammy1,192.168.0.219
Oracle9,192.168.0.222
Rocky9,192.168.0.223
Fedora40,192.168.0.224

If you wanted to do different networks, that is a bit more complicated, but just need a variable to render.

Sorry on my tablet and I don't have a better example handy. But this should get you close, and can be handy when you need to stand up a few dozen VM's.

dj423

@encryptblockr said in Ubuntu 24.04 Cloud Image Template Not Working:

did you mount the guest tools cdrom to the cloud image before you created a template from it?

No, I just make my base images (for the end template) as generic as possible, and I do all mounts, package installs, accounts, keys, including the xe-guest utils package all in the user-data config for cloud-init. Makes the configurations more uniform across all stacks I have to maintain and scales really well with jinja templates.

also what is point of {name}% in below? can you explain? where does it get {name} from and what is % use?

That pulls the hostname from metadata that is provided by the nocloud datasource.

If you ever want to see all the metadata available (for example, say your provisioning plane does dynamic configuration from some backend IAS platform, and you need logic that renders metadata to run different configurations; run the following after login:

cloud-init query -a

For example, I run different configurations based on what the 'distro' value is -
example:

cloud-init query -f {{v1.distro}}

{% if distro in ['rocky', 'ol', 'centos', 'fedora', 'redhat', 'almalinux'] -%}
//do the rpm things//

This allows us to run completely different configuration settings based on what distro its being run in, or even what virtualization platform it is running on. Some platforms won't use the hostname, or fqdn meta values, so I have some if logic that only adds it when vendordata is null, example;

{% if vendordata == '' %}
fqdn: {name}
{% endif %}

Probably more than you wanted to know, but that's a brief summary of cloud-init metadata.

dj423

@encryptblockr

For reference, if you are testing network configs, here is a basic static ip v1 example I use to configure all distros, debian, ubuntu, RHEL, Rocky, Alma, Oracle linux, etc.

network:
  version: 1
  config:
    - type: physical
      name: enX0
      subnets:
        - type: static
          address: 192.168.0.xx/24
          gateway: 192.168.0.1
    - type: nameserver
      address:
        - 192.168.0.1
        - 192.168.90.50
      search:
        - example.tld

Works very well for me, as I am lazy and want to do as little manual configuration as possible, and XCP-ng and XO make that so easy. Most ENI, network-manager, systemd-networkd, netplan network stacks render this config fine. FYI

dj423

@encryptblockr
Oh good! Glad you got it working.

Keep in mind for those 'public' cloud images, (for anyone that uses them) you can have cloud-init install the xe guest utils at initialization as long as you have the guest-tools.iso mounted at bootup in the console within xo;

Sounds like you are getting the hang of it. Best of luck!

dj423

@encryptblockr

Just add the commands to your cloud config in user-data;

runcmd:
  - mount /dev/cdrom /mnt
  - bash /mnt/Linux/install.sh
  - umount /dev/cdrom

dj423

@encryptblockr

You are using version 2 of your network config, did you happen to install netplanio package to consume your version 2 config, and is it enabled in the image you created? Otherwise, I have better luck with the version 1 network configs.

dj423

@encryptblockr

Pretty sure you want the 'nocloud' data source as well. You can have cloud-init mount and install the xe guest tools if the virtual cd is attached at bootup.

dj423

@encryptblockr

I don't see your users block in that config

Example:

users:
## Add users
  - name: user1
    groups: sudo
    lock_passwd: false
    passwd: $6$xxxx.......
    ssh_authorized_keys:
      - ssh-ed25519 <my-key-ed25519>

So that may be an issue. See my examples for a guide.

dj423

@encryptblockr

That guide may be a bit outdated. I would suggest following the guide I have posted over at https://xcp-ng.org/forum/topic/6821/cloud-init-success/3 for more recent noble/jammy cloud images.

and if you can, use the cloud images provided by ubuntu/canonical as the base. Over here: https://cloud-images.ubuntu.com/noble/current/

Can you provide any details on what didn't work? If you need to generate the base image from scratch, I can help but that is a bit more involved and challenging. Much better to go with the pre-canned cloud images when possible.

dj423

@encryptblockr
Yes, and I documented the process I used a while back here: https://xcp-ng.org/forum/topic/6821/cloud-init-success

I just use the stock cloud image .ova file as my base image. for Ubuntu templates. I use the noble (24.04) image in quite a few instances from my template with success.

Your post mentions cloud-init is not working - some tips to try troubleshooting, when the new VM is provisioned, (assuming you are able to login) I usually run "cloud-init status --wait" and when that completes I run "cloud-init status --long" for a summary of the state. Then you can grep for WARNING in /var/log/cloud-init.log for clues.

If you weren't able to login to the VM - make sure your cloud-init config has the userid's, passwords and SSH keys you will need for login. See the example I posted here, since those are what I use in my lab or some variation.

Feel free to share any troubleshooting steps you have taken, and I will help any way I can. I have setup hundreds of VM's from templates using cloud-init so I have seen many of the challenges it brings.

dj423

@julien-f said in % characters in cloud-init configs render as 0 in instance:

@dj423 It should be fixed in the master branch: https://github.com/vatesfr/xen-orchestra/commit/30e6d4b4f14ddbb6ee34cd456e4ca707486df2db

{index} is now used in place of %

% is still working when Multiple VMs is enabled

% and \% will be replaced by % when Multiple VMs is disabled

Wow! First of all big Thank you for getting that in place so fast. I updated, and for grins provisioned an Oracle 9.4 VM in XO using the same templated config (without the escaped /%) and it worked flawless.

snippet:
{% if distro == 'rocky' or distro == 'centos' or distro == 'ol' or distro == 'almalinux' or distro == 'fedora' or distro == 'redhat' %}
{% set group = 'wheel' %}

Thanks again Julien!

dj423

@julien-f said in % characters in cloud-init configs render as 0 in instance:

@dj423 I know it's a bit cumbersome but in the meantime, you can escape the problematic character: \%.

Ok cool, I was wondering if that would work, and my testing worked perfectly. Thank you!

dj423

@julien-f said in % characters in cloud-init configs render as 0 in instance:

@dj423only useful when the advanced setting Multiple VMs is enabled, always 0 otherwise

I found that setting in advanced, and never even knew it existed lol

I am not using that feature.

Was hoping I could just disable it and see if it changes behavior.

dj423

@julien-f

Possibly, I just need the jinja lines to render with the % characters in place.

Example:

{% elif distro == 'ubuntu' or distro == 'debian' %}
{% set group = 'sudo' %}

Is there a config setting to disable the Multiple VM setting I missed?
Thanks for your help!

dj423

@dj423

After a bit more digging, I see from the XO cloud config, to the config drive is where the characters get switched - here is the output of
xo-cli cloudConfig.getAll

id: 'e7655d9f-c442-4c0f-8e16-35bf3e5e5685',
    name: 'Prod-RL9-jinja',
    template: '## template: jinja\n' +
      '#cloud-config\n' +
      'fqdn: {name}%\n' +
      "{% set u1 = 'user' %}\n" +
      "{% set u1pass = '$6$xxxxxxK0q11t/' %}\n" +

Everything looks correct, and matches the config as in XO cloud configs. All good so far.

Mounting the cloud config drive I see something different, and why it is failing - cloud init is choking on the {0 's

mount /dev/xvda /mnt
cat /mnt/user-data
root@test1:/mnt/userdata# cat user-data

## template: jinja
#cloud-config
fqdn: RL9-jinjatest0
{0 set u1 = 'user' 0}
{0 set u1pass = 'xxx/' 0}
{0 set u1key = 'xxxx xx-ed25519-ansible' 0}

All of the % characters are now 0's. See it in the hostname as well as jinja.

Anyone know what part of the process (where it reads the cloud-init userdata and packages into the config disk) would flip all %'s to 0's?? When I run tests on the original userdata config, everything renders fine when I test it with cloud init in the VM instance, so I don't think the issue is with cloud-init.

ex: cloud-init query --format="$( cat userdata.yaml )"

When I feed the same userdata config to xo-cli vm.create everything renders fine, it just doesn't have the metadata for the hostname. Really strange. I will keep digging, and try a few test configs for giggles.

dj423

Anyone else using jinja template directives for provisioning new VMs?

Im running 8.2 XO commit a5967 and Master ba375.

I started noticing that my cloud-init configs are behaving a bit different now when I provision new VM's.

In the hostname variable for example, I have used {name}% as the hostname variable to pass the VM name meta data to cloud-init.
Example:

#cloud-config
fqdn: {name}%

Now the hostname will render with a 0 on the end, ie: somehost.example.com0 rather than the vm name of somehost.example.com.

So I removed the % from the {name}% directive and now it renders the hostname correct.

However, any cloud-init configs with jinja templating in them, (using the same % character) no longer work, since the jinja variables show up in the user-data file presented to cloud-init in the nocloud datasource as {0 if distro == ... 0} when it used to pass the jinja coding to cloud-init.

As used in the cloud-init config in XOA:

{% if distro == 'rocky' or distro == 'centos' or distro == 'ol' or distro == 'almalinux' or distro == 'fedora' or distro == 'redhat' %}

renders as in user-data file as:

{0 if distro == 'rocky' or distro == 'centos' or distro == 'ol' or distro == 'almalinux' or distro == 'fedora' or distro == 'redhat' 0}

So this hoses jinja from rendering the variables.

Anyone else notice a change in how metadata is being presented to cloud-init? Possible bug, or some change I didn't notice until now? Not a deal breaker for me, since I have other configs that work fine, just odd how it changed. If I find a solution to this, I will share here.
TIA!

dj423

Re: Cloud-init success

Had a chance to "refresh" some cloud-init configs for some later distros (Debian 12, Ubuntu 24.04, Rocky Linux 9.3, etc.) so thought I would share some example configs that are used mainly for bootstrapping new VM's and or containers.

This first one is targeted at deb based distros and sets up an Incus container host. Like most configs I use this one is pretty specific pulling key files and configs from an NFS share, rsyslog target, etc., but gives an idea what can be done for detailed provisioning of instances, aside from just slapping your SSH keys on and letting ansible take over. With this I can have a fresh VM provisioned in about 5 minutes.

I always like to run a status check after init:

root@IncusTEST20:~# cloud-init status --long
status: done
boot_status_code: enabled-by-generator
last_update: Sat, 26 Jul 2024 17:29:50 +0000
detail:
DataSourceNoCloud [seed=/dev/xvdc][dsmode=net]

Then we can query the user-data to see how it rendered from the config injected into the nocloud datasource:
root@IncusTEST20:~# cloud-init query userdata

#cloud-config
    hostname: IncusTEST20
    users:
      - name: foo
        groups: sudo
        lock_passwd: false
        passwd: $6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725
        shell: /bin/bash
      - name: bar
        groups: sudo
        lock_passwd: false
        passwd: $6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725
        shell: /bin/bash
    locale: en_US.UTF-8
    timezone: America/New_York
    resize_rootfs: true
    mounts:
      - ["192.168.99.2:/mnt/Vault/lxd", "/mnt/lxd", "nfs", "auto,nofail,noatime,nolock,intr,tcp,actimeo=1800,user,suid", "0", "0"]
      - ["192.168.0.54:/mnt/raid/backup", "/mnt/nas", "nfs", "auto,nofail,noatime,nolock,intr,tcp,actimeo=1800,user,suid", "0", "0"]
    rsyslog:
      remotes:
        log_serv: "192.168.50.35:5140"
    write_files:
      - path: /etc/init.d/incus.sh
        owner: root:root
        permissions: 0o755
        defer: true
        content: |
          #!/bin/bash
          curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
          sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-lts-6.0.sources
          Enabled: yes
          Types: deb
          URIs: https://pkgs.zabbly.com/incus/lts-6.0
          Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
          Components: main
          Architectures: $(dpkg --print-architecture)
          Signed-By: /etc/apt/keyrings/zabbly.asc
          EOF'
          apt update -y
          dpkg --configure -a
          apt install incus incus-ui-canonical -y
          mkdir -p /root/.config
          mkdir -p /root/.config/rclone
          mount -a
          cp /mnt/lxd/.config/.encode /root/.encode
          cp /mnt/lxd/.config/rclone.conf /root/.config/rclone/rclone.conf
          chmod 600 /root/.config/rclone/rclone.conf
          # Delete self
          rm "${0}"
    runcmd:
      - mkdir /mnt/lxd
      - mkdir /mnt/nas
      - date > /etc/birth_certificate
      - [ mount /dev/cdrom /mnt ]
      - [ bash /mnt/Linux/install.sh ]
      - [ umount /dev/cdrom ]
      - [ sh, /etc/init.d/incus.sh ]
    package_update: true
    package_upgrade: true
    packages:
      - htop
      - nano
      - curl
      - wget
      - nfs-common
      - btrfs-progs
      - bridge-utils
      - build-essential
      - rclone

Here is a more universal config with jinja templating syntax that can be targeted at many distros and using an 'if distro' will configure the instance based on the distro value in metadata:

This one targets debian, ubuntu, centos, redhat and rocky linux 8 and up - notice the ##template: jinja at the top, this allows jinja to render and process:

## template: jinja
#cloud-config
{% set u1 = 'foobar' %}
{% set u1pass = '$6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1' %}
{% set u1key = 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725' %}
{% set u2 = 'ansible' %}
{% set u2pass = '$6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1' %}
{% set u2key = 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725' %}
locale: en_US.UTF-8
timezone: America/New_York
runcmd:
  - mkdir /mnt/v-nas
  - mkdir /mnt/home
  - date > /etc/birth_certificate
rsyslog:
  remotes:
    log_serv: "192.168.50.35:5140"
{% if distro == 'rocky' or distro == 'centos' or distro == 'redhat' %}
{% set group = 'wheel' %}
repo_update: true
repo_upgrade: all
yum_repos:
  epel-release:
    name: Extra Packages for Enterprise Linux 9 - Release
    baseurl: http://download.fedoraproject.org/pub/epel/9/Everything/$basearch
    enabled: true
    failovermethod: priority
    gpgcheck: true
    gpgkey: http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9
package_upgrade: true
packages:
  - htop
  - nano
{% elif distro == 'ubuntu' or distro == 'debian' %}
{% set group = 'sudo' %}
package_update: true
package_upgrade: true
packages:
  - htop
  - nano
  - build-essential
users:
  - name: {{ u1 }}
    groups: {{ group }}
    lock_passwd: false
    passwd: {{ u1pass }}
    ssh_authorized_keys:
      - {{ u1key }}
    shell: /bin/bash
  - name: {{ u2 }}
    groups: {{ group }}
    lock_passwd: false
    passwd: {{ u2pass }}
    ssh_authorized_keys:
      - {{ u2key }}
    shell: /bin/bash
{%- endif %}

Just a note, to consume jinja templating, you need cloud-init 22.x or higher, with the jinja package installed in your template/image.

For anyone wanting to tinker with these, (these are not my keys or password hashes, these I just threw in for demonstration purposes and for the wiki page). The password for the accounts is "foobar" and the matching private key is:

Enjoy!

dj423

@runevn
A bit more from my 'troubleshooting notes' I gathered back when I was prototyping a lot of very complex configurations and worked with Chad Smith:

Test user-data rendering: cloud-init query --format "$(sudo cloud-init query userdata)"
Test jinja template rendering: cloud-init query --format="$( cat test.cfg )" # test.cfg would be the test cloud-config.cfg file to render

Test jinja template detailed: cloud-init devel render test.cfg -d
Query metadata: cloud-init query -f {{v1.distro}}

========= Debug user-data rendering =============
cloud-init devel render /var/lib/cloud/instance/user-data.txt -d

Check actual seed file from provider

cloud-init devel render /var/lib/cloud/seed/nocloud-net/user-data -d

On a system where cloud-config wasn't honored, run: sudo cloud-init query --system # or you can add --annotate which will annotate the specific lines where an error is in the #cloud-config YAML
per Chad Smith;

All cloud-init instance data /var/lib/cloud/instance/
Userdata /var/lib/cloud/instance/user-data.txt
Datasource /var/lib/cloud/instance/datasource
metadata /run/cloud-init/instance-data.json
Seed files from provider/hypervisor: /var/lib/cloud/seed/nocloud-net

For anyone doing a deep dive into troubleshooting cloud-init configuration rendering.