@acebmxer said in Custom config / cloud-init:

If convert this newly created vm to a template. Then create another vm from this template. Even though the VM is generating a new mac address when the vm boots its still getting / using the same ip from vm used to create the template.

You will have to carefully read cloud-init documentation to understand how it works and what phases it has.

Short answer is that VM template creation is a bit more work than just preparing some base VM and than converting it to the template or use it to clone other VMs. For a start, cloud-init has to be "cleaned" so that it is reset to the initial state.

cloud-init, or better said most of it's modules, run only once - just on first boot. So... when you create a VM from a hub template, cloud-init will run but if you use that VM to clone another VM, cloud-init will not run like on first boot of the original VM. Aside from that, to prepare some VM to be a template for other VMs, other "stuff" beside cloud-init has to be cleaned up like machine-id, SSH host keys, network configuration, logs... There is some official documentation on this here:

https://docs.xcp-ng.org/guides/create-use-custom-xcpng-ubuntu-templates/

and some forum addendum here:

https://xcp-ng.org/forum/topic/11008/ubuntu-22.04.5-custom-template-additional-steps-missing-from-documentation

read carefully

@acebmxer said in Custom config / cloud-init:

Think I figured it out. first i was using the wrong key. A lot of going back and forth trying new keys forgot to swap back to original...

new config...

#cloud-config
hostname: {name}
users:
  - name: newusername
    gecos: New User
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3....18ZbA

Any suggestions should I not use something in the above config if the server was in production?

• Well... NOPASSWD:ALL can be considered a security issue because user is not required to type a password to gain root privileges. If someone gains access to this user via for example stolen SSH key or some exploit, it will automatically have access to the root user.

• It is enough to put the user into "sudo" group for it to gain sudo privileges (with password required) because there is already this global sudoers rule:

  %sudo ALL=(ALL:ALL) ALL

• Adding the user to users and admin groups is a little bit contradictory. It should either be users or admin. Also, admin group does not exist on Ubuntu 24.04. It existed in some earlier versions but I'm not sure when and why it disappeared. It's best to just add the user to the sudo group so it can run commands as root and adm group (yes adm, not "admin") so it can view system logs (/var/log) without using sudo. This is if you are creating admin type user. If the user is just a regular user, you can add it to the users group but it's not necessary. If you are confused, don't worry, I'm too .

This is for Ubuntu, other operating systems have other principles and rules so there is no universal solution.

@Pilow I'm not so impressed with the hardware stack ... but this Cloudbox GUI is much more interesting. Is this something you developed from scratch in house for VMware and that ported to XO or...?

@SethNY Unfortunately there is currently zero perceived interest in further development of Ansible support for XenServer/XCP-ng, from the user side I mean. Since module inception, there were only a few bug reports and one feature request (to support VM snapshotting), at least judging by GitHub issues. There are no Ansible forum discussions either. This pretty much killed my interest in developing new modules. If there were more interest among users, I could reconsider.

Ansible module development is h-a-r-d, especially if you want to support idempotency properly. Upstreaming the module to community.general collection is even harder because of strict rules about code quality and automated unit testing. For small and very specific tasks like VM migration in your examples I tend to write bash or Python scripts to do the job, either by calling them from Ansible or manually. Developing a module to do the same task would take much much more time.

On one side, Terraform is much more popular for VM management. Most people do not even consider using Ansible for doing the same. On the other side, XenServer/XCP-ng is way way less popular compared to all the other virtualization solution which is shame because it is technically quite capable. I blame Citrix for making a few very bad decisions:

• Using ocaml for developing xapi daemon. It's a nice and powerful language but hard to digest by most people. This killed outside contributions even though it is open source. It also made a hard time for Citrix to find engineers for expanding their team.
• Citrix failed to develop web based replacement for XenCenter. VMware did it right on time. Rest is history.

All this does not spell hope especially for large, enterprise users which can drive the development.

@SethNY said in Ansible and XAPI first playbook (Ansible):

@bvitnik Thanks for that feedback.

XO understands multiple hosts and should be able to see what host a VM is on.

Yes and no. VM names are not unique so there could be multiple VMs with same name on multiple hosts/pools. Even on a single host/pool there can be VMs with same name. VM UUIDs are unique but only inside a single host/pool. Theoreticaly, you could have multiple VMs with same name and same UUID if you have multiple independant hosts or pools. XO would not be able to distinguish them when searching.

Instead of using Ansible directly against individual hosts, I am looking about how to use Ansible to work with the management layer.

You can loop over your hosts in Ansible to find the host running the VMs. Unless the list of your hosts is constantly changing, this should be a single additional Ansible task. No more than that .

The way I handle it is to specify each VM to be in a particular group in Ansible inventory. Each group points to a specific pool. This is even before VM is provisioned. That way Ansible alway knows where the VM is.

@SethNY There are no Ansible modules for Xen Orchestra (XO) so there are no such examples. You could theoreticaly call Xen Orchestra REST API directly using Ansible url module but that is a stretch.

I think you are missunderstanding what XAPI is. It is an API that is used for management of XenServer and XCP-ng hosts and pools, including any object on them like VMs, networks, disks etc. It is feature full. Ansible modules use XAPI to manage hosts/pools regardless of them being XenServer (commercial solution by Citrix) or XCP-ng (free and open source solution). Both are supported and mostly have cross compatibility.

Xen Orchestra is a management tool that itself uses XAPI in the background to manage XenServer/XCP-ng hosts/pools

Now a question. What are you trying to accomplish by managing hosts/pools through Xen Orchestra instead of managing them directly using Ansible modules?

@Cyrille My concern is that you are closing the door for people that do not need (or do not want) XO in their stack. Maybe they are using other ways to manage the stack, possibly custom developed, and XO would just be one more point of failure, another security concern etc.

From what I can gather, XO effectively acts as an API proxy here, plus as a list of pools. That's a rather insignificant (and forced?) role, from a technical point of view, considering XO has much much more functionality outside of what XCP-ng and XAPI offer themselves. All of that unused and not required for this integration.

@olivierlambert That's all fine and understandable but my question is more on the technical side of things... and still not answered

@Cyrille Hi. For what functionality does this plugin require XO? VDI operations are well covered with XAPI alone.

@BSmithITGuy said in Ubuntu 22.04.5 custom template - additional steps missing from documentation:

If you are trying to create an Ubuntu template and are having trouble on XCP-NG 8.3 (some or all probably applies to previous versions), keep reading.

Very nice. Very thorough. This inspired me to share a relevant part of my cleanup script. Maybe it will be of some additional help to everyone:

echo "Resetting machine ID..."
> /etc/machine-id
rm -f /etc/machine-info

echo "Cleaning APT cache..."
apt-get clean

echo "Removing netplan configuration..."
# Ubuntu < 24.04
rm -f /etc/netplan/00-installer-config.yaml
# Ubuntu >= 24.04
rm -f /etc/netplan/50-cloud-init.yaml

echo "Cloud-init cleanup..."
if [[ -f "/etc/cloud/clean.d/99-installer" ]]; then
    chmod a-x /etc/cloud/clean.d/99-installer
fi

# Ubuntu < 24.04
rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
# Ubuntu >= 24.04
rm -f /etc/cloud/cloud.cfg.d/90-installer-network.cfg

if [[ -x "$(command -v cloud-init)" ]]; then
    cloud-init clean --logs --seed
else
    echo "  No cloud-init detected. Skipping cloud-init cleanup!"
fi

echo "Removing SSH host keys..."
find /etc/ssh/ -type f -name 'ssh_host_*' -delete

The code is self explanatory but here are some additional tips:

• It's nice to do cleanup of APT cache. This will free up some space but also force the user to update APT cache on first run after VM has been provisioned. If APT cache is not cleaned, it will grow stale over time inside a template and, after some time, package installation in new VMs will start to break. Doing apt update before any package installation is good practice but people tend to easily forget it. This will force them to do it.
• Ubuntu installer relies on some cloud-init config for some first boot setup which has to be removed if cloud-init is to be used once again for provisioning VMs. You correctly identified these files for removal but there is a difference in Ubuntu older than 24.04 and Ubuntu 24.04 and newer.
• The reason I do chmod a-x /etc/cloud/clean.d/99-installer is to prevent cloud-init from removing /etc/cloud/ds-identify.cfg when doing cloud-init clean (default behavior). Because I override the default /etc/cloud/ds-identify.cfg, I don't want cloud-init clean to remove it. I remove all the other files explicitly. Additional note, /etc/cloud/clean.d/99-installer should not be removed. It should just be chmoded because it is part of the cloud-init package and will reappear on cloud-init package update.
• It's better to use cloud-init clean --logs --seed command to clean any runtime cloud-init leftovers and logs instead of removing them explicitly like rm -rf /var/lib/cloud/instance.
• SSH host keys should be removed so that they can be regenerated on first boot of newly provisioned VM. If you don't do it, all your VMs will have same host keys which could be considered as security issue.

@bufanda said in Automating VM builds on XO with Redhat AAP (Ansible):

... as far as I know there is no collection that is able to manage VMs and sadly Satellite's virt-who doesn't seem to have XenOrchestra support at the moment and since RedHat is pushing OpenShift I don't think it will ever come.

But there was a wish for a colletion on the DevOps Megathread
https://xcp-ng.org/forum/post/88993

There are Ansible modules for managing VMs in the comunity.general collection. They just don't do it through XO but through XenServer/XCP-ng hosts (or pool masters) directly, using XenAPI. Search for xenserver_* modules in Ansible documentation.

@DustinB While that may be true, the reality is that VMware and Microsoft have integrated such functionality in their virtualization products and it has become de facto standard. Users are expecting such functionality and taking it for granted. No one is bothering with sysprep any more. This is one of those things that makes the life much easier and which XenServer/XCP-ng should support to stay relevant in the market.

Maybe something can be done trough Cloudbase-Init? I have not delved into that.

@TeddyAstie Is the list of "issues and limitations" of nested virtualization under Xen documented somewhere?

What he is talking about is Security Identifier (SID) and is specific to Windows. Each Windows machine must have a unique SID in an AD environment. Cloned machines will have identical SID which is bad.

The solution to this problem is sysprep, a Windows tool that will reset SID and other parameters so that each Windows installation is uniquely identifiable. This is something done inside a machine, of course, and is not something that can be done on HV level. VMware, and Hyper-V as seen here, have integrated support for invoking sysprep (or equivalent) during the machine cloning process. This is achieved by sending a signal to the management agent inside a machine (e.g. VMware Guest Tools).

As far as I know, XenServer/XCP-ng management agent is rudimentary and does not have this functionality. In world of XenServer/XCP-ng, machine has to be syspreped manually before it is cloned.

@cairoti It's explained here:

https://docs.xenserver.com/en-us/citrix-hypervisor/dr/machine-failures.html#master-failures

Quote:

If you repair or replace the server that was the original master, you can simply bring it up, install the Citrix Hypervisor software, and add it to the pool. Since the Citrix Hypervisor servers in the pool are enforced to be homogeneous, there is no real need to make the replaced server the master.

Now, there is a catch. I'm not sure what happens with the old master from a pool perspective after a new master is delegated. Is it still considered (and shown) as member of the pool just shut down, or is it kicked out of the pool? Anyway, if old master is returned to the pool, i.e. a join operation is performed, then it's configuration is reset and it will not cause any conflict.

If you don't want to risk anything, the best way to go is to remove old master from the pool, reinstall it and re-add it. That's the clean way. The reinstall will make old master forget it was ever a master

@bug-meister This means that VM was running at the CPU feature level of the original host when you added a new host. For a VM to drop the CPU feature level to a level of the new host, it has to be shutdown and started again. After that, VM should be able to migrate in both ways.

The CPU feature level of a pool is determined by the CPU feature level of a host that has the lowest CPU feature level (new host in your example) but is not applied retroactively to already running VMs.

Because of these issues, it is not advisable to have pools with mixed CPU models/generations.

@irtaza9 bro, believe it or not, I already answered your question . If you have your templates in XVA format and want to have them accessible to all your pool members, you have to put them on a shared SR. They will still be per pool. I don't do not know of a way to have them centralized so that they are available cross pool.

Update:

Theoretically an NFS shared SR could be mounted to multiple pools but it would have to be read only except for one pool. I have not seen something like that in action.

@irtaza9 The way you are describing your intent and what you are trying to accomplish makes me think you are doing it the wrong way. Cloud-ready or not, your templates have to be XenServer/XCP-ng ready. To accomplish that you have two options:

• Create an empty VM from one of built in default templates, boot an ISO/PXE and install your system. Add XenServer/XCP-ng guest tools and cloud-init to the VM, shutdown and convert the VM to the template. You can automate this process using Packer or Ansible if needed.

• Import your VM template from OVA/OVF.

NFS ISO library is ment to be used only for ISOs. While you can store qcow2 images on it, they will not be shown. Qcow2 images are disk images, not templates... at least not in a sense of XenServer/XCP-ng.

To create a VM in XenServer/XCP-ng using qcow2 image, you have to:

1. Store the image to some shared SR of NFS or SMB type (not NFS ISO library)
2. Create a VDI associated with the qcow2 image file
3. Create an empty VM using one of builtin default templates and attach the VDI to the VM

@olivierlambert Maybe a suggestion for XO dev team to implement a way to filter out default templates from drop down menus and other listings in XO. There are a lot of them and they can pretty much spam the lists.

Idea could be to show only "real" templates by default but let the user chose an option to include builtin/default templates, snapshots, and other shut down VMs that can be used as templates.

@irtaza9 Isn't it dangerous to remove default templates? They are used to create VMs from scratch by booting ISO image (or PXE) and installing a system. In a sense, they are not real templates. They are just metadata related to particular guest OS that describe expected guest OS of the VM instantiated from it, resource limits and restrictions, virtualization type (i.e. PV vs HVM) etc. XenServer/XCP-ng is using this metadata to possibly activate OS specific optimizations or workarounds. In VMware world the equivalent is more or less the so called Guest ID.

How do you intend to create VM templates? You can always just import premade templates from external sources or use "Other install media" as a base but it could introduce some issues later like XenServer/XCP-ng having more strict restrictions on resources (CPUs, number of disks, number of network interfaces...) compared to templates created by using some of these built in default "templates".

I would strongly advise you against removing default templates.