@BSmithITGuy said in Ubuntu 22.04.5 custom template - additional steps missing from documentation:

If you are trying to create an Ubuntu template and are having trouble on XCP-NG 8.3 (some or all probably applies to previous versions), keep reading.

Very nice. Very thorough. This inspired me to share a relevant part of my cleanup script. Maybe it will be of some additional help to everyone:

echo "Resetting machine ID..."
> /etc/machine-id
rm -f /etc/machine-info

echo "Cleaning APT cache..."
apt-get clean

echo "Removing netplan configuration..."
# Ubuntu < 24.04
rm -f /etc/netplan/00-installer-config.yaml
# Ubuntu >= 24.04
rm -f /etc/netplan/50-cloud-init.yaml

echo "Cloud-init cleanup..."
if [[ -f "/etc/cloud/clean.d/99-installer" ]]; then
    chmod a-x /etc/cloud/clean.d/99-installer
fi

# Ubuntu < 24.04
rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
# Ubuntu >= 24.04
rm -f /etc/cloud/cloud.cfg.d/90-installer-network.cfg

if [[ -x "$(command -v cloud-init)" ]]; then
    cloud-init clean --logs --seed
else
    echo "  No cloud-init detected. Skipping cloud-init cleanup!"
fi

echo "Removing SSH host keys..."
find /etc/ssh/ -type f -name 'ssh_host_*' -delete

The code is self explanatory but here are some additional tips:

• It's nice to do cleanup of APT cache. This will free up some space but also force the user to update APT cache on first run after VM has been provisioned. If APT cache is not cleaned, it will grow stale over time inside a template and, after some time, package installation in new VMs will start to break. Doing apt update before any package installation is good practice but people tend to easily forget it. This will force them to do it.
• Ubuntu installer relies on some cloud-init config for some first boot setup which has to be removed if cloud-init is to be used once again for provisioning VMs. You correctly identified these files for removal but there is a difference in Ubuntu older than 24.04 and Ubuntu 24.04 and newer.
• The reason I do chmod a-x /etc/cloud/clean.d/99-installer is to prevent cloud-init from removing /etc/cloud/ds-identify.cfg when doing cloud-init clean (default behavior). Because I override the default /etc/cloud/ds-identify.cfg, I don't want cloud-init clean to remove it. I remove all the other files explicitly. Additional note, /etc/cloud/clean.d/99-installer should not be removed. It should just be chmoded because it is part of the cloud-init package and will reappear on cloud-init package update.
• It's better to use cloud-init clean --logs --seed command to clean any runtime cloud-init leftovers and logs instead of removing them explicitly like rm -rf /var/lib/cloud/instance.
• SSH host keys should be removed so that they can be regenerated on first boot of newly provisioned VM. If you don't do it, all your VMs will have same host keys which could be considered as security issue.

@bufanda said in Automating VM builds on XO with Redhat AAP (Ansible):

... as far as I know there is no collection that is able to manage VMs and sadly Satellite's virt-who doesn't seem to have XenOrchestra support at the moment and since RedHat is pushing OpenShift I don't think it will ever come.

But there was a wish for a colletion on the DevOps Megathread
https://xcp-ng.org/forum/post/88993

There are Ansible modules for managing VMs in the comunity.general collection. They just don't do it through XO but through XenServer/XCP-ng hosts (or pool masters) directly, using XenAPI. Search for xenserver_* modules in Ansible documentation.

@DustinB While that may be true, the reality is that VMware and Microsoft have integrated such functionality in their virtualization products and it has become de facto standard. Users are expecting such functionality and taking it for granted. No one is bothering with sysprep any more. This is one of those things that makes the life much easier and which XenServer/XCP-ng should support to stay relevant in the market.

Maybe something can be done trough Cloudbase-Init? I have not delved into that.

@TeddyAstie Is the list of "issues and limitations" of nested virtualization under Xen documented somewhere?

What he is talking about is Security Identifier (SID) and is specific to Windows. Each Windows machine must have a unique SID in an AD environment. Cloned machines will have identical SID which is bad.

The solution to this problem is sysprep, a Windows tool that will reset SID and other parameters so that each Windows installation is uniquely identifiable. This is something done inside a machine, of course, and is not something that can be done on HV level. VMware, and Hyper-V as seen here, have integrated support for invoking sysprep (or equivalent) during the machine cloning process. This is achieved by sending a signal to the management agent inside a machine (e.g. VMware Guest Tools).

As far as I know, XenServer/XCP-ng management agent is rudimentary and does not have this functionality. In world of XenServer/XCP-ng, machine has to be syspreped manually before it is cloned.

@cairoti It's explained here:

https://docs.xenserver.com/en-us/citrix-hypervisor/dr/machine-failures.html#master-failures

Quote:

If you repair or replace the server that was the original master, you can simply bring it up, install the Citrix Hypervisor software, and add it to the pool. Since the Citrix Hypervisor servers in the pool are enforced to be homogeneous, there is no real need to make the replaced server the master.

Now, there is a catch. I'm not sure what happens with the old master from a pool perspective after a new master is delegated. Is it still considered (and shown) as member of the pool just shut down, or is it kicked out of the pool? Anyway, if old master is returned to the pool, i.e. a join operation is performed, then it's configuration is reset and it will not cause any conflict.

If you don't want to risk anything, the best way to go is to remove old master from the pool, reinstall it and re-add it. That's the clean way. The reinstall will make old master forget it was ever a master

@bug-meister This means that VM was running at the CPU feature level of the original host when you added a new host. For a VM to drop the CPU feature level to a level of the new host, it has to be shutdown and started again. After that, VM should be able to migrate in both ways.

The CPU feature level of a pool is determined by the CPU feature level of a host that has the lowest CPU feature level (new host in your example) but is not applied retroactively to already running VMs.

Because of these issues, it is not advisable to have pools with mixed CPU models/generations.

@irtaza9 bro, believe it or not, I already answered your question . If you have your templates in XVA format and want to have them accessible to all your pool members, you have to put them on a shared SR. They will still be per pool. I don't do not know of a way to have them centralized so that they are available cross pool.

Update:

Theoretically an NFS shared SR could be mounted to multiple pools but it would have to be read only except for one pool. I have not seen something like that in action.

@irtaza9 The way you are describing your intent and what you are trying to accomplish makes me think you are doing it the wrong way. Cloud-ready or not, your templates have to be XenServer/XCP-ng ready. To accomplish that you have two options:

• Create an empty VM from one of built in default templates, boot an ISO/PXE and install your system. Add XenServer/XCP-ng guest tools and cloud-init to the VM, shutdown and convert the VM to the template. You can automate this process using Packer or Ansible if needed.

• Import your VM template from OVA/OVF.

NFS ISO library is ment to be used only for ISOs. While you can store qcow2 images on it, they will not be shown. Qcow2 images are disk images, not templates... at least not in a sense of XenServer/XCP-ng.

To create a VM in XenServer/XCP-ng using qcow2 image, you have to:

1. Store the image to some shared SR of NFS or SMB type (not NFS ISO library)
2. Create a VDI associated with the qcow2 image file
3. Create an empty VM using one of builtin default templates and attach the VDI to the VM

@olivierlambert Maybe a suggestion for XO dev team to implement a way to filter out default templates from drop down menus and other listings in XO. There are a lot of them and they can pretty much spam the lists.

Idea could be to show only "real" templates by default but let the user chose an option to include builtin/default templates, snapshots, and other shut down VMs that can be used as templates.

@irtaza9 Isn't it dangerous to remove default templates? They are used to create VMs from scratch by booting ISO image (or PXE) and installing a system. In a sense, they are not real templates. They are just metadata related to particular guest OS that describe expected guest OS of the VM instantiated from it, resource limits and restrictions, virtualization type (i.e. PV vs HVM) etc. XenServer/XCP-ng is using this metadata to possibly activate OS specific optimizations or workarounds. In VMware world the equivalent is more or less the so called Guest ID.

How do you intend to create VM templates? You can always just import premade templates from external sources or use "Other install media" as a base but it could introduce some issues later like XenServer/XCP-ng having more strict restrictions on resources (CPUs, number of disks, number of network interfaces...) compared to templates created by using some of these built in default "templates".

I would strongly advise you against removing default templates.

@Byte0 When xe utility is running on a XenServer/XCP-ng host itself, without any additional connection related command line parameters, it defaults to using XenAPI Unix socket:

# ls -la /var/xapi/xapi
srwxr-xr-x 1 root root 0 feb 18 11:07 /var/xapi/xapi

Since this socket is owned by root user and writable only by root user and only accessible from inside the dom0 system, then it is assumed to be safe and xe does not require any kind of authentication. xe can freely send API requests trough the socket.

So can PowerShell module do the same? Possibly, but PowerShell is way less documented than, let's say, Python modules, so I can't tell without looking at the code.

To check it yourself, you will first have to install PowerShell modules on XenServer/XCP-ng itself and make it work in standard way (Connect-XenServer). Then take a look at PowerShell modules source code to see if there is any trace of support for XenAPI Unix socket. If there is none and you are good with PowerShell, maybe you can add support for Unix socket.

For XenAPI Python modules you can use either http/httpd or unix socket connection.

Update:

A few Googles later. It seems that PowerShell does not have support for Unix sockets at all. It requires some kind of TCP to Unix socket proxy like socat. So making XenAPI PowerShell modules work over Unix socket could be non trivial undertaking and some considerable hacking would be required.

@irtaza9 OK so you have VLANs per DC. That means VLAN5 in KPK-DC1 and VLAN5 in RWP-DC4 are unrelated and machines in those VLANs cannot communicate directly without going trough some router? This goes back to my original answer. You have to take into account network connectivity when deciding what DC/pool to select for VM provisioning. If you have a few VMs in VLAN5 in KPK-DC1 and you decide to provision some more VMs at latter time that require direct connectivity with existing VMs, you have no choice but to provision them in VLAN5 in KPK-DC1.

Regarding VLANs in general, this is all fine except you can scale it up to a few thousand of VLANs per DC so it will impose a hard limit on the number of isolated network segments (i.e. users/customers). Some weapon of choice among most cloud providers is to use some kind of network overlay like VX-LANs, GRE tunnels and such but this has issues of it's own. For a start, these kinds of networks are software defined in nature (SDN), require an orchestrator and network nodes that run in software. You can leverage some hardware acceleration here and there but it will still be software oriented and have scaling and latency issues. There is no silver bullet and most providers hide their network implementation details.

All in all, a good, scalable and fast networking i probably the toughest thing to implement in any large scale cloud provider... and I'm definitely not a network engineer so I can't say much about it .

@irtaza9 You have VLANs stretched across your whole infrastructure? That means your broadcast domain is your whole infra ? That's a bit risky. A broadcast storm can bring your whole infra down instead of being contained in a single DC for example.

IP address availability is also a criteria, but OK, it can be considered a "resource" like CPU, RAM and storage. There is also network connectivity issue. If you are using some private addressing, either any private subnet has to be addressable/connectable from any part of your whole infrastructure or you have to group your VMs per IP subnet availability (regions?) so that VMs belonging to the same logical group can have network connectivity between themselves.

Just out of curiosity, is this a hypothetical question or do you really have this huge infrastructure? Multiple pools with 64 hosts sounds like a considerable amount of resources. My experience is that if you go above 20 hosts per pool, you get all kinds of scalability issues.

The issues described here usually require some cloud platform deployed on top of XO/XCP-ng if you want to manage infrastructure as large as this and provide full flexibility to you users.

Has anyone been able to install Ubuntu 24.04 in VM from current official ISO? It seems that official ISOs (i.e. installer) still use unpatched kernel 6.8.0-22. Are there any newer ISO builds that I'm not aware of?

EDIT:
Sorry. False alarm. I screwed up my PXE settings. There was some leftover kernel and initrd images from beta versions of ISO. Kernel and initrd from latest ISOs work properly.

@olivierlambert Will 8.1 to 8.3 upgrade path also be supported?

@olivierlambert I would have never guessed that I sound so pessimistic ... I just wrote my experience... reality is sometimes harsh.

Glad to hear you are investing in dedicated people. With more people, a good team and being a company, you certainly have more leverage than independent volunteers like myself. Any road map you can share publicly?

@john-c

I feel your pain, however, the main difference between VMware support in Ansible and XenServer/XCP-ng is that VMware has a whole working group with a dozen of regular members and contributors:

https://github.com/ansible/community/wiki/VMware

Major contributors are all Red Hat or VMware employees i.e. people paid to do it. There is no such thing for XenServer/XCP-ng. Citrix never showed any interest in supporting Ansible. Netscaler is the only Citrix product that has a decent Ansible support.

To help you better understand how Ansible as a project works, here are some points from my personal adventure:

• To be able to contribute new modules to Ansible or any of the official collections, you need to implement extensive unit and integration tests. I understand the requirement. Ansible/Red Hat wants to maintain a high level of quality and to easily (and in automated way) detect any regressions. That's all good but implementing tests is harder and more work than implementing modules themselves. What's very very helpful in case of VMware is that there is a whole simulator called govcsim developed by VMware. You can test your modules against the simulator with ease and automate all the tests with little effort. To my knowledge, there is no simulator available for XenAPI. If such simulator does exist, it is most likely kept in secret by Citrix. If Citrix was ever to release this simulator, that would be a HUGE step forward.
• If you want to contribute new modules to Ansible or any of the official collections, someone has to review your code. Not many people are willing to do so and have the power to include your code to Ansible. As a matter of fact, finding reviewers and begging them for help is the hardest thing of all. I had some tremendous luck to acquire the interest of Abhijeet Kasurde, one of the top Ansible guys, to review my code and to eventually include xenserver_guest_* modules into Ansible. The guy handles VMware in Ansible... surprise! My xenserver_guest module was included without any unit or integration tests but for other modules I had to implement them. Luckily, they were simple and I had a luck to find a reviewer for tests also. When I wanted to upgrade xenserver_guest module with new functionality, they required unit and integration tests. I eventually implemented tests for xenserver_guest module but it was a huge undertaking and the amount of code involved easily dwarfed the module itself. I basically ended up implementing a barebone XenAPI simulator. This is where I hit a road block. No one, even the people that initially supported me, wanted to review this monstrosity of test+simulator. It was never included in Ansible.
• If you don't want to rely on external reviewers then you have to form a team, or if possible, a work group. That way you can review each others code and include it in Ansible without external support. Everything is pretty much handled by bots. If you gain a high enough status in Ansible project, you could get permissions to merge the code yourself without relying on anyone, not even bots. Should I mention that I failed to ever find any good Python programmer that is into Ansible and interested enough to form a team with me?
• You can skip all this struggle if you just maintain you own collection of modules but then you cannot rely on existing Ansible tooling that will do all the testing, linting, sanity checks, spell checks and such. You are on your own.

After a lot of struggle I eventually lost any interest as I was wasting a lot of time and life had to go on. Not much people showed interest in xenserver_guest_* Ansible modules either. My employer also ditched XenServer/XCP-ng in favor of VMware a few years back. Even with all the Broadcom/VMware situation, we got a super good deal with Broadcom because of our deployment size and commitment so we are sticking with VMWare.

All in all, if Ansible support for XenServer/XCP-ng and Xen Orchestra on par with VMware support is ever to see the light of day, these prerequisites are required:

• Publicly available XenAPI simulator is a must
• A working group of at least three people with knowledge in Python, Ansible and XenAPI committed to the cause
• Possibly corporate and financial backing by Citrix, Vates? or some other third party

Having any official Ansible support for XenServer/XCP-ng was (and is) a miracle to this day. A miracle I was blessed with and a huge learning experience for me.

Sorry for the long post. It is not my intention to discourage people but I think everyone should understand why XenServer/XCP-ng does not enjoy better Ansible support. There is much much more to it than just having a willingness to do anything.

XCP-NG / xen guest tools support network configuration through guest agent but for Windows guests only:

https://www.citrix.com/community/citrix-developer/citrix-hypervisor-developer/citrix-hypervisor-developing-products/citrix-hypervisor-staticip.html

or even better, the original article:

https://web.archive.org/web/20180218110151/https://xenserver.org/blog/entry/set-windows-guest-vm-static-ip-address-in-xenserver.html

Using the same mechanism (XAPI->xenstore), an agent for *nix guest can be implemented too. Unfortunately, no one has stepped in for that job. Citrix seems to be uninterested. I have a basic idea how to implement the agent but I lack free time to do so.