@CodeMercenary Thanks for providing this. My fans suddenly spun up like crazy and I used that github container to calm them down. Works a treat!

That package installs a container that runs every X seconds and connects to the server and resets the fans. I don't find that necessary. In fact, my fans went nuts after I installed a 3rd party (non-Dell) SAS controller. I notice that one of the IPMI parameters it sets is THIRD_PARTY_PCIE_CARD_COOLING_RESPONSE. I set that to false. It hasn't spun the fans up super high. I think setting that once was persistent and there's no need to have this script connecting every so often resetting the fans.

If anyone is looking for a one-time fix to this, I think running this command one time is sufficient.

@Danp As soon as I read that, I thought "I think I've been down this road before." Sure enough. I gave it 2G of RAM and it wouldn't build. I gave it 4G of RAM and I watched using systat and that final step went to about 52% of RAM usage. It has completed. I routinely run Xen Orchestra on a 1G VM because it's rarely used and it runs fine that way. But building it clearly takes a lot more RAM. I'll update my internal notes. It might be worth it to update the building from source docs to mention having enough RAM.

Thanks for hitting the nail on the head with that advice!

@olivierlambert You expect a system administrator to read documentation!?

Anyways, that was it. Sorry I didn't look more carefully. I rebuilt using node8 and poof it was running. Thanks guys.

I'm willing to contribute some documentation on templates, because I keep looking up how to work with them. Every so often another person posts some basic questions about them, and there isn't one central place to send them.

What I wanted to do was collect the basics of what I know here in this thread, and give people an opportunity to add other things they think should be in a documentation page on templates, and then I'll go write the page and contribute it to the docs.

My thinking is to organize it around "CRUD": Create, Read, Update, Delete.

Create

There's basically 3 ways to create templates.

1. From an existing VM. For example, you can install Linux or FreeBSD or whatever on your VM at some basic level with a few customizations, and then make a template from that.
2. Duplicate one of the default templates and then modify it (subject to the limitations in "Update" below)
3. Create a VM, then convert it to a template. The only difference between this and #1 above is that if you never boot it or install any OS, you create a template that has an unformatted disk. This just gives you customized RAM, CPU and other values. See the section on "Delete" where I talk about diskless templates. Am I understanding it right?

Any other useful ways to create that I haven't mentioned?

Read

How can I see the properties of a template in Xen Orchestra? If I click the hamburger menu, all I can see are tags, CPU and RAM. If I want to see other values, it seems like I have to start to create a VM, choose the template, and then click on the "Advanced Settings" to see what the template sets them to.

Pull from the API? This forum post mentions that. Is that this documentation on the xo command or something else? What should I link to for more information on invoking the API?

From the CLI you can get all the information from a template. If I've created a template called My Debian Template, this is how to see its values.

xe template-list name-label="My Debian Template"
uuid=$(xe template-list name-label="My Debian Template" --minimal)
xe template-param-list uuid=$uuid | less

Update

I can't find a way in Xen Orchestra to modify the parameters of a template. There's a thread here that talks about finding it, clicking the hamburger menu, then editing the values. When I do that, I see a little green check mark next to the value I changed, but the values don't actually change. Is this meant to work? And those are really basic settings. What about changing advanced settings in Xen Orchestra?

I guess there's a cumbersome way to modify a template in XO: I could create a VM from my template, modifying the values to what I want them to be, and then convert that VM to a new template. Then I delete the old template? Does that work?

The only way I have found to update a template is to set values from the command line following this post. For example (assuming $uuid is set from my prior example) this will set memory-static-min to 512Mb.

xe template-param-set memory-static-min=$((1024 * 1024 * 512)) uuid=$uuid

Any other ways to update the properties of an existing template?

Delete

It seems simple enough to find a template in Xen Orchestra, select the box next to it, and click the trash can. According to this post, that will leave orphan VDIs that have to be deleted manually. But this is not true of the default templates, right? So:

1. Deleting a default template: you just find it in the GUI and delete it.
2. Deleting a custom template you created: you need to do the steps in that post: find the drives, delete them, then delete the template?

This also leads me to ask whether it's possible to create a template like the default templates, where there's no associated disk? I guess you just create a VM without a disk and turn that into a template?

Also, you can delete the default templates, but they get re-installed again during a major version upgrade, right? So you either live with them, or you have to delete them again after every major upgrade?

Cloud-init and Cloudbase-Init

There's some useful docs on cloud-init and cloudbase-init that could move to this page if we have a whole page devoted to working with templates.

Thoughts?

What Am I Missing?

This is as much as I know right now. Anything else we should add to docs on templates?

@stormi Thanks. I appreciate it. But unfortunately, I'm unable to move workloads off the master in order to take it offline because of this situation.

If the solution is to turn off a host while it has live workloads, then I'm just going to shutdown the 8.2.1 slave and upgrade it to 8.3. Then I have 2 members in the pool and it's fully upgraded.

Let me tell you another edge case I encountered. There are some clear mistakes in here that I made, but it is related to this issue. When I took C offline and upgraded it to 8.3, I took the opportunity to convert it to UEFI boot. That meant reformatting the boot drive, not upgrading it. I wasn't worried about that. I took it out of the pool, reformatted it, and created a one-node 8.3 pool that has just node C in it. No biggie, right? I'll just have it join the pool with the 8.3 master and all is well. No, that's not going to work. Can I at least move some workloads onto it? Nope.

When you do a fresh install, pool-enable-certificate-verification defaults to yes. When you upgrade a pool, pool-enable-certificate-verification defaults to no. So I have a half-upgraded pool with 2 nodes with certificate verification disabled, and a single-node 8.3 pool with certificate verification enabled.

If I try to enable certificate verification on my half-upgraded pool? Our good friend NOT_SUPPORTED_DURING_UPGRADE() comes back to say "hi". As far as I can tell, it is not possible to disable certificate verification on the single-node 8.3 pool.

So I have a one-node pool where I can't turn verification off and a 2-node, half-upgraded pool where I can't turn it on. That makes it really difficult for the two pools to interoperate.

If I have to be known for something, let me be known as a cautionary tale.

@julien-f Yeah, I have just found the useSudo option in the docs. I swear things change fast enough on xen orchestra that "upgrading" almost doesn't make sense. I rebuild it like every 3-6 months and so much changes that it's almost easier to start over. If I had started over and pretended I didn't know anything, I would have read the documentation and seen the useSudo option. That's not a complaint. Fast development and adding features is awesome.

@stormi I really appreciate the continued help. I'm not sure it works.

On my single-host pool master—the one I want to join into my half-upgraded pool—I run the command:

xe host-emergency-disable-tls-verification

It returns with no errors.

If I do xe host-param-list and look through the parameters for my host, I see:

tls-verification-enabled ( RO): false

This is the only thing I see documented that can turn off TLS verification. If some other command like pool-uninstall-ca-certificate would have the effect of disabling TLS verification, I can't see that documented anywhere. In fact, even though xe pool-uninstall-ca-certificate appears to be a valid command that my xe binary knows about, I can't find that particular command documented at all.

Even after the emergency-disable-tls-verification, if I attempt to join that host to the half-upgraded pool (using xe pool-join...) I get:

Error code: POOL_JOINING_HOST_TLS_VERIFICATION_MISMATCH

It seems like even though the host has TLS verification off, the fact that its pool has verification enabled is blocking the action.

@stormi Thanks. I appreciate it. But unfortunately, I'm unable to move workloads off the master in order to take it offline because of this situation.

If the solution is to turn off a host while it has live workloads, then I'm just going to shutdown the 8.2.1 slave and upgrade it to 8.3. Then I have 2 members in the pool and it's fully upgraded.

Let me tell you another edge case I encountered. There are some clear mistakes in here that I made, but it is related to this issue. When I took C offline and upgraded it to 8.3, I took the opportunity to convert it to UEFI boot. That meant reformatting the boot drive, not upgrading it. I wasn't worried about that. I took it out of the pool, reformatted it, and created a one-node 8.3 pool that has just node C in it. No biggie, right? I'll just have it join the pool with the 8.3 master and all is well. No, that's not going to work. Can I at least move some workloads onto it? Nope.

When you do a fresh install, pool-enable-certificate-verification defaults to yes. When you upgrade a pool, pool-enable-certificate-verification defaults to no. So I have a half-upgraded pool with 2 nodes with certificate verification disabled, and a single-node 8.3 pool with certificate verification enabled.

If I try to enable certificate verification on my half-upgraded pool? Our good friend NOT_SUPPORTED_DURING_UPGRADE() comes back to say "hi". As far as I can tell, it is not possible to disable certificate verification on the single-node 8.3 pool.

So I have a one-node pool where I can't turn verification off and a 2-node, half-upgraded pool where I can't turn it on. That makes it really difficult for the two pools to interoperate.

If I have to be known for something, let me be known as a cautionary tale.

@olivierlambert Sadly, no. And now that I'm in the middle of upgrading, I can't create storages. I could stand up an NFS server with some shared storage to help. Unfortunately, every attempt at creating a storage (NFS or otherwise) results in NOT_SUPPORTED_DURING_UPGRADE(). If I had created shared storage before I upgraded the master, I could use it. But now that I'm part-way through the upgrade process, I can't.

@Pilow I detached that VDI from the VM and the command failed for the same reason, just complaining about the other VDI.

@olivierlambert Thanks.

I found this Citrix page ("Live migration within a pool that doesn't have shared storage by using the xe CLI") that seems to correspond to what I'm doing. (The 2 hosts have no shared storage)

I ran:
xe vm-migrate uuid=00b4cf39-f954-6ab3-9977-c4c2809c5324 remote-master=<A> remote-username=root remote-password='stuff' host-uuid=<A's uuid>

I got the following results:

Performing a storage live migration. Your VM's VDIs will be migrated with the VM.

Will migrate to remote host: A, using remote network: internal. Here is the VDI mapping:
VDI 8e8a2679-cf0d-44c1-a3dd-f69edc82d849 -> SR 5bb37e13-61d7-69b3-7de3-091a7866c4d8
VDI 3c9f7815-0547-4237-949d-27ac3d80b4a6 -> SR 5bb37e13-61d7-69b3-7de3-091a7866c4d8

The requested operation is not allowed for VDIs with CBT enabled or VMs having such VDIs, and CBT is enabled for the specified VDI.
vdi: 8e8a2679-cf0d-44c1-a3dd-f69edc82d849 (XO CloudConfigDrive omd)

So I ran: xe vdi-list uuid=8e8a2679-cf0d-44c1-a3dd-f69edc82d849

uuid ( RO)                : 8e8a2679-cf0d-44c1-a3dd-f69edc82d849
          name-label ( RW): XO CloudConfigDrive omd
    name-description ( RW):
             sr-uuid ( RO): e258dec5-d1b1-ceef-b489-f2a2d219bf9b
        virtual-size ( RO): 10485760
            sharable ( RO): false
           read-only ( RO): false

The VDI does have CBT enabled. The VM has 2 VDIs. Both have CBT enabled. Neither VDI has any current snapshots.

I ran xe vdi-disable-cbt uuid=8e8a2679-cf0d-44c1-a3dd-f69edc82d849 (and for the other VDI). For both VDIs I get

This operation is not supported during an upgrade.

Any thoughts?

One more bit of data. It might be that a specific host has a problem. Because I'm changing racks, I was only trying to evacuate one specific host. That's one host is failing. But the other 8.2.1 slave in the cluster can migrate to the 8.3 master just fine. If B (8.2.1) => A (8.3) and C (8.2.1)=> B (8.2.1), then C is empty and can be upgraded. It's convoluted, but if it works, that's fine. I'll know in a couple hours whether this at least gives me a path forward.

@olivierlambert Are you saying that I will be able to move from an 8.2.1 slave to an 8.3 slave, but I can't move from an 8.2.1 slave to the 8.3 master?

My context (I was too brief) is that I upgraded the pool master to 8.3. The pool is up and mostly seems normal. When I try to move a VM from an 8.2.1 slave in the same pool to the 8.3 master, I get this NOT_SUPPORTED_DURING_UPGRADE() error. I'm clicking migrate in Xen Orchestra on a stopped or a running VM on an 8.2.1 slave, targeting the 8.3, master. If I could migrate to the master, I would be fine. Any 2 of my 3 hosts can run everything while the third does its upgrade.

Maybe something is bugged in my setup. It sounds like this is unexpected.

My Xen Orchestra is from open source, Xen Orchestra, commit 88b88, Master, commit 1640a
The master XCP-ng host is running 8.3.0 fully patched as of yesterday
Both slaves XCP-ng hosts are running 8.2.1 fully patched

What should I check?

I am running into the same issue as in this post. But I'm confused as to how one upgrades a cluster of hosts from 8.2.1 to 8.3 without massive downtime. I have 3 hosts, A, B, and C. A is the master. I moved all the workloads off of A, and then upgraded it to 8.3.

I'd like to move workloads off one of the slaves, so the slave can take as long as necessary to upgrade. The upgrade is not quick.

The only way to upgrade from 8.2.1 to 8.3 is to boot from the ISO, which is fine. But once a node is upgraded, I can't migrate workloads to it from the non-upgraded nodes. How do I roll this upgrade through the cluster without just taking an entire host and all its workloads offline for 45 minutes while it upgrades?

I have been able to move workloads from old to new by shutting down a VM on an old node, using the copy function in Xen Orchestra to copy it to the upgraded master, and then booting the new copy. But that takes the VM offline for the duration of the copy. A few of my VMs can tolerate that, but not many.

What am I missing?

This is an old thread, but I ran into this myself recently. While there is a link to deleting the entire XO configuration, I think I fixed it with a less drastic solution. Remember: I'm just a random dude on the internet posting dangerous commands to try. It worked for me, but your mileage might vary.

I run Xen Orchestra in a container (Xen Orchestra, commit e8733 at the time of writing). So I got a command line in the container with:

docker exec -it xoa bash

Then I ran redis-cli to get a redis command prompt. I typed KEYS * to get a list of keys. One key I saw was: 7) xo:server_host:172.30.0.214". That's the IP of the host I was trying to join (the master of a single-host pool). So I ran:

del xo:server_host:172.30.0.214

Then I restarted my container with docker restart xoa. After that, I was able to successfully add the host to Xen Orchestra.

Maybe this will help someone else. It got me working again.

@CodeMercenary I spoke too soon. Came in last night and it sounded like it was ready for take-off. So I restarted the container. This is so ridiculous...

I think I did get 24-48 hours of quiet. So, a recheck interval of many minutes is probably fine. I think by default they start at like 5 seconds, which is needless.