XO Packer template disk issues
-
Update 1
I rebuilt my template with all of the exact same settings but instead of cloning from the "Ubuntu Focal Fossa 20.04" default template within XO, I told Packer to clone from the "Other install media" default template. Doing that now makes XO show the cloudinit settings when creating a vm from the Packer generated template. I dont know why changing what template im cloning from changes the results but I think this is still a issue that needs to be fixed. To me it seems like its a disconnect somewhere between Packer and XO. On one side or the other its applying some parameters or not applying some parameters thats needed to see the existing disk. Any ideas?
Debug Logs
For clarity im attaching the same debug info that I did for the other two templates. The new template that I created has a name label of "ubuntu-2004-server-test-golden-v2023.01" and a UUID of "42d967db-44f9-8ba5-37b7-2a09da8b5b7e"
xe template-param-list uuid=<templateUUID>
ubuntu-2004-server-test-golden-v2023.01
xe template-param-list uuid=42d967db-44f9-8ba5-37b7-2a09da8b5b7euuid ( RO) : 42d967db-44f9-8ba5-37b7-2a09da8b5b7e name-label ( RW): ubuntu-2004-server-test-golden-v2023.01 name-description ( RW): Version: v2023.01 | Built on: 2023-01-11 00:33 UTC | Built by: HashiCorp Packer 1.8.5 user-version ( RW): 1 is-a-template ( RW): true is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): <not in database> memory-target ( RO): 0 memory-overhead ( RO): 37748736 memory-static-max ( RW): 4294967296 memory-dynamic-max ( RW): 4294967296 memory-dynamic-min ( RW): 4294967296 memory-static-min ( RW): 4294967296 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): VCPUs-max ( RW): 2 VCPUs-at-startup ( RW): 2 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: 1; device-model: qemu-upstream-compat; viridian: false; nx: true; pae: true; apic: true; acpi: 1 allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8 domain-type ( RW): hvm current-domain-type ( RO): <not in database> HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: cd; firmware: bios HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): base_template_name: Other install media; import_task: OpaqueRef:02ba17a5-f1c4-4bdd-826b-a980c3d89869; mac_seed: dea90f6d-ee36-57a5-60fd-bef7554ce70b; install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="yes"/><restriction field="supports-secure-boot" value="yes"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): <unknown time> install-time ( RO): <unknown time> VCPUs-number ( RO): <not in database> VCPUs-utilisation (MRO): os-version (MRO): name: Ubuntu 20.04.5 LTS; uname: 5.4.0-136-generic; distro: ubuntu; major: 20; minor: 04 PV-drivers-version (MRO): major: 6; minor: 6; micro: 80; build: 0 PV-drivers-up-to-date ( RO) [DEPRECATED]: false memory (MRO): disks (MRO): VBDs (SRO): 7bf8749b-16a5-5b0a-3351-1271396ecd49 networks (MRO): 0/ip: 172.18.20.218; 0/ipv4/0: 172.18.20.218; 0/ipv6/0: fe80::90f2:61ff:fe17:8221 PV-drivers-detected ( RO): false other (MRO): shutdown: ; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-vcpu-hotplug: 1; feature-suspend: 1; feature-reboot: 1; feature-poweroff: 1; feature-balloon: 1 live ( RO): true guest-metrics-last-updated ( RO): 20230111T00:44:17Z can-use-hotplug-vbd ( RO): unspecified can-use-hotplug-vif ( RO): unspecified cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): other-install-media bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3dxe vbd-list | grep <templateUUID> -B1 -A2
ubuntu-2004-server-test-golden-v2023.01
xe vbd-list | grep 42d967db-44f9-8ba5-37b7-2a09da8b5b7e -B1 -A2uuid ( RO) : 7bf8749b-16a5-5b0a-3351-1271396ecd49 vm-uuid ( RO): 42d967db-44f9-8ba5-37b7-2a09da8b5b7e vm-name-label ( RO): ubuntu-2004-server-test-golden-v2023.01 vdi-uuid ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117xe vbd-param-list uuid=<template-VBD-UUID>
ubuntu-2004-server-test-golden-v2023.01
xe vbd-param-list uuid=7bf8749b-16a5-5b0a-3351-1271396ecd49uuid ( RO) : 7bf8749b-16a5-5b0a-3351-1271396ecd49 vm-uuid ( RO): 42d967db-44f9-8ba5-37b7-2a09da8b5b7e vm-name-label ( RO): ubuntu-2004-server-test-golden-v2023.01 vdi-uuid ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117 vdi-name-label ( RO): Packer-disk allowed-operations (SRO): attach current-operations (SRO): empty ( RO): false device ( RO): xvda userdevice ( RW): 0 bootable ( RW): false mode ( RW): RW type ( RW): Disk unpluggable ( RW): false currently-attached ( RO): false attachable ( RO): true storage-lock ( RO): false status-code ( RO): 0 status-detail ( RO): qos_algorithm_type ( RW): qos_algorithm_params (MRW): qos_supported_algorithms (SRO): other-config (MRW): io_read_kbs ( RO): <unknown> io_write_kbs ( RO): <unknown>xe vdi-param-list uuid=<template-VDI-UUID>
ubuntu-2004-server-test-golden-v2023.01
xe vdi-param-list uuid=5c9214b3-231f-4b5c-ab96-7c1bdbbb6117uuid ( RO) : 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117 name-label ( RW): Packer-disk name-description ( RW): is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone current-operations (SRO): sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc sr-name-label ( RO): m3i-nfs-vm-data-ssd vbd-uuids (SRO): 7bf8749b-16a5-5b0a-3351-1271396ecd49 crashdump-uuids (SRO): virtual-size ( RO): 10737418240 physical-utilisation ( RO): 24064 location ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117 type ( RO): User sharable ( RO): false read-only ( RO): false storage-lock ( RO): false managed ( RO): true parent ( RO) [DEPRECATED]: <not in database> missing ( RO): false is-tools-iso ( RO): false other-config (MRW): content_id: 10d4bc7b-9d36-ce3b-83a0-3c4baed0c8e1; temp: temp xenstore-data (MRO): sm-config (MRO): on-boot ( RW): persist allow-caching ( RW): false metadata-latest ( RO): false metadata-of-pool ( RO): <not in database> tags (SRW): cbt-enabled ( RO): falseEdit: added debug logs
-
@marcungeschikts someone in XO team should take a look. It's not overcomplicated, just a "logic behind a choice we made" and see if it's still relevant. If it is, what should be done to get Cloudinit displayed in the end.
-
I investigated this with @BMeach in Discord (link). The issue is that the broken packer run is creating a template that contains a populated
disksproperty in theother-configparameter (more details here).XO's logic for hiding this is located here. I don't know the historical context, but when I initially created the terraform provider I modeled VM creation off that logic in order to get the provider's
vm.startRPC calls to succeed with cloud-init.I will investigate why packer is doing something differently the manual terraform template creation process described here, but I'm also interested in hearing more from the XO team for why it is setup this way.
-
So it's been a long time we did that, so I can't trust at 100% my memory. That's what I recall ( @julien-f correct me if I'm wrong)
Default templates
For "default templates" (templates without any actual disk present, ie coming from the XCP-ng installation), the
disksfield contains a list of suggested disks to generate on VM creation. It's a guide that can be fetched by XAPI clients (XO, XenCenter) to pre-fill the disks creation fields in their respective UI. Eg: a Windows template will usually advise to create a rather large disk (40GiB) by default vs a Linux template. XenCenter and XO aren't aware of those things, they just fetch the "provision" to display what's expected in the template itself to present it to the user.During the VM creation, this
provisionfield should disappear (by XAPI, automatically, on provisioning), since it's not needed anymore: the VM is now created, and it won't be an empty template anymore. Either it stays a regular VM, or it becomes a template with a disk (see below).User created templates
Any other template that the "original/default" ones will very likely already contain a disk with a system installed (whatever this system is). Only those templates can be used with cloudinit, because you need to have a disk where cloudinit is installed.
A way to detect that a template was provisioned in the past is to rely on this information. There's one exception (again IIRC): Other install media, which is a special template without any guidance on disks to be created since it's a "misc" template.
-
I can confirm that XAPI is removing the
diskskey inother_configduring theVM.provisioncall: -
So maybe the Packer plugin isn't doing what it should to create the template? We probably need to discuss this together
-
@olivierlambert thanks for that background on the Default vs user created templates and that matches my mental model of how things work with XO and terraform -- that a template must have an installed OS (disk) in order to work.
So maybe the Packer plugin isn't doing what it should to create the template?
As for what packer is doing, it's using the xapi VM.set_is_a_template call.
I believe this is identical to what XO does when you convert a VM to a template via the Advanced tab (source). That handler ultimately sets is a template, which seems equivalent to what packer is calling.
I definitely agree we need to discuss what the right solution is, but from the information I shared above I'm still unsure that we understand why a packer built template is behaving the way it is.
-
I can confirm that XAPI is removing the
diskskey inother_configduring theVM.provisioncall:I understand very little ocaml, but is the code that you linked to part of the VM creation lifecycle? It seems like it's a performance test from what I can tell.
-
Okay so after closer inspection, it's not XAPI but the client responsability to clean this key after install, as we do it in XO:
https://github.com/vatesfr/xen-orchestra/blob/e6b893977205a90f5d7b96e8e0e0f65b030fa195/packages/xo-server/src/xapi/mixins/vm.mjs#L83-L85C37// Removes disks from the provision XML, we will create them by // ourselves. await this.call('VM.remove_from_other_config', vmRef, 'disks')::ignoreErrors()However, it appears the be the standard way to do so. Let's see, as a first hint, the examples given by XAPI project on their Python bindings:
https://github.com/xapi-project/xen-api/blob/14ee043bcabf16cb58c2fbb974fd90ee81fb067e/scripts/examples/python/provision.py#L86session.xenapi.VM.remove_from_other_config(vm, "disks")XenCenter is also doing exactly this:
// Removes disks from the provision XML, we will create them by // ourselves. await this.call('VM.remove_from_other_config', vmRef, 'disks')::ignoreErrors()So the issue is the Packer plugin not having this "normal" behavior, and leading to the original problem
In other words, a template with this key is considered as "without any already installed system" and adding Cloudinit config for it doesn't make sense at all => the logic is good.Maybe it's another argument to get a Packer plugin on top of XO API to hide those implementation details…
-
@olivierlambert I think I have found a fix on the Packer side like you were talking about. I have the templates working being built from the "Ubuntu Focal Fossa 20.04" default template. I created a pull request here let me know what you think, this is my first excursion with golang
-
@olivierlambert ah, nice find and that explains the final missing piece. The xenserver packer plugin should definitely follow the same steps as XO and XenCenter.
Definitely agree that it would be worth considering a XO specific packer plugin, but for now that should address this bug.
-
The ddelnano/packer-plugin-xenserver release v0.5.1 has been released and includes @bagas's fix for this issue.
