XO Packer template disk issues
-
To provide some background I was talking with olivierlambert over on ddelnano's Discord about an issue I was having with the cloudinit options not showing up in XO on any templates I made with the XenServer packer.io builder made by ddelnano.
xcp-ng / xen orchestra setup
- XCP-NG Hosts
- 3x Dell r630's
- XCP-NG Version: XCP-ng 8.2.1
- Build Number: release/yangtze/master/58
- Xen Orchestra
- Using this install script
- commit f95a2
- xo-server 5.107.4
- xo-web 5.108.0
- 4 vcpu's, 8G RAM, 2 VIFS (one to mgmt net, the other to the storage net), 25G disk
What im trying to do
Im trying to have packer run every month via a CI/CD pipeline to generate a golden vm template every month. My goal for the template is to have it "cloudinit ready" so I can use it in my Terraform configs to provision workload vm's. I also want Packer to use Ansible to install basic system monitoring utilities such as netdata and any other necessary packages for example the xe-guest-utilities.
Where I am right now
Right now I have got Packer working up to a degree, I can create the template completely fine. I have gotten Packer to run my ansible playbooks on the host and have also used the file provisioner and shell provisioner to test installing packages and moving files onto the template vm. Where I am stuck though is getting the cloudinit options to show up inside of XO. I have validated multiple times that my vm template has a disk attached to it and if I create a vm from the Packer generated template I get the options shown below
As you can see the disk from the template is there for sure under the name "Packer-disk" but I still get prompted to insert a iso/dvd instead of prompting me for a ssh key or cloudinit config which by my understanding it should show those feilds if there is a disk in the template which there is but thats obviously not working. If I continue and create the vm with the options as shown below it boots into a fully installed ubuntu 20.04 os and is configured just like it should be with the ansible playbooks, shell provisioner, and file provisioner taking affect and working perfectly. The issue is there is no cloudinit disk which leaves the cloudinit config in the os "broken" due to there being nothing for cloudinit's nocloud or configdrive datasource to read.
Troubleshooting
The first thing that I did to troubleshoot is I downloaded the Ubuntu 20.04 cloudimg ova file and imported it into XO through the VM import utility in the web console. I then immediately turned it into a template and tried to create a vm from that and that showed all of the cloudinit options as expected.
The template has a name label of "ubuntu-focal-20.04-cloudimg" and the template UUID is "d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8". I then wanted to compare the template parameters so I went and looked for my Packer generated template, it has a name label of "ubuntu-2004-server-golden-v2023.01" and a UUID of "422081e2-6b73-f642-fc82-6cba4313a9da". From there I then went out and got the following data that has been posted in the Debug Logs Section, a list of the template parameters, a list of vbd's attached to each template, a list of the vbd's parameters, and a list of the vdi parameters.
Ive only been using the XCP-NG and XO platform for about 5-6 months so I am sorry if there is somewhat obvious diagnostic data missing, let me know and im happy to gather any information needed to resolve this issue. From my limited knowledge this seems like its somehow related to XO not seeing the attached disk on the template or the Packer plugin not setting some parameters that are required for XO to see the attached disk when converting to a template. If anyone has any suggestions or any ways to help me out it would be very appreciated.
Debug Logs
xe template-param-list uuid=<templateUUID>
ubuntu-focal-20.04-cloudimg
xe template-param-list uuid=d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8uuid ( RO) : d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8 name-label ( RW): ubuntu-focal-20.04-cloudimg name-description ( RW): Ubuntu Linux (64-bit) user-version ( RW): 1 is-a-template ( RW): true is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): <not in database> memory-target ( RO): 0 memory-overhead ( RO): 12582912 memory-static-max ( RW): 1073741824 memory-dynamic-max ( RW): 1073741824 memory-dynamic-min ( RW): 1073741824 memory-static-min ( RW): 1073741824 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): VCPUs-max ( RW): 2 VCPUs-at-startup ( RW): 2 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: 0; nx: true; acpi: 1; apic: true; pae: true; hpet: true; viridian: true allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8 domain-type ( RW): hvm current-domain-type ( RO): <not in database> HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: cdn HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): mac_seed: 7fa98bf1-ff15-c50a-0ea4-92d46cee2e3a; vgpu_pci: ; base_template_name: Other install media; install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472" /><restriction field="vcpus-max" max="32" /><restriction property="number-of-vbds" max="255" /><restriction property="number-of-vifs" max="7" /><restriction field="has-vendor-device" value="false" /></restrictions> xenstore-data (MRW): ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): <unknown time> install-time ( RO): <unknown time> VCPUs-number ( RO): <not in database> VCPUs-utilisation (MRO): os-version (MRO): <not in database> PV-drivers-version (MRO): <not in database> PV-drivers-up-to-date ( RO) [DEPRECATED]: <not in database> memory (MRO): <not in database> disks (MRO): <not in database> VBDs (SRO): 6d154703-7f69-d72d-cbab-b62be607b65b networks (MRO): <not in database> PV-drivers-detected ( RO): <not in database> other (MRO): <not in database> live ( RO): <not in database> guest-metrics-last-updated ( RO): <not in database> can-use-hotplug-vbd ( RO): <not in database> can-use-hotplug-vif ( RO): <not in database> cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 0:0 hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): bios-strings (MRO):ubuntu-2004-server-golden-v2023.01
xe template-param-list uuid=422081e2-6b73-f642-fc82-6cba4313a9dauuid ( RO) : 422081e2-6b73-f642-fc82-6cba4313a9da name-label ( RW): ubuntu-2004-server-golden-v2023.01 name-description ( RW): Version: v2023.01 | Built on: 2023-01-10 22:00 UTC | Built by: HashiCorp Packer 1.8.5 user-version ( RW): 1 is-a-template ( RW): true is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): <not in database> memory-target ( RO): 0 memory-overhead ( RO): 37748736 memory-static-max ( RW): 4294967296 memory-dynamic-max ( RW): 4294967296 memory-dynamic-min ( RW): 4294967296 memory-static-min ( RW): 4294967296 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): VCPUs-max ( RW): 2 VCPUs-at-startup ( RW): 2 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: 1; device-model: qemu-upstream-compat; viridian: false; nx: true; pae: true; apic: true; acpi: 1 allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8 domain-type ( RW): hvm current-domain-type ( RO): <not in database> HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: cd; firmware: bios HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): base_template_name: Ubuntu Focal Fossa 20.04; import_task: OpaqueRef:1373f40a-13bb-4749-b314-546a6513ce1e; mac_seed: d16763c1-691e-ac82-01fe-056877eb0b75; disks: <provision><disk bootable="true" device="0" size="10737418240" sr="" type="system"/></provision>; install-methods: cdrom,nfs,http,ftp; linux_template: true dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="allow-gpu-passthrough" value="1"/><restriction field="allow-vgpu" value="1"/><restriction field="allow-network-sriov" value="1"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="no"/><restriction field="supports-secure-boot" value="no"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): <unknown time> install-time ( RO): <unknown time> VCPUs-number ( RO): <not in database> VCPUs-utilisation (MRO): os-version (MRO): name: Ubuntu 20.04.5 LTS; uname: 5.4.0-136-generic; distro: ubuntu; major: 20; minor: 04 PV-drivers-version (MRO): major: 6; minor: 6; micro: 80; build: 0 PV-drivers-up-to-date ( RO) [DEPRECATED]: false memory (MRO): disks (MRO): VBDs (SRO): 455a4d18-37d7-0083-3123-868ee0ec832f networks (MRO): 0/ip: 172.18.20.217; 0/ipv4/0: 172.18.20.217; 0/ipv6/0: fe80::8cc3:15ff:fe81:ce1b PV-drivers-detected ( RO): false other (MRO): shutdown: ; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-vcpu-hotplug: 1; feature-suspend: 1; feature-reboot: 1; feature-poweroff: 1; feature-balloon: 1 live ( RO): true guest-metrics-last-updated ( RO): 20230110T22:11:48Z can-use-hotplug-vbd ( RO): unspecified can-use-hotplug-vif ( RO): unspecified cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): ubuntu-20.04 bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3dxe vbd-list | grep <templateUUID> -B1 -A2
ubuntu-focal-20.04-cloudimg
xe vbd-list | grep d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8 -B1 -A2uuid ( RO) : 6d154703-7f69-d72d-cbab-b62be607b65b vm-uuid ( RO): d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8 vm-name-label ( RO): ubuntu-focal-20.04-cloudimg vdi-uuid ( RO): db3b06bf-d292-4a00-b916-c496b9de8aa5ubuntu-2004-server-golden-v2023.01
xe vbd-list | grep 422081e2-6b73-f642-fc82-6cba4313a9da -B1 -A2uuid ( RO) : 455a4d18-37d7-0083-3123-868ee0ec832f vm-uuid ( RO): 422081e2-6b73-f642-fc82-6cba4313a9da vm-name-label ( RO): ubuntu-2004-server-golden-v2023.01 vdi-uuid ( RO): 9c1403bb-fde9-415f-9ae1-9511b84a60bbxe vbd-param-list uuid=<template-VBD-UUID>
ubuntu-focal-20.04-cloudimg
xe vbd-param-list uuid=6d154703-7f69-d72d-cbab-b62be607b65buuid ( RO) : 6d154703-7f69-d72d-cbab-b62be607b65b vm-uuid ( RO): d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8 vm-name-label ( RO): ubuntu-focal-20.04-cloudimg vdi-uuid ( RO): db3b06bf-d292-4a00-b916-c496b9de8aa5 vdi-name-label ( RO): Hard Disk 1 allowed-operations (SRO): attach current-operations (SRO): empty ( RO): false device ( RO): userdevice ( RW): 0 bootable ( RW): false mode ( RW): RW type ( RW): Disk unpluggable ( RW): false currently-attached ( RO): false attachable ( RO): true storage-lock ( RO): false status-code ( RO): 0 status-detail ( RO): qos_algorithm_type ( RW): qos_algorithm_params (MRW): qos_supported_algorithms (SRO): other-config (MRW): io_read_kbs ( RO): <unknown> io_write_kbs ( RO): <unknown>ubuntu-2004-server-golden-v2023.01
xe vbd-param-list uuid=455a4d18-37d7-0083-3123-868ee0ec832fuuid ( RO) : 455a4d18-37d7-0083-3123-868ee0ec832f vm-uuid ( RO): 422081e2-6b73-f642-fc82-6cba4313a9da vm-name-label ( RO): ubuntu-2004-server-golden-v2023.01 vdi-uuid ( RO): 9c1403bb-fde9-415f-9ae1-9511b84a60bb vdi-name-label ( RO): Packer-disk allowed-operations (SRO): attach current-operations (SRO): empty ( RO): false device ( RO): xvda userdevice ( RW): 0 bootable ( RW): false mode ( RW): RW type ( RW): Disk unpluggable ( RW): false currently-attached ( RO): false attachable ( RO): true storage-lock ( RO): false status-code ( RO): 0 status-detail ( RO): qos_algorithm_type ( RW): qos_algorithm_params (MRW): qos_supported_algorithms (SRO): other-config (MRW): io_read_kbs ( RO): <unknown> io_write_kbs ( RO): <unknown>xe vdi-param-list uuid=<template-VDI-UUID>
ubuntu-focal-20.04-cloudimg
xe vdi-param-list uuid=db3b06bf-d292-4a00-b916-c496b9de8aa5uuid ( RO) : db3b06bf-d292-4a00-b916-c496b9de8aa5 name-label ( RW): Hard Disk 1 name-description ( RW): No description is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone current-operations (SRO): sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc sr-name-label ( RO): m3i-nfs-vm-data-ssd vbd-uuids (SRO): 6d154703-7f69-d72d-cbab-b62be607b65b crashdump-uuids (SRO): virtual-size ( RO): 10737418240 physical-utilisation ( RO): 24064 location ( RO): db3b06bf-d292-4a00-b916-c496b9de8aa5 type ( RO): User sharable ( RO): false read-only ( RO): false storage-lock ( RO): false managed ( RO): true parent ( RO) [DEPRECATED]: <not in database> missing ( RO): false is-tools-iso ( RO): false other-config (MRW): content_id: 3429ce87-38f3-c3ac-ac9a-ed373355c24f xenstore-data (MRO): sm-config (MRO): vhd-parent: e949503d-018c-4630-9dc8-56354fe93340 on-boot ( RW): persist allow-caching ( RW): false metadata-latest ( RO): false metadata-of-pool ( RO): <not in database> tags (SRW): cbt-enabled ( RO): falseubuntu-2004-server-golden-v2023.01
xe vdi-param-list uuid=9c1403bb-fde9-415f-9ae1-9511b84a60bbuuid ( RO) : 9c1403bb-fde9-415f-9ae1-9511b84a60bb name-label ( RW): Packer-disk name-description ( RW): is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone current-operations (SRO): sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc sr-name-label ( RO): m3i-nfs-vm-data-ssd vbd-uuids (SRO): 455a4d18-37d7-0083-3123-868ee0ec832f crashdump-uuids (SRO): virtual-size ( RO): 10737418240 physical-utilisation ( RO): 24064 location ( RO): 9c1403bb-fde9-415f-9ae1-9511b84a60bb type ( RO): User sharable ( RO): false read-only ( RO): false storage-lock ( RO): false managed ( RO): true parent ( RO) [DEPRECATED]: <not in database> missing ( RO): false is-tools-iso ( RO): false other-config (MRW): content_id: 3958b983-1b38-ebb2-55d3-e76a137524ca; temp: temp xenstore-data (MRO): sm-config (MRO): on-boot ( RW): persist allow-caching ( RW): false metadata-latest ( RO): false metadata-of-pool ( RO): <not in database> tags (SRW): cbt-enabled ( RO): falseEdit: formatting
- XCP-NG Hosts
-
Update 1
I rebuilt my template with all of the exact same settings but instead of cloning from the "Ubuntu Focal Fossa 20.04" default template within XO, I told Packer to clone from the "Other install media" default template. Doing that now makes XO show the cloudinit settings when creating a vm from the Packer generated template. I dont know why changing what template im cloning from changes the results but I think this is still a issue that needs to be fixed. To me it seems like its a disconnect somewhere between Packer and XO. On one side or the other its applying some parameters or not applying some parameters thats needed to see the existing disk. Any ideas?
Debug Logs
For clarity im attaching the same debug info that I did for the other two templates. The new template that I created has a name label of "ubuntu-2004-server-test-golden-v2023.01" and a UUID of "42d967db-44f9-8ba5-37b7-2a09da8b5b7e"
xe template-param-list uuid=<templateUUID>
ubuntu-2004-server-test-golden-v2023.01
xe template-param-list uuid=42d967db-44f9-8ba5-37b7-2a09da8b5b7euuid ( RO) : 42d967db-44f9-8ba5-37b7-2a09da8b5b7e name-label ( RW): ubuntu-2004-server-test-golden-v2023.01 name-description ( RW): Version: v2023.01 | Built on: 2023-01-11 00:33 UTC | Built by: HashiCorp Packer 1.8.5 user-version ( RW): 1 is-a-template ( RW): true is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): <not in database> memory-target ( RO): 0 memory-overhead ( RO): 37748736 memory-static-max ( RW): 4294967296 memory-dynamic-max ( RW): 4294967296 memory-dynamic-min ( RW): 4294967296 memory-static-min ( RW): 4294967296 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): VCPUs-max ( RW): 2 VCPUs-at-startup ( RW): 2 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: 1; device-model: qemu-upstream-compat; viridian: false; nx: true; pae: true; apic: true; acpi: 1 allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8 domain-type ( RW): hvm current-domain-type ( RO): <not in database> HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: cd; firmware: bios HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): base_template_name: Other install media; import_task: OpaqueRef:02ba17a5-f1c4-4bdd-826b-a980c3d89869; mac_seed: dea90f6d-ee36-57a5-60fd-bef7554ce70b; install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="yes"/><restriction field="supports-secure-boot" value="yes"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): <unknown time> install-time ( RO): <unknown time> VCPUs-number ( RO): <not in database> VCPUs-utilisation (MRO): os-version (MRO): name: Ubuntu 20.04.5 LTS; uname: 5.4.0-136-generic; distro: ubuntu; major: 20; minor: 04 PV-drivers-version (MRO): major: 6; minor: 6; micro: 80; build: 0 PV-drivers-up-to-date ( RO) [DEPRECATED]: false memory (MRO): disks (MRO): VBDs (SRO): 7bf8749b-16a5-5b0a-3351-1271396ecd49 networks (MRO): 0/ip: 172.18.20.218; 0/ipv4/0: 172.18.20.218; 0/ipv6/0: fe80::90f2:61ff:fe17:8221 PV-drivers-detected ( RO): false other (MRO): shutdown: ; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-vcpu-hotplug: 1; feature-suspend: 1; feature-reboot: 1; feature-poweroff: 1; feature-balloon: 1 live ( RO): true guest-metrics-last-updated ( RO): 20230111T00:44:17Z can-use-hotplug-vbd ( RO): unspecified can-use-hotplug-vif ( RO): unspecified cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): other-install-media bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3dxe vbd-list | grep <templateUUID> -B1 -A2
ubuntu-2004-server-test-golden-v2023.01
xe vbd-list | grep 42d967db-44f9-8ba5-37b7-2a09da8b5b7e -B1 -A2uuid ( RO) : 7bf8749b-16a5-5b0a-3351-1271396ecd49 vm-uuid ( RO): 42d967db-44f9-8ba5-37b7-2a09da8b5b7e vm-name-label ( RO): ubuntu-2004-server-test-golden-v2023.01 vdi-uuid ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117xe vbd-param-list uuid=<template-VBD-UUID>
ubuntu-2004-server-test-golden-v2023.01
xe vbd-param-list uuid=7bf8749b-16a5-5b0a-3351-1271396ecd49uuid ( RO) : 7bf8749b-16a5-5b0a-3351-1271396ecd49 vm-uuid ( RO): 42d967db-44f9-8ba5-37b7-2a09da8b5b7e vm-name-label ( RO): ubuntu-2004-server-test-golden-v2023.01 vdi-uuid ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117 vdi-name-label ( RO): Packer-disk allowed-operations (SRO): attach current-operations (SRO): empty ( RO): false device ( RO): xvda userdevice ( RW): 0 bootable ( RW): false mode ( RW): RW type ( RW): Disk unpluggable ( RW): false currently-attached ( RO): false attachable ( RO): true storage-lock ( RO): false status-code ( RO): 0 status-detail ( RO): qos_algorithm_type ( RW): qos_algorithm_params (MRW): qos_supported_algorithms (SRO): other-config (MRW): io_read_kbs ( RO): <unknown> io_write_kbs ( RO): <unknown>xe vdi-param-list uuid=<template-VDI-UUID>
ubuntu-2004-server-test-golden-v2023.01
xe vdi-param-list uuid=5c9214b3-231f-4b5c-ab96-7c1bdbbb6117uuid ( RO) : 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117 name-label ( RW): Packer-disk name-description ( RW): is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone current-operations (SRO): sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc sr-name-label ( RO): m3i-nfs-vm-data-ssd vbd-uuids (SRO): 7bf8749b-16a5-5b0a-3351-1271396ecd49 crashdump-uuids (SRO): virtual-size ( RO): 10737418240 physical-utilisation ( RO): 24064 location ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117 type ( RO): User sharable ( RO): false read-only ( RO): false storage-lock ( RO): false managed ( RO): true parent ( RO) [DEPRECATED]: <not in database> missing ( RO): false is-tools-iso ( RO): false other-config (MRW): content_id: 10d4bc7b-9d36-ce3b-83a0-3c4baed0c8e1; temp: temp xenstore-data (MRO): sm-config (MRO): on-boot ( RW): persist allow-caching ( RW): false metadata-latest ( RO): false metadata-of-pool ( RO): <not in database> tags (SRW): cbt-enabled ( RO): falseEdit: added debug logs
-
marcungeschikts someone in XO team should take a look. It's not overcomplicated, just a "logic behind a choice we made" and see if it's still relevant. If it is, what should be done to get Cloudinit displayed in the end.
-
I investigated this with BMeach in Discord (link). The issue is that the broken packer run is creating a template that contains a populated
disksproperty in theother-configparameter (more details here).XO's logic for hiding this is located here. I don't know the historical context, but when I initially created the terraform provider I modeled VM creation off that logic in order to get the provider's
vm.startRPC calls to succeed with cloud-init.I will investigate why packer is doing something differently the manual terraform template creation process described here, but I'm also interested in hearing more from the XO team for why it is setup this way.
-
So it's been a long time we did that, so I can't trust at 100% my memory. That's what I recall ( julien-f correct me if I'm wrong)
Default templates
For "default templates" (templates without any actual disk present, ie coming from the XCP-ng installation), the
disksfield contains a list of suggested disks to generate on VM creation. It's a guide that can be fetched by XAPI clients (XO, XenCenter) to pre-fill the disks creation fields in their respective UI. Eg: a Windows template will usually advise to create a rather large disk (40GiB) by default vs a Linux template. XenCenter and XO aren't aware of those things, they just fetch the "provision" to display what's expected in the template itself to present it to the user.During the VM creation, this
provisionfield should disappear (by XAPI, automatically, on provisioning), since it's not needed anymore: the VM is now created, and it won't be an empty template anymore. Either it stays a regular VM, or it becomes a template with a disk (see below).User created templates
Any other template that the "original/default" ones will very likely already contain a disk with a system installed (whatever this system is). Only those templates can be used with cloudinit, because you need to have a disk where cloudinit is installed.
A way to detect that a template was provisioned in the past is to rely on this information. There's one exception (again IIRC): Other install media, which is a special template without any guidance on disks to be created since it's a "misc" template.
-
I can confirm that XAPI is removing the
diskskey inother_configduring theVM.provisioncall: -
So maybe the Packer plugin isn't doing what it should to create the template? We probably need to discuss this together
-
olivierlambert thanks for that background on the Default vs user created templates and that matches my mental model of how things work with XO and terraform -- that a template must have an installed OS (disk) in order to work.
So maybe the Packer plugin isn't doing what it should to create the template?
As for what packer is doing, it's using the xapi VM.set_is_a_template call.
I believe this is identical to what XO does when you convert a VM to a template via the Advanced tab (source). That handler ultimately sets is a template, which seems equivalent to what packer is calling.
I definitely agree we need to discuss what the right solution is, but from the information I shared above I'm still unsure that we understand why a packer built template is behaving the way it is.
-
I can confirm that XAPI is removing the
diskskey inother_configduring theVM.provisioncall:I understand very little ocaml, but is the code that you linked to part of the VM creation lifecycle? It seems like it's a performance test from what I can tell.
-
Okay so after closer inspection, it's not XAPI but the client responsability to clean this key after install, as we do it in XO:
https://github.com/vatesfr/xen-orchestra/blob/e6b893977205a90f5d7b96e8e0e0f65b030fa195/packages/xo-server/src/xapi/mixins/vm.mjs#L83-L85C37// Removes disks from the provision XML, we will create them by // ourselves. await this.call('VM.remove_from_other_config', vmRef, 'disks')::ignoreErrors()However, it appears the be the standard way to do so. Let's see, as a first hint, the examples given by XAPI project on their Python bindings:
https://github.com/xapi-project/xen-api/blob/14ee043bcabf16cb58c2fbb974fd90ee81fb067e/scripts/examples/python/provision.py#L86session.xenapi.VM.remove_from_other_config(vm, "disks")XenCenter is also doing exactly this:
// Removes disks from the provision XML, we will create them by // ourselves. await this.call('VM.remove_from_other_config', vmRef, 'disks')::ignoreErrors()So the issue is the Packer plugin not having this "normal" behavior, and leading to the original problem
In other words, a template with this key is considered as "without any already installed system" and adding Cloudinit config for it doesn't make sense at all => the logic is good.Maybe it's another argument to get a Packer plugin on top of XO API to hide those implementation details…
-
olivierlambert I think I have found a fix on the Packer side like you were talking about. I have the templates working being built from the "Ubuntu Focal Fossa 20.04" default template. I created a pull request here let me know what you think, this is my first excursion with golang
-
olivierlambert ah, nice find and that explains the final missing piece. The xenserver packer plugin should definitely follow the same steps as XO and XenCenter.
Definitely agree that it would be worth considering a XO specific packer plugin, but for now that should address this bug.
-
The ddelnano/packer-plugin-xenserver release v0.5.1 has been released and includes bagas's fix for this issue.
