To provide some background I was talking with @olivierlambert over on ddelnano's Discord about an issue I was having with the cloudinit options not showing up in XO on any templates I made with the XenServer packer.io builder made by ddelnano.
xcp-ng / xen orchestra setup
- XCP-NG Hosts
- 3x Dell r630's
- XCP-NG Version: XCP-ng 8.2.1
- Build Number: release/yangtze/master/58
- Xen Orchestra
- Using this install script
- commit f95a2
- xo-server 5.107.4
- xo-web 5.108.0
- 4 vcpu's, 8G RAM, 2 VIFS (one to mgmt net, the other to the storage net), 25G disk
What im trying to do
Im trying to have packer run every month via a CI/CD pipeline to generate a golden vm template every month. My goal for the template is to have it "cloudinit ready" so I can use it in my Terraform configs to provision workload vm's. I also want Packer to use Ansible to install basic system monitoring utilities such as netdata and any other necessary packages for example the xe-guest-utilities.
Where I am right now
Right now I have got Packer working up to a degree, I can create the template completely fine. I have gotten Packer to run my ansible playbooks on the host and have also used the file provisioner and shell provisioner to test installing packages and moving files onto the template vm. Where I am stuck though is getting the cloudinit options to show up inside of XO. I have validated multiple times that my vm template has a disk attached to it and if I create a vm from the Packer generated template I get the options shown below
As you can see the disk from the template is there for sure under the name "Packer-disk" but I still get prompted to insert a iso/dvd instead of prompting me for a ssh key or cloudinit config which by my understanding it should show those feilds if there is a disk in the template which there is but thats obviously not working. If I continue and create the vm with the options as shown below it boots into a fully installed ubuntu 20.04 os and is configured just like it should be with the ansible playbooks, shell provisioner, and file provisioner taking affect and working perfectly. The issue is there is no cloudinit disk which leaves the cloudinit config in the os "broken" due to there being nothing for cloudinit's nocloud or configdrive datasource to read.
Troubleshooting
The first thing that I did to troubleshoot is I downloaded the Ubuntu 20.04 cloudimg ova file and imported it into XO through the VM import utility in the web console. I then immediately turned it into a template and tried to create a vm from that and that showed all of the cloudinit options as expected.
The template has a name label of "ubuntu-focal-20.04-cloudimg" and the template UUID is "d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8". I then wanted to compare the template parameters so I went and looked for my Packer generated template, it has a name label of "ubuntu-2004-server-golden-v2023.01" and a UUID of "422081e2-6b73-f642-fc82-6cba4313a9da". From there I then went out and got the following data that has been posted in the Debug Logs Section, a list of the template parameters, a list of vbd's attached to each template, a list of the vbd's parameters, and a list of the vdi parameters.
Ive only been using the XCP-NG and XO platform for about 5-6 months so I am sorry if there is somewhat obvious diagnostic data missing, let me know and im happy to gather any information needed to resolve this issue. From my limited knowledge this seems like its somehow related to XO not seeing the attached disk on the template or the Packer plugin not setting some parameters that are required for XO to see the attached disk when converting to a template. If anyone has any suggestions or any ways to help me out it would be very appreciated.
Debug Logs
xe template-param-list uuid=<templateUUID>
ubuntu-focal-20.04-cloudimg
xe template-param-list uuid=d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8
uuid ( RO) : d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8
name-label ( RW): ubuntu-focal-20.04-cloudimg
name-description ( RW): Ubuntu Linux (64-bit)
user-version ( RW): 1
is-a-template ( RW): true
is-default-template ( RW): false
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
snapshot-info ( RO):
parent ( RO): <not in database>
children ( RO):
is-control-domain ( RO): false
power-state ( RO): halted
memory-actual ( RO): <not in database>
memory-target ( RO): 0
memory-overhead ( RO): 12582912
memory-static-max ( RW): 1073741824
memory-dynamic-max ( RW): 1073741824
memory-dynamic-min ( RW): 1073741824
memory-static-min ( RW): 1073741824
suspend-VDI-uuid ( RW): <not in database>
suspend-SR-uuid ( RW): <not in database>
VCPUs-params (MRW):
VCPUs-max ( RW): 2
VCPUs-at-startup ( RW): 2
actions-after-shutdown ( RW): Destroy
actions-after-reboot ( RW): Restart
actions-after-crash ( RW): Restart
console-uuids (SRO):
hvm ( RO): false
platform (MRW): timeoffset: 0; nx: true; acpi: 1; apic: true; pae: true; hpet: true; viridian: true
allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy
current-operations (SRO):
blocked-operations (MRW):
allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8
domain-type ( RW): hvm
current-domain-type ( RO): <not in database>
HVM-boot-policy ( RW): BIOS order
HVM-boot-params (MRW): order: cdn
HVM-shadow-multiplier ( RW): 1.000
PV-kernel ( RW):
PV-ramdisk ( RW):
PV-args ( RW):
PV-legacy-args ( RW):
PV-bootloader ( RW):
PV-bootloader-args ( RW):
last-boot-CPU-flags ( RO):
last-boot-record ( RO): ''
resident-on ( RO): <not in database>
affinity ( RW): <not in database>
other-config (MRW): mac_seed: 7fa98bf1-ff15-c50a-0ea4-92d46cee2e3a; vgpu_pci: ; base_template_name: Other install media; install-methods: cdrom
dom-id ( RO): -1
recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472" /><restriction field="vcpus-max" max="32" /><restriction property="number-of-vbds" max="255" /><restriction property="number-of-vifs" max="7" /><restriction field="has-vendor-device" value="false" /></restrictions>
xenstore-data (MRW):
ha-always-run ( RW) [DEPRECATED]: false
ha-restart-priority ( RW):
blobs ( RO):
start-time ( RO): <unknown time>
install-time ( RO): <unknown time>
VCPUs-number ( RO): <not in database>
VCPUs-utilisation (MRO):
os-version (MRO): <not in database>
PV-drivers-version (MRO): <not in database>
PV-drivers-up-to-date ( RO) [DEPRECATED]: <not in database>
memory (MRO): <not in database>
disks (MRO): <not in database>
VBDs (SRO): 6d154703-7f69-d72d-cbab-b62be607b65b
networks (MRO): <not in database>
PV-drivers-detected ( RO): <not in database>
other (MRO): <not in database>
live ( RO): <not in database>
guest-metrics-last-updated ( RO): <not in database>
can-use-hotplug-vbd ( RO): <not in database>
can-use-hotplug-vif ( RO): <not in database>
cooperative ( RO) [DEPRECATED]: true
tags (SRW):
appliance ( RW): <not in database>
snapshot-schedule ( RW): <not in database>
is-vmss-snapshot ( RO): false
start-delay ( RW): 0
shutdown-delay ( RW): 0
order ( RW): 0
version ( RO): 0
generation-id ( RO): 0:0
hardware-platform-version ( RO): 0
has-vendor-device ( RW): false
requires-reboot ( RO): false
reference-label ( RO):
bios-strings (MRO):
ubuntu-2004-server-golden-v2023.01
xe template-param-list uuid=422081e2-6b73-f642-fc82-6cba4313a9da
uuid ( RO) : 422081e2-6b73-f642-fc82-6cba4313a9da
name-label ( RW): ubuntu-2004-server-golden-v2023.01
name-description ( RW): Version: v2023.01 | Built on: 2023-01-10 22:00 UTC | Built by: HashiCorp Packer 1.8.5
user-version ( RW): 1
is-a-template ( RW): true
is-default-template ( RW): false
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
snapshot-info ( RO):
parent ( RO): <not in database>
children ( RO):
is-control-domain ( RO): false
power-state ( RO): halted
memory-actual ( RO): <not in database>
memory-target ( RO): 0
memory-overhead ( RO): 37748736
memory-static-max ( RW): 4294967296
memory-dynamic-max ( RW): 4294967296
memory-dynamic-min ( RW): 4294967296
memory-static-min ( RW): 4294967296
suspend-VDI-uuid ( RW): <not in database>
suspend-SR-uuid ( RW): <not in database>
VCPUs-params (MRW):
VCPUs-max ( RW): 2
VCPUs-at-startup ( RW): 2
actions-after-shutdown ( RW): Destroy
actions-after-reboot ( RW): Restart
actions-after-crash ( RW): Restart
console-uuids (SRO):
hvm ( RO): false
platform (MRW): timeoffset: 1; device-model: qemu-upstream-compat; viridian: false; nx: true; pae: true; apic: true; acpi: 1
allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy
current-operations (SRO):
blocked-operations (MRW):
allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8
domain-type ( RW): hvm
current-domain-type ( RO): <not in database>
HVM-boot-policy ( RW): BIOS order
HVM-boot-params (MRW): order: cd; firmware: bios
HVM-shadow-multiplier ( RW): 1.000
PV-kernel ( RW):
PV-ramdisk ( RW):
PV-args ( RW):
PV-legacy-args ( RW):
PV-bootloader ( RW):
PV-bootloader-args ( RW):
last-boot-CPU-flags ( RO): vendor: GenuineIntel; features:
last-boot-record ( RO): ''
resident-on ( RO): <not in database>
affinity ( RW): <not in database>
other-config (MRW): base_template_name: Ubuntu Focal Fossa 20.04; import_task: OpaqueRef:1373f40a-13bb-4749-b314-546a6513ce1e; mac_seed: d16763c1-691e-ac82-01fe-056877eb0b75; disks: <provision><disk bootable="true" device="0" size="10737418240" sr="" type="system"/></provision>; install-methods: cdrom,nfs,http,ftp; linux_template: true dom-id ( RO): -1
recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="allow-gpu-passthrough" value="1"/><restriction field="allow-vgpu" value="1"/><restriction field="allow-network-sriov" value="1"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="no"/><restriction field="supports-secure-boot" value="no"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data:
ha-always-run ( RW) [DEPRECATED]: false
ha-restart-priority ( RW):
blobs ( RO):
start-time ( RO): <unknown time>
install-time ( RO): <unknown time>
VCPUs-number ( RO): <not in database>
VCPUs-utilisation (MRO):
os-version (MRO): name: Ubuntu 20.04.5 LTS; uname: 5.4.0-136-generic; distro: ubuntu; major: 20; minor: 04
PV-drivers-version (MRO): major: 6; minor: 6; micro: 80; build: 0
PV-drivers-up-to-date ( RO) [DEPRECATED]: false
memory (MRO):
disks (MRO):
VBDs (SRO): 455a4d18-37d7-0083-3123-868ee0ec832f
networks (MRO): 0/ip: 172.18.20.217; 0/ipv4/0: 172.18.20.217; 0/ipv6/0: fe80::8cc3:15ff:fe81:ce1b
PV-drivers-detected ( RO): false
other (MRO): shutdown: ; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-vcpu-hotplug: 1; feature-suspend: 1; feature-reboot: 1; feature-poweroff: 1; feature-balloon: 1
live ( RO): true
guest-metrics-last-updated ( RO): 20230110T22:11:48Z
can-use-hotplug-vbd ( RO): unspecified
can-use-hotplug-vif ( RO): unspecified
cooperative ( RO) [DEPRECATED]: true
tags (SRW):
appliance ( RW): <not in database>
snapshot-schedule ( RW): <not in database>
is-vmss-snapshot ( RO): false
start-delay ( RW): 0
shutdown-delay ( RW): 0
order ( RW): 0
version ( RO): 0
generation-id ( RO):
hardware-platform-version ( RO): 0
has-vendor-device ( RW): false
requires-reboot ( RO): false
reference-label ( RO): ubuntu-20.04
bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d
xe vbd-list | grep <templateUUID> -B1 -A2
ubuntu-focal-20.04-cloudimg
xe vbd-list | grep d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8 -B1 -A2
uuid ( RO) : 6d154703-7f69-d72d-cbab-b62be607b65b
vm-uuid ( RO): d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8
vm-name-label ( RO): ubuntu-focal-20.04-cloudimg
vdi-uuid ( RO): db3b06bf-d292-4a00-b916-c496b9de8aa5
ubuntu-2004-server-golden-v2023.01
xe vbd-list | grep 422081e2-6b73-f642-fc82-6cba4313a9da -B1 -A2
uuid ( RO) : 455a4d18-37d7-0083-3123-868ee0ec832f
vm-uuid ( RO): 422081e2-6b73-f642-fc82-6cba4313a9da
vm-name-label ( RO): ubuntu-2004-server-golden-v2023.01
vdi-uuid ( RO): 9c1403bb-fde9-415f-9ae1-9511b84a60bb
xe vbd-param-list uuid=<template-VBD-UUID>
ubuntu-focal-20.04-cloudimg
xe vbd-param-list uuid=6d154703-7f69-d72d-cbab-b62be607b65b
uuid ( RO) : 6d154703-7f69-d72d-cbab-b62be607b65b
vm-uuid ( RO): d1d97fd9-4d01-39cb-f5b8-e34c3a97ccc8
vm-name-label ( RO): ubuntu-focal-20.04-cloudimg
vdi-uuid ( RO): db3b06bf-d292-4a00-b916-c496b9de8aa5
vdi-name-label ( RO): Hard Disk 1
allowed-operations (SRO): attach
current-operations (SRO):
empty ( RO): false
device ( RO):
userdevice ( RW): 0
bootable ( RW): false
mode ( RW): RW
type ( RW): Disk
unpluggable ( RW): false
currently-attached ( RO): false
attachable ( RO): true
storage-lock ( RO): false
status-code ( RO): 0
status-detail ( RO):
qos_algorithm_type ( RW):
qos_algorithm_params (MRW):
qos_supported_algorithms (SRO):
other-config (MRW):
io_read_kbs ( RO): <unknown>
io_write_kbs ( RO): <unknown>
ubuntu-2004-server-golden-v2023.01
xe vbd-param-list uuid=455a4d18-37d7-0083-3123-868ee0ec832f
uuid ( RO) : 455a4d18-37d7-0083-3123-868ee0ec832f
vm-uuid ( RO): 422081e2-6b73-f642-fc82-6cba4313a9da
vm-name-label ( RO): ubuntu-2004-server-golden-v2023.01
vdi-uuid ( RO): 9c1403bb-fde9-415f-9ae1-9511b84a60bb
vdi-name-label ( RO): Packer-disk
allowed-operations (SRO): attach
current-operations (SRO):
empty ( RO): false
device ( RO): xvda
userdevice ( RW): 0
bootable ( RW): false
mode ( RW): RW
type ( RW): Disk
unpluggable ( RW): false
currently-attached ( RO): false
attachable ( RO): true
storage-lock ( RO): false
status-code ( RO): 0
status-detail ( RO):
qos_algorithm_type ( RW):
qos_algorithm_params (MRW):
qos_supported_algorithms (SRO):
other-config (MRW):
io_read_kbs ( RO): <unknown>
io_write_kbs ( RO): <unknown>
xe vdi-param-list uuid=<template-VDI-UUID>
ubuntu-focal-20.04-cloudimg
xe vdi-param-list uuid=db3b06bf-d292-4a00-b916-c496b9de8aa5
uuid ( RO) : db3b06bf-d292-4a00-b916-c496b9de8aa5
name-label ( RW): Hard Disk 1
name-description ( RW): No description
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone
current-operations (SRO):
sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc
sr-name-label ( RO): m3i-nfs-vm-data-ssd
vbd-uuids (SRO): 6d154703-7f69-d72d-cbab-b62be607b65b
crashdump-uuids (SRO):
virtual-size ( RO): 10737418240
physical-utilisation ( RO): 24064
location ( RO): db3b06bf-d292-4a00-b916-c496b9de8aa5
type ( RO): User
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW): content_id: 3429ce87-38f3-c3ac-ac9a-ed373355c24f
xenstore-data (MRO):
sm-config (MRO): vhd-parent: e949503d-018c-4630-9dc8-56354fe93340
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
ubuntu-2004-server-golden-v2023.01
xe vdi-param-list uuid=9c1403bb-fde9-415f-9ae1-9511b84a60bb
uuid ( RO) : 9c1403bb-fde9-415f-9ae1-9511b84a60bb
name-label ( RW): Packer-disk
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone
current-operations (SRO):
sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc
sr-name-label ( RO): m3i-nfs-vm-data-ssd
vbd-uuids (SRO): 455a4d18-37d7-0083-3123-868ee0ec832f
crashdump-uuids (SRO):
virtual-size ( RO): 10737418240
physical-utilisation ( RO): 24064
location ( RO): 9c1403bb-fde9-415f-9ae1-9511b84a60bb
type ( RO): User
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW): content_id: 3958b983-1b38-ebb2-55d3-e76a137524ca; temp: temp
xenstore-data (MRO):
sm-config (MRO):
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
Edit: formatting
