XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO Packer template disk issues

    Scheduled Pinned Locked Moved Xen Orchestra
    13 Posts 3 Posters 1.7k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B Offline
      BMeach
      last edited by BMeach

      Update 1

      I rebuilt my template with all of the exact same settings but instead of cloning from the "Ubuntu Focal Fossa 20.04" default template within XO, I told Packer to clone from the "Other install media" default template. Doing that now makes XO show the cloudinit settings when creating a vm from the Packer generated template. I dont know why changing what template im cloning from changes the results but I think this is still a issue that needs to be fixed. To me it seems like its a disconnect somewhere between Packer and XO. On one side or the other its applying some parameters or not applying some parameters thats needed to see the existing disk. Any ideas?

      Debug Logs

      For clarity im attaching the same debug info that I did for the other two templates. The new template that I created has a name label of "ubuntu-2004-server-test-golden-v2023.01" and a UUID of "42d967db-44f9-8ba5-37b7-2a09da8b5b7e"

      xe template-param-list uuid=<templateUUID>

      ubuntu-2004-server-test-golden-v2023.01

      xe template-param-list uuid=42d967db-44f9-8ba5-37b7-2a09da8b5b7e

      uuid ( RO)                                  : 42d967db-44f9-8ba5-37b7-2a09da8b5b7e
                                  name-label ( RW): ubuntu-2004-server-test-golden-v2023.01
                            name-description ( RW): Version: v2023.01 | Built on: 2023-01-11 00:33 UTC | Built by: HashiCorp Packer 1.8.5
                                user-version ( RW): 1
                               is-a-template ( RW): true
                         is-default-template ( RW): false
                               is-a-snapshot ( RO): false
                                 snapshot-of ( RO): <not in database>
                                   snapshots ( RO): 
                               snapshot-time ( RO): 19700101T00:00:00Z
                               snapshot-info ( RO): 
                                      parent ( RO): <not in database>
                                    children ( RO): 
                           is-control-domain ( RO): false
                                 power-state ( RO): halted
                               memory-actual ( RO): <not in database>
                               memory-target ( RO): 0
                             memory-overhead ( RO): 37748736
                           memory-static-max ( RW): 4294967296
                          memory-dynamic-max ( RW): 4294967296
                          memory-dynamic-min ( RW): 4294967296
                           memory-static-min ( RW): 4294967296
                            suspend-VDI-uuid ( RW): <not in database>
                             suspend-SR-uuid ( RW): <not in database>
                                VCPUs-params (MRW): 
                                   VCPUs-max ( RW): 2
                            VCPUs-at-startup ( RW): 2
                      actions-after-shutdown ( RW): Destroy
                        actions-after-reboot ( RW): Restart
                         actions-after-crash ( RW): Restart
                               console-uuids (SRO): 
                                         hvm ( RO): false
                                    platform (MRW): timeoffset: 1; device-model: qemu-upstream-compat; viridian: false; nx: true; pae: true; apic: true; acpi: 1
                          allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy
                          current-operations (SRO): 
                          blocked-operations (MRW): 
                         allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                         allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
                              possible-hosts ( RO): 016d3cac-76dc-438f-89e3-5fe3bda1fc4b; 38e90b77-028c-4f5a-9b62-1b9e5c252ee8
                                 domain-type ( RW): hvm
                         current-domain-type ( RO): <not in database>
                             HVM-boot-policy ( RW): BIOS order
                             HVM-boot-params (MRW): order: cd; firmware: bios
                       HVM-shadow-multiplier ( RW): 1.000
                                   PV-kernel ( RW): 
                                  PV-ramdisk ( RW): 
                                     PV-args ( RW): 
                              PV-legacy-args ( RW): 
                               PV-bootloader ( RW): 
                          PV-bootloader-args ( RW): 
                         last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 
                            last-boot-record ( RO): ''
                                 resident-on ( RO): <not in database>
                                    affinity ( RW): <not in database>
                                other-config (MRW): base_template_name: Other install media; import_task: OpaqueRef:02ba17a5-f1c4-4bdd-826b-a980c3d89869; mac_seed: dea90f6d-ee36-57a5-60fd-bef7554ce70b; install-methods: cdrom
                                      dom-id ( RO): -1
                             recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="yes"/><restriction field="supports-secure-boot" value="yes"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                               xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: 
                  ha-always-run ( RW) [DEPRECATED]: false
                         ha-restart-priority ( RW): 
                                       blobs ( RO): 
                                  start-time ( RO): <unknown time>
                                install-time ( RO): <unknown time>
                                VCPUs-number ( RO): <not in database>
                           VCPUs-utilisation (MRO): 
                                  os-version (MRO): name: Ubuntu 20.04.5 LTS; uname: 5.4.0-136-generic; distro: ubuntu; major: 20; minor: 04
                          PV-drivers-version (MRO): major: 6; minor: 6; micro: 80; build: 0
          PV-drivers-up-to-date ( RO) [DEPRECATED]: false
                                      memory (MRO): 
                                       disks (MRO): 
                                        VBDs (SRO): 7bf8749b-16a5-5b0a-3351-1271396ecd49
                                    networks (MRO): 0/ip: 172.18.20.218; 0/ipv4/0: 172.18.20.218; 0/ipv6/0: fe80::90f2:61ff:fe17:8221
                         PV-drivers-detected ( RO): false
                                       other (MRO): shutdown: ; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-vcpu-hotplug: 1; feature-suspend: 1; feature-reboot: 1; feature-poweroff: 1; feature-balloon: 1
                                        live ( RO): true
                  guest-metrics-last-updated ( RO): 20230111T00:44:17Z
                         can-use-hotplug-vbd ( RO): unspecified
                         can-use-hotplug-vif ( RO): unspecified
                    cooperative ( RO) [DEPRECATED]: true
                                        tags (SRW): 
                                   appliance ( RW): <not in database>
                           snapshot-schedule ( RW): <not in database>
                            is-vmss-snapshot ( RO): false
                                 start-delay ( RW): 0
                              shutdown-delay ( RW): 0
                                       order ( RW): 0
                                     version ( RO): 0
                               generation-id ( RO): 
                   hardware-platform-version ( RO): 0
                           has-vendor-device ( RW): false
                             requires-reboot ( RO): false
                             reference-label ( RO): other-install-media
                                bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d
      

      xe vbd-list | grep <templateUUID> -B1 -A2

      ubuntu-2004-server-test-golden-v2023.01

      xe vbd-list | grep 42d967db-44f9-8ba5-37b7-2a09da8b5b7e -B1 -A2

      uuid ( RO)             : 7bf8749b-16a5-5b0a-3351-1271396ecd49
                vm-uuid ( RO): 42d967db-44f9-8ba5-37b7-2a09da8b5b7e
          vm-name-label ( RO): ubuntu-2004-server-test-golden-v2023.01
               vdi-uuid ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117
      

      xe vbd-param-list uuid=<template-VBD-UUID>

      ubuntu-2004-server-test-golden-v2023.01

      xe vbd-param-list uuid=7bf8749b-16a5-5b0a-3351-1271396ecd49

      uuid ( RO)                        : 7bf8749b-16a5-5b0a-3351-1271396ecd49
                           vm-uuid ( RO): 42d967db-44f9-8ba5-37b7-2a09da8b5b7e
                     vm-name-label ( RO): ubuntu-2004-server-test-golden-v2023.01
                          vdi-uuid ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117
                    vdi-name-label ( RO): Packer-disk
                allowed-operations (SRO): attach
                current-operations (SRO): 
                             empty ( RO): false
                            device ( RO): xvda
                        userdevice ( RW): 0
                          bootable ( RW): false
                              mode ( RW): RW
                              type ( RW): Disk
                       unpluggable ( RW): false
                currently-attached ( RO): false
                        attachable ( RO): true
                      storage-lock ( RO): false
                       status-code ( RO): 0
                     status-detail ( RO): 
                qos_algorithm_type ( RW): 
              qos_algorithm_params (MRW): 
          qos_supported_algorithms (SRO): 
                      other-config (MRW): 
                       io_read_kbs ( RO): <unknown>
                      io_write_kbs ( RO): <unknown>
      

      xe vdi-param-list uuid=<template-VDI-UUID>

      ubuntu-2004-server-test-golden-v2023.01

      xe vdi-param-list uuid=5c9214b3-231f-4b5c-ab96-7c1bdbbb6117

      uuid ( RO)                    : 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117
                    name-label ( RW): Packer-disk
              name-description ( RW): 
                 is-a-snapshot ( RO): false
                   snapshot-of ( RO): <not in database>
                     snapshots ( RO): 
                 snapshot-time ( RO): 19700101T00:00:00Z
            allowed-operations (SRO): generate_config; update; forget; destroy; snapshot; resize; copy; clone
            current-operations (SRO): 
                       sr-uuid ( RO): 4d3961af-db6d-480c-e8bf-086f2c8c9dfc
                 sr-name-label ( RO): m3i-nfs-vm-data-ssd
                     vbd-uuids (SRO): 7bf8749b-16a5-5b0a-3351-1271396ecd49
               crashdump-uuids (SRO): 
                  virtual-size ( RO): 10737418240
          physical-utilisation ( RO): 24064
                      location ( RO): 5c9214b3-231f-4b5c-ab96-7c1bdbbb6117
                          type ( RO): User
                      sharable ( RO): false
                     read-only ( RO): false
                  storage-lock ( RO): false
                       managed ( RO): true
           parent ( RO) [DEPRECATED]: <not in database>
                       missing ( RO): false
                  is-tools-iso ( RO): false
                  other-config (MRW): content_id: 10d4bc7b-9d36-ce3b-83a0-3c4baed0c8e1; temp: temp
                 xenstore-data (MRO): 
                     sm-config (MRO): 
                       on-boot ( RW): persist
                 allow-caching ( RW): false
               metadata-latest ( RO): false
              metadata-of-pool ( RO): <not in database>
                          tags (SRW): 
                   cbt-enabled ( RO): false
      

      Edit: added debug logs

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates πŸͺ Co-Founder CEO
        last edited by

        @marcungeschikts someone in XO team should take a look. It's not overcomplicated, just a "logic behind a choice we made" and see if it's still relevant. If it is, what should be done to get Cloudinit displayed in the end.

        D 1 Reply Last reply Reply Quote 0
        • D Offline
          ddelnano Terraform Team @olivierlambert
          last edited by

          I investigated this with @BMeach in Discord (link). The issue is that the broken packer run is creating a template that contains a populated disks property in the other-config parameter (more details here).

          XO's logic for hiding this is located here. I don't know the historical context, but when I initially created the terraform provider I modeled VM creation off that logic in order to get the provider's vm.start RPC calls to succeed with cloud-init.

          I will investigate why packer is doing something differently the manual terraform template creation process described here, but I'm also interested in hearing more from the XO team for why it is setup this way.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates πŸͺ Co-Founder CEO
            last edited by olivierlambert

            So it's been a long time we did that, so I can't trust at 100% my memory. That's what I recall ( @julien-f correct me if I'm wrong)

            Default templates

            For "default templates" (templates without any actual disk present, ie coming from the XCP-ng installation), the disks field contains a list of suggested disks to generate on VM creation. It's a guide that can be fetched by XAPI clients (XO, XenCenter) to pre-fill the disks creation fields in their respective UI. Eg: a Windows template will usually advise to create a rather large disk (40GiB) by default vs a Linux template. XenCenter and XO aren't aware of those things, they just fetch the "provision" to display what's expected in the template itself to present it to the user.

            During the VM creation, this provision field should disappear (by XAPI, automatically, on provisioning), since it's not needed anymore: the VM is now created, and it won't be an empty template anymore. Either it stays a regular VM, or it becomes a template with a disk (see below).

            User created templates

            Any other template that the "original/default" ones will very likely already contain a disk with a system installed (whatever this system is). Only those templates can be used with cloudinit, because you need to have a disk where cloudinit is installed.

            A way to detect that a template was provisioned in the past is to rely on this information. There's one exception (again IIRC): Other install media, which is a special template without any guidance on disks to be created since it's a "misc" template.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates πŸͺ Co-Founder CEO
              last edited by

              I can confirm that XAPI is removing the disks key in other_config during the VM.provision call:

              https://github.com/xapi-project/xen-api/blob/14ee043bcabf16cb58c2fbb974fd90ee81fb067e/ocaml/perftest/createVM.ml#L129

              D 1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates πŸͺ Co-Founder CEO
                last edited by

                So maybe the Packer plugin isn't doing what it should to create the template? We probably need to discuss this together πŸ™‚

                D 1 Reply Last reply Reply Quote 0
                • D Offline
                  ddelnano Terraform Team @olivierlambert
                  last edited by

                  @olivierlambert thanks for that background on the Default vs user created templates and that matches my mental model of how things work with XO and terraform -- that a template must have an installed OS (disk) in order to work.

                  So maybe the Packer plugin isn't doing what it should to create the template?

                  As for what packer is doing, it's using the xapi VM.set_is_a_template call.

                  I believe this is identical to what XO does when you convert a VM to a template via the Advanced tab (source). That handler ultimately sets is a template, which seems equivalent to what packer is calling.

                  I definitely agree we need to discuss what the right solution is, but from the information I shared above I'm still unsure that we understand why a packer built template is behaving the way it is.

                  1 Reply Last reply Reply Quote 0
                  • D Offline
                    ddelnano Terraform Team @olivierlambert
                    last edited by

                    I can confirm that XAPI is removing the disks key in other_config during the VM.provision call:

                    https://github.com/xapi-project/xen-api/blob/14ee043bcabf16cb58c2fbb974fd90ee81fb067e/ocaml/perftest/createVM.ml#L129

                    I understand very little ocaml, but is the code that you linked to part of the VM creation lifecycle? It seems like it's a performance test from what I can tell.

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates πŸͺ Co-Founder CEO
                      last edited by olivierlambert

                      Okay so after closer inspection, it's not XAPI but the client responsability to clean this key after install, as we do it in XO:
                      https://github.com/vatesfr/xen-orchestra/blob/e6b893977205a90f5d7b96e8e0e0f65b030fa195/packages/xo-server/src/xapi/mixins/vm.mjs#L83-L85C37

                          // Removes disks from the provision XML, we will create them by
                          // ourselves.
                          await this.call('VM.remove_from_other_config', vmRef, 'disks')::ignoreErrors()
                      

                      However, it appears the be the standard way to do so. Let's see, as a first hint, the examples given by XAPI project on their Python bindings:
                      https://github.com/xapi-project/xen-api/blob/14ee043bcabf16cb58c2fbb974fd90ee81fb067e/scripts/examples/python/provision.py#L86

                      session.xenapi.VM.remove_from_other_config(vm, "disks")
                      

                      XenCenter is also doing exactly this:

                      https://github.com/vatesfr/xen-orchestra/blob/e6b893977205a90f5d7b96e8e0e0f65b030fa195/packages/xo-server/src/xapi/mixins/vm.mjs#L83-L85C37

                          // Removes disks from the provision XML, we will create them by
                          // ourselves.
                          await this.call('VM.remove_from_other_config', vmRef, 'disks')::ignoreErrors()
                      

                      So the issue is the Packer plugin not having this "normal" behavior, and leading to the original problem πŸ™‚ In other words, a template with this key is considered as "without any already installed system" and adding Cloudinit config for it doesn't make sense at all => the logic is good.

                      Maybe it's another argument to get a Packer plugin on top of XO API to hide those implementation details…

                      B D 2 Replies Last reply Reply Quote 0
                      • B Offline
                        BMeach @olivierlambert
                        last edited by

                        @olivierlambert I think I have found a fix on the Packer side like you were talking about. I have the templates working being built from the "Ubuntu Focal Fossa 20.04" default template. I created a pull request here let me know what you think, this is my first excursion with golang πŸ™‚

                        1 Reply Last reply Reply Quote 1
                        • D Offline
                          ddelnano Terraform Team @olivierlambert
                          last edited by

                          @olivierlambert ah, nice find and that explains the final missing piece. The xenserver packer plugin should definitely follow the same steps as XO and XenCenter.

                          Definitely agree that it would be worth considering a XO specific packer plugin, but for now that should address this bug.

                          D 1 Reply Last reply Reply Quote 0
                          • D Offline
                            ddelnano Terraform Team @ddelnano
                            last edited by

                            The ddelnano/packer-plugin-xenserver release v0.5.1 has been released and includes @bagas's fix for this issue.

                            1 Reply Last reply Reply Quote 1
                            • First post
                              Last post