XCP-ng

mietek

@tjkreidl This is surprising then that VM spawned from the template would even be able to change NIC order. This make no sense.
On the other hand how does the template know what is the list on NICs attached to the VM created from it and what should be the order of them then?

mietek

@tjkreidl Thanks for the article. Unfortunately this machine is full of running VMs and changing NICs order on the host might have a global impact on all of them so it seems like an overkill.
I have created this template on this machine and created VM based on that template right away and I experience this issue. It looks like there is a problem with the template creation or template based VM creation process.
Is there a way to change it NIC order in the template itself?

mietek

@olivierlambert
Thanks for the info and get well soon!

mietek

@olivierlambert - any ideas as I ran out of options.

mietek

@olivierlambert - any ideas?

Thanks

mietek

@olivierlambert
I have recreated the base VM [debian11-template], convert it to template and generated net VM [zzz-nettest2] based on that template.
These are the differences in VIF config:

original VM:

uuid ( RO)                  : 84c4bd3e-4f50-48dc-4050-19dd4e8a8a8a
         vm-name-label ( RO): Debian11-template
                **device ( RO): 0**
                   MAC ( RO): 9e:82:ec:4e:53:4f
          network-uuid ( RO): 1f7bec14-2ff5-471d-ba9c-bd84882ede17
    **network-name-label ( RO): Net1**


uuid ( RO)                  : 1d0aca18-e4f2-8b08-9ae6-022846241ab1
         vm-name-label ( RO): Debian11-template
                **device ( RO): 1**
                   MAC ( RO): 02:76:1f:22:c0:be
          network-uuid ( RO): 79fd2cb6-762a-e433-129a-edbb32612735
    **network-name-label ( RO): Net2**

new VM

uuid ( RO)                  : f4e200d6-2ba4-d2a4-961e-a0431951d25a
         vm-name-label ( RO): zzz-nettest2
                **device ( RO): 1**
                   MAC ( RO): 6e:18:1e:93:38:e9
          network-uuid ( RO): 1f7bec14-2ff5-471d-ba9c-bd84882ede17
    **network-name-label ( RO): Net1**


uuid ( RO)                  : 16fbbd49-bb80-d111-dc83-5c47447ac174
         vm-name-label ( RO): zzz-nettest2
                **device ( RO): 0**
                   MAC ( RO): a2:fe:cd:37:58:9f
          network-uuid ( RO): 79fd2cb6-762a-e433-129a-edbb32612735
    **network-name-label ( RO): Net2**

It happens no matter if I convert the template from the VM or create it from snapshot.

Template details below:

uuid ( RO)                                  : f5b33b1c-798c-7f01-3ad6-9f5a44d1a25e
                            name-label ( RW): Debian11-template
                      name-description ( RW): [<env>/<DC>] <application/infra>: <app name/infra role>
                          user-version ( RW): 1
                         is-a-template ( RW): true
                   is-default-template ( RW): false
                         is-a-snapshot ( RO): false
                           snapshot-of ( RO): <not in database>
                             snapshots ( RO): 
                         snapshot-time ( RO): 19700101T00:00:00Z
                         snapshot-info ( RO): 
                                parent ( RO): <not in database>
                              children ( RO): 
                     is-control-domain ( RO): false
                           power-state ( RO): halted
                         memory-actual ( RO): <not in database>
                         memory-target ( RO): 0
                       memory-overhead ( RO): 19922944
                     memory-static-max ( RW): 2147483648
                    memory-dynamic-max ( RW): 2147483648
                    memory-dynamic-min ( RW): 2147483648
                     memory-static-min ( RW): 268435456
                      suspend-VDI-uuid ( RW): <not in database>
                       suspend-SR-uuid ( RW): <not in database>
                          VCPUs-params (MRW): 
                             VCPUs-max ( RW): 1
                      VCPUs-at-startup ( RW): 1
                actions-after-shutdown ( RW): Destroy
                  actions-after-reboot ( RW): Restart
                   actions-after-crash ( RW): Restart
                         console-uuids (SRO): 
                                   hvm ( RO): false
                              platform (MRW): timeoffset: 1; secureboot: false; device-model: qemu-upstream-compat; videoram: 8; hpet: true; apic: true; device_id: 0001; vga: std; nx: true; pae: true; viridian: false; acpi: 1
                    allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy
                    current-operations (SRO): 
                    blocked-operations (MRW): 
                   allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                   allowed-VIF-devices (SRO): 2; 3; 4; 5; 6
                        possible-hosts ( RO): 9a33c146-c64d-4933-a54a-18941cb825f0
                           domain-type ( RW): hvm
                   current-domain-type ( RO): <not in database>
                       HVM-boot-policy ( RW): BIOS order
                       HVM-boot-params (MRW): firmware: bios; order: cdn
                 HVM-shadow-multiplier ( RW): 1.000
                             PV-kernel ( RW): 
                            PV-ramdisk ( RW): 
                               PV-args ( RW): 
                        PV-legacy-args ( RW): 
                         PV-bootloader ( RW): 
                    PV-bootloader-args ( RW): 
                   last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-f7fa3223-2d93fbff-00000523-0000000f-019c47ab-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                      last-boot-record ( RO): ''
                           resident-on ( RO): <not in database>
                              affinity ( RW): <not in database>
                          other-config (MRW): import_task: OpaqueRef:eccf2505-7aaf-4592-b9b0-9887cfa77b33; mac_seed: 1c02ad94-8a62-f861-eabd-0baf9b7c32d4; base_template_name: Debian Stretch 9.0; install-methods: cdrom,nfs,http,ftp; linux_template: true
                                dom-id ( RO): -1
                       recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="allow-gpu-passthrough" value="1"/><restriction field="allow-vgpu" value="1"/><restriction field="allow-network-sriov" value="1"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="no"/><restriction field="supports-secure-boot" value="no"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                         xenstore-data (MRW): vm-data: ; vm-data/mmio-hole-size: 268435456
            ha-always-run ( RW) [DEPRECATED]: false
                   ha-restart-priority ( RW): 
                                 blobs ( RO): 
                            start-time ( RO): <unknown time>
                          install-time ( RO): <unknown time>
                          VCPUs-number ( RO): <not in database>
                     VCPUs-utilisation (MRO): 
                            os-version (MRO): 
                    PV-drivers-version (MRO): 
    PV-drivers-up-to-date ( RO) [DEPRECATED]: false
                                memory (MRO): 
                                 disks (MRO): 
                                  VBDs (SRO): fd0f2074-c6e8-b505-b5dc-95ac02fc4e42; 0bc0c5cd-afb5-54f3-8a48-815539029c4c
                              networks (MRO): 
                   PV-drivers-detected ( RO): false
                                 other (MRO): platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0
                                  live ( RO): true
            guest-metrics-last-updated ( RO): 20220711T14:23:59Z
                   can-use-hotplug-vbd ( RO): unspecified
                   can-use-hotplug-vif ( RO): unspecified
              cooperative ( RO) [DEPRECATED]: true
                                  tags (SRW): 
                             appliance ( RW): <not in database>
                     snapshot-schedule ( RW): <not in database>
                      is-vmss-snapshot ( RO): false
                           start-delay ( RW): 0
                        shutdown-delay ( RW): 0
                                 order ( RW): 0
                               version ( RO): 0
                         generation-id ( RO): 
             hardware-platform-version ( RO): 0
                     has-vendor-device ( RW): false
                       requires-reboot ( RO): false
                       reference-label ( RO): debian-9
                          bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/d43fa6d3fe8aaef04c2bad9d3e3daaef04c2bad9d3

mietek

@olivierlambert I am surprised as well, that is why I have raised this question.
Template is saved with the networks order Net1 Net2.
VM spawned from it is having networks order Net2, Net1.

Is there a way to check which order the template was created with?
Maybe the networks were renamed in the mean time or something?
I am shooting in the dark here ...

mietek

@olivierlambert I can see template having allowed-VIF-devices list:
allowed-VIF-devices (SRO): 2; 3; 4; 5; 6

but I do not see any list of preferred order of them.
Is there a way to force template to to use it in specific order?
And why would VM created based on the template have that order reversed?

mietek

@olivierlambert Anything else I may provide to help to fix this issue?
Thanks

mietek

I do not see any VIF order here.
Is there a way to force it on the template config level?

mietek

@olivierlambert

uuid ( RO)                                  : 6f7611e7-00aa-07ac-66cc-4bf709576af7
                            name-label ( RW): Debian11-template
                      name-description ( RW): [<env>/<DC>] <application/infra>: <app name/infra role>
                          user-version ( RW): 1
                         is-a-template ( RW): true
                   is-default-template ( RW): false
                         is-a-snapshot ( RO): false
                           snapshot-of ( RO): <not in database>
                             snapshots ( RO): 
                         snapshot-time ( RO): 19700101T00:00:00Z
                         snapshot-info ( RO): 
                                parent ( RO): <not in database>
                              children ( RO): 
                     is-control-domain ( RO): false
                           power-state ( RO): halted
                         memory-actual ( RO): <not in database>
                         memory-target ( RO): 0
                       memory-overhead ( RO): 19922944
                     memory-static-max ( RW): 2147483648
                    memory-dynamic-max ( RW): 2147483648
                    memory-dynamic-min ( RW): 2147483648
                     memory-static-min ( RW): 268435456
                      suspend-VDI-uuid ( RW): <not in database>
                       suspend-SR-uuid ( RW): <not in database>
                          VCPUs-params (MRW): 
                             VCPUs-max ( RW): 1
                      VCPUs-at-startup ( RW): 1
                actions-after-shutdown ( RW): Destroy
                  actions-after-reboot ( RW): Restart
                   actions-after-crash ( RW): Restart
                         console-uuids (SRO): 
                                   hvm ( RO): false
                              platform (MRW): timeoffset: 1; secureboot: false; device-model: qemu-upstream-compat; videoram: 8; hpet: true; apic: true; device_id: 0001; vga: std; nx: true; pae: true; viridian: false; acpi: 1
                    allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; migrate_send; provision; destroy; export; clone; copy
                    current-operations (SRO): 
                    blocked-operations (MRW): 
                   allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                   allowed-VIF-devices (SRO): 2; 3; 4; 5; 6
                        possible-hosts ( RO): 6db7ad46-c045-41f8-b2b2-ab1d6ad845e3
                           domain-type ( RW): hvm
                   current-domain-type ( RO): <not in database>
                       HVM-boot-policy ( RW): BIOS order
                       HVM-boot-params (MRW): firmware: bios; order: cdn
                 HVM-shadow-multiplier ( RW): 1.000
                             PV-kernel ( RW): 
                            PV-ramdisk ( RW): 
                               PV-args ( RW): 
                        PV-legacy-args ( RW): 
                         PV-bootloader ( RW): 
                    PV-bootloader-args ( RW): 
                   last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-f7fa3223-2d93fbff-00000523-0000000f-019c47ab-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                      last-boot-record ( RO): ''
                           resident-on ( RO): <not in database>
                              affinity ( RW): <not in database>
                          other-config (MRW): import_task: OpaqueRef:432bd179-55fa-4c25-bcbf-59a6245819c6; mac_seed: d506686a-830e-d4b4-3814-33868a1315ac; base_template_name: Debian Stretch 9.0; install-methods: cdrom,nfs,http,ftp; linux_template: true
                                dom-id ( RO): -1
                       recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="false"/><restriction field="allow-gpu-passthrough" value="1"/><restriction field="allow-vgpu" value="1"/><restriction field="allow-network-sriov" value="1"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="no"/><restriction field="supports-secure-boot" value="no"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                         xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: 
            ha-always-run ( RW) [DEPRECATED]: false
                   ha-restart-priority ( RW): 
                                 blobs ( RO): 
                            start-time ( RO): <unknown time>
                          install-time ( RO): <unknown time>
                          VCPUs-number ( RO): <not in database>
                     VCPUs-utilisation (MRO): 
                            os-version (MRO): 
                    PV-drivers-version (MRO): 
    PV-drivers-up-to-date ( RO) [DEPRECATED]: false
                                memory (MRO): 
                                 disks (MRO): 
                                  VBDs (SRO): d5154254-a1dd-9873-2590-de8031f20614; 40f49a62-ad2c-1ecd-69f7-e39417f1b67c
                              networks (MRO): 
                   PV-drivers-detected ( RO): false
                                 other (MRO): has-vendor-device: 0; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1
                                  live ( RO): true
            guest-metrics-last-updated ( RO): 20220622T13:42:25Z
                   can-use-hotplug-vbd ( RO): unspecified
                   can-use-hotplug-vif ( RO): unspecified
              cooperative ( RO) [DEPRECATED]: true
                                  tags (SRW): 
                             appliance ( RW): <not in database>
                     snapshot-schedule ( RW): <not in database>
                      is-vmss-snapshot ( RO): false
                           start-delay ( RW): 0
                        shutdown-delay ( RW): 0
                                 order ( RW): 0
                               version ( RO): 0
                         generation-id ( RO): 
             hardware-platform-version ( RO): 0
                     has-vendor-device ( RW): false
                       requires-reboot ( RO): false
                       reference-label ( RO): debian-9
                          bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d

mietek

@olivierlambert Hey Oliver.
I have 2 networks configured in XO: Net1 and Net2.
Two different address pool and 2 different VLANs.
The order on the host supposed to be ETH0 -> Net1 and ETH1 -> Net2.
I have created VM with that network order and then converted it to template.
After spawning VM from that template the network order is Net2, Net1 which causes problem with my scripts.
Any reason/solution/ideas how to deal with it?

Thanks

mietek

I have a problem with spawning VMs from template in XO.
All work fine but for some reason it spawns it
with the VIFs Networks in reversed order
than original VM converted to that template.

Any tips on what might be the reason and how to make it work?
I am using xo-server 5.87.0 and xo-web 5.92.0

Thanks

mietek

I was able to find the reason XO is not working with cloud-init.
XO attaches iso with the cloud-config data as additional disk xvdb and it does not work.
But if you provide it as a cd? It will.

That is why - as for now you will need to create your own iso and attach it as a cd-rom.
You do that this way:

Get your files ready (meta-data network-config user-data). My examples below:

metadata:

instance-id: local-nocloud
local-hostname: myhost.example.com

network-config:

network:
  version: 1
  config:
    - type: physical
      name: eth0
      subnets:
         - type: static
           address: 10.0.0.5/24
           gateway: 10.0.0.254
    - type: nameserver
      address:
        - 8.8.8.8
        - 8.8.4.4
      search:
        - example.com
    
    - type: physical
      name: eth1
      subnets:
         - type: static
           address: 192.168.0.5/24

user-data:

#cloud-config

### RUN CMD
runcmd:
  - /usr/bin/apt -y install mc
  - /usr/bin/date >> /root/testfile
  
# set system default user
system_info:
  default_user:
    name: testuser
    gecos: Default Cloud User

# password auth - comment out if not using passwords
password: testpassword
ssh_pwauth: true
chpasswd: { expire: false }

# do some package management
#package_update: true
packages:
 - iptraf
 - mtr
 - screen
 - net-tools
 - atop

Generate iso

genisoimage -output myiso.iso -volid cidata -joliet -rock user-data meta-data network-config

Upload iso to XO
home -> storage -> iso's -> disks -> new disk

Prepare VM/template

Install Debian 10.6 with xo-tools
install cloud-init and cloud-initramfs-growroot
remove all network configuration leave just "allow-hotplug ethxxx"
(network config will be added in /etc/network/interfaces/50-cloud-init. It will not overwrite yours)
shutdown the machine and create the template if you want or just restart VM.
DO NOT LET IT BOOT! - keep it in grub.
attach your iso
let it run

It will do whatever you need on the machine.

mietek

This is a great news as I will be able to have all my scripts written as of now and xo-cli in one place.
Thanks @olivierlambert

For those who are looking for direct instructions ... on Debian 10.x:

# install npm:

apt install npm

# install xo-cli:

npm install -g xo-cli

mietek

following this xo-cli thread though - is there a possibility to install xo-cli on the server which is not a xen server? Or it has to be the one and even more - the one with xen-orchestra?

mietek

OK - found the problem.
I do not need to add CD I need to insert it.

xe vm-cd-insert vm=<name_of_your_vm> cd-name=<name_of_your_file.iso>

If xe will say that there is no empty cdrom - eject inserted media:

xe vm-cd-eject vm=<name_of_your_vm>

and then insert it ...

mietek

Thanks @olivierlambert - I will try that just to compare maybe it will give me results I need.

mietek

Hi all,
I am having issues with cd images attaching to the template created.
I need this for the cloud-init configuration.
There are 2 ways I try to accomplish that on freshly created Debian 10 template with cloud-init packages installed:

I create new vm in XO and boot it. I stop it on grub level, then I attach my cd image with cloud-init user, meta and network data in XO, continue with the boot - cloud-init works as expected.
I create vm in CLI:

xe vm-install new-name-label=<vm_name> template=Debian10_cloud-init_template
xe vm-cd-add vm=Debian10_cloud-init_template cd-name=nciso11.iso device=4
xe vm-start vm=Debian10_cloud-init_template

and cloud-init does not work, cd is not moutable as it says that there is noting to mount.

Only difference I see is that when I create VM in XO I got two cd devices:

CD 0 VBD:
uuid ( RO)             : 7663f608-efdd-38e0-581a-87d2a21da3fc
    vm-name-label ( RO): Debian_10_cloud_init_template
            empty ( RO): false
       userdevice ( RW): 3

CD 0 VDI:
uuid ( RO)             : 105c96fb-0d1f-4df3-bf6e-e8edf51ee78a
       name-label ( RW): nciso11.iso
    sr-name-label ( RO): ISO's
     virtual-size ( RO): 376832

When I create it in CLI I have 3 fore some reason:

CD 0 VBD:
uuid ( RO)             : 3f18cca9-2630-0e2f-d146-ddd1f716ea65
    vm-name-label ( RO): Debian_10_cloud_init_template
            empty ( RO): false
       userdevice ( RW): 4


CD 0 VDI:
uuid ( RO)             : 105c96fb-0d1f-4df3-bf6e-e8edf51ee78a
       name-label ( RW): nciso11.iso
    sr-name-label ( RO): ISO's
     virtual-size ( RO): 376832


CD 1 VBD:
uuid ( RO)             : d28e5090-4e2c-cbaf-fbab-4ccbdd54c3ca
    vm-name-label ( RO): Debian_10_cloud_init_template
            empty ( RO): true
       userdevice ( RW): 3

... and then coud-init does not work as cd is not mountable - no media (???).

Could someone explain to me why CLI creates additional and empty CD? Thanks!

mietek

Hey @p4-k4, thank you for the clarification. Appreciate it.
I've tested that last week based on your post but it does not work for me with or withough network config present for some reason. (DS not found either way).
I am thinking about preparing a vm template with manually patched cloud-init. I will be trying that approach this week. I will post if I'll find a temp solution for this.

Cheers!

Posts made by mietek

original VM:

new VM

Template details below: