Cloudinit does not work correctly

mlustosa

I'm using Xenserver 7.1 and XenOrchestra (xo-server 5.62.0 and xo-web 5.63.0).
I'm following this tutorial and installed a CentOS 7.0 minimal to create a cloudinit template from scratch.
When trying to upload a VM from the created template, it can only load the cloud drive if I use the SSH-Key option (the first option below).
alt text

However, if I put the same configuration in the custom config option it doesn't work. Below is the configuration I am using (replacing with my corresponding key in ssh-rsa <myKey>). for example, following this tutorial.

# cloud-config
hostname: proxy
ssh_authorized_keys:
   - ssh-rsa <myKey>

When I try to connect via ssh to the newly created VM, it gives a permission error.

ssh -i ~/.ssh/mykey centos@10.xx.xx.109

so...

Warning: Permanently added '10.xx.xx.109 '(ECDSA) to the list of known hosts.
centos@10.xx.xx.109: Permission denied (publickey, gssapi-keyex, gssapi-with-mic).

Right after that I connect to the VM via root and go to /home/centos/.ssh/authorized_keys and there's nothing inside.
I tested it by also putting parameters for changing the hostname and installing some packages in the custom config and none of them worked for me.

I looked at /var/log/cloud-init to check for something strange and saw nothing that could indicate an error. He can even mount the disk containing the settings (config drive) /dev/xvdb. Here is a part of a part of the attempt to connect to the config drive and read the files.

2020-07-15 22:12:06,723 - util.py[DEBUG]: Running command ['mount', '-o', 'ro,sync', '-t', 'auto', u'/dev/xvdb', '/run/cloud-init/tmp/tmplcFFJh'] with allowed return codes [0] (shell=False, capture=True)
2020-07-15 22:12:06,801 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//user-data (quiet=False)
2020-07-15 22:12:06,802 - util.py[DEBUG]: Read 232 bytes from /run/cloud-init/tmp/tmplcFFJh//user-data
2020-07-15 22:12:06,802 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//meta-data (quiet=False)
2020-07-15 22:12:06,803 - util.py[DEBUG]: Read 50 bytes from /run/cloud-init/tmp/tmplcFFJh//meta-data
2020-07-15 22:12:06,803 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//vendor-data (quiet=False)
2020-07-15 22:12:06,803 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//network-config (quiet=False)
2020-07-15 22:12:06,805 - util.py[DEBUG]: Read 104 bytes from /run/cloud-init/tmp/tmplcFFJh//network-config
2020-07-15 22:12:06,805 - util.py[DEBUG]: Running command ['umount', '/run/cloud-init/tmp/tmplcFFJh'] with allowed return codes [0] (shell=False, capture=True)
2020-07-15 22:12:06,825 - util.py[DEBUG]: Attempting to load yaml from string of length 50 with allowed root types (<type 'dict'>,)
2020-07-15 22:12:06,826 - util.py[DEBUG]: Attempting to load yaml from string of length 104 with allowed root types (<type 'dict'>,)
2020-07-15 22:12:06,827 - util.py[DEBUG]: loaded blob returned None, returning default.
2020-07-15 22:12:06,827 - handlers.py[DEBUG]: finish: init-local/search-NoCloud: FAIL: no local data found from DataSourceNoCloud
2020-07-15 22:12:06,827 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed
2020-07-15 22:12:06,828 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 743, in find_source
    if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 632, in update_metadata
    result = self.get_data()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 256, in get_data
    return_value = self._get_data()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceNoCloud.py", line 126, in _get_data
    mydata = _merge_new_seed(mydata, seeded)
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceNoCloud.py", line 353, in _merge_new_seed
    util.load_yaml(seeded.get('network-config')))
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceNoCloud.py", line 324, in _maybe_remove_top_network
    network_val = cfg.get('network', nullval)
AttributeError: 'NoneType' object has no attribute 'get'
2020-07-15 22:12:06,842 - main.py[DEBUG]: No local datasource found

The full cloud-init.log file is here.

What is the explanation for the ssh key config drive to work and the xen orchestra custom config not?

olivierlambert

Have you compared the two generated files using the two different methods? That would be helpful to spot a difference (or not)

mlustosa

@olivierlambert
I compared and as I said, really with SSH-key only he can read the data and with custom config not.

/var/log/cloud-init.log after applying SSH-key:

2020-07-16 11:56:48,074 - util.py[DEBUG]: Running command ['mount', '-o', 'ro,sync', '-t', 'auto', u'/dev/xvdb', '/run/cloud-init/tmp/tmpmejw9b'] with allowed return codes [0] (shell=False, capture=True)
2020-07-16 11:56:48,141 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpmejw9b//user-data (quiet=False)
2020-07-16 11:56:48,143 - util.py[DEBUG]: Read 475 bytes from /run/cloud-init/tmp/tmpmejw9b//user-data
2020-07-16 11:56:48,143 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpmejw9b//meta-data (quiet=False)
2020-07-16 11:56:48,144 - util.py[DEBUG]: Read 50 bytes from /run/cloud-init/tmp/tmpmejw9b//meta-data
2020-07-16 11:56:48,144 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpmejw9b//vendor-data (quiet=False)
2020-07-16 11:56:48,144 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpmejw9b//network-config (quiet=False)
2020-07-16 11:56:48,144 - util.py[DEBUG]: Running command ['umount', '/run/cloud-init/tmp/tmpmejw9b'] with allowed return codes [0] (shell=False, capture=True)
2020-07-16 11:56:48,159 - util.py[DEBUG]: Attempting to load yaml from string of length 50 with allowed root types (<type 'dict'>,)
2020-07-16 11:56:48,160 - DataSourceNoCloud.py[DEBUG]: Using data from /dev/xvdb
2020-07-16 11:56:48,161 - util.py[DEBUG]: Reading from /etc/hosts (quiet=False)
2020-07-16 11:56:48,161 - util.py[DEBUG]: Read 158 bytes from /etc/hosts
2020-07-16 11:56:48,164 - handlers.py[DEBUG]: finish: init-local/search-NoCloud: SUCCESS: found local data from DataSourceNoCloud
2020-07-16 11:56:48,164 - stages.py[INFO]: Loaded datasource DataSourceNoCloud - DataSourceNoCloud [seed=/dev/xvdb][dsmode=net]

/var/log/cloud-init.log after applying custom config:

2020-07-15 22:12:06,723 - util.py[DEBUG]: Running command ['mount', '-o', 'ro,sync', '-t', 'auto', u'/dev/xvdb', '/run/cloud-init/tmp/tmplcFFJh'] with allowed return codes [0] (shell=False, capture=True)
2020-07-15 22:12:06,801 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//user-data (quiet=False)
2020-07-15 22:12:06,802 - util.py[DEBUG]: Read 232 bytes from /run/cloud-init/tmp/tmplcFFJh//user-data
2020-07-15 22:12:06,802 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//meta-data (quiet=False)
2020-07-15 22:12:06,803 - util.py[DEBUG]: Read 50 bytes from /run/cloud-init/tmp/tmplcFFJh//meta-data
2020-07-15 22:12:06,803 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//vendor-data (quiet=False)
2020-07-15 22:12:06,803 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmplcFFJh//network-config (quiet=False)
2020-07-15 22:12:06,805 - util.py[DEBUG]: Read 104 bytes from /run/cloud-init/tmp/tmplcFFJh//network-config
2020-07-15 22:12:06,805 - util.py[DEBUG]: Running command ['umount', '/run/cloud-init/tmp/tmplcFFJh'] with allowed return codes [0] (shell=False, capture=True)
2020-07-15 22:12:06,825 - util.py[DEBUG]: Attempting to load yaml from string of length 50 with allowed root types (<type 'dict'>,)
2020-07-15 22:12:06,826 - util.py[DEBUG]: Attempting to load yaml from string of length 104 with allowed root types (<type 'dict'>,)
2020-07-15 22:12:06,827 - util.py[DEBUG]: loaded blob returned None, returning default.
2020-07-15 22:12:06,827 - handlers.py[DEBUG]: finish: init-local/search-NoCloud: FAIL: no local data found from DataSourceNoCloud
2020-07-15 22:12:06,827 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed
2020-07-15 22:12:06,828 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 743, in find_source
    if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 632, in update_metadata
    result = self.get_data()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 256, in get_data
    return_value = self._get_data()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceNoCloud.py", line 126, in _get_data
    mydata = _merge_new_seed(mydata, seeded)
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceNoCloud.py", line 353, in _merge_new_seed
    util.load_yaml(seeded.get('network-config')))
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceNoCloud.py", line 324, in _maybe_remove_top_network
    network_val = cfg.get('network', nullval)
AttributeError: 'NoneType' object has no attribute 'get'
2020-07-15 22:12:06,842 - main.py[DEBUG]: No local datasource found

In summary, in the first case I have:
Loaded datasource DataSourceNoCloud - DataSourceNoCloud [seed=/dev/xvdb][dsmode=net]

And in the second I have:
handlers.py[DEBUG]: finish: init-local/search-NoCloud: FAIL: no local data found from DataSourceNoCloud

Another question, according to this tutorial, besides the possibility of configuring the public key, the config drive (SSH-key) also changes the hostname to the name defined in the template, however, even with SSH-key enabled it changes the name for localhost (and not for the name fixed in the template).
Below is a demonstration after booting the VM with SSH-key cloud drive enabled.

[root@localhost cloud] cat /var/lib/cloud/data/set-hostname 
{
 "fqdn": "localhost", 
 "hostname": "centos-app"
}

[root@localhost cloud] hostname
localhost

mlustosa

I took a new test. I saved the VDI created (config drive) by Xen Orchestra using the SSH-Key option. I created a VM with cloud-init installed and before loading it I connected this VDI to it. It worked, including the packages option, as shown below:

#cloud-config
hostname: centos-app
ssh_authorized_keys:
  - ssh-rsa <myKey>
packages:
  - vim
  - git
  - tcpdump

When the VM starts, the public key ssh is in ~/.ssh/authorized_keys and the vim, git and tcpdump packages are installed.

Which generates the following hypothesis: The problem is in the generation of the VDI config drive (with custom config) using Xen Orchestra.

Any direction?

olivierlambert

That's why I asked in the first place, to compare the generated files in the cloud init disk.

mlustosa

/mnt/dados -> /dev/xvdb (config drive with SSH-key)
/mnt/dados2 -> /dev/xvdc (config drive with custom config)

[centos@localhost ~]$ blkid /dev/xvdb
/dev/xvdb: SEC_TYPE="msdos" LABEL="cidata" UUID="355A-4FC2" TYPE="vfat"

[centos@localhost ~]$ blkid /dev/xvdc
/dev/xvdc: SEC_TYPE="msdos" LABEL="cidata" UUID="355A-4FC2" TYPE="vfat"

[centos@localhost ~]$ fdisk -l

Disk /dev/xvdb: 10 MB, 10485760 bytes, 20480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

    Device Boot      Start         End      Blocks   Id  System

Disk /dev/xvdc: 10 MB, 10485760 bytes, 20480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

    Device Boot      Start         End      Blocks   Id  System

[centos@localhost ~]$ diff /mnt/dados /mnt/dados2

diff /mnt/dados/meta-data /mnt/dados2/meta-data
1c1
< instance-id: 88d79afd-2970-176a-e9e9-916540d993c2
---
> instance-id: 9a9aad16-b51e-cd6c-4f6e-5afe9b757d4e
Only in /mnt/dados: network-config
Common subdirectories: /mnt/dados/openstack and /mnt/dados2/openstack
diff /mnt/dados/user-data /mnt/dados2/user-data
2c2
< hostname: centos7-xentools0
---
> hostname: CentOS-7-cloudinit-withxentools
4c4,5
<    - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM2eZQHBkRY7BUZ+sJG9JH9nngFOmednoMG6Gxn7Trrq local marllus
---
>   - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM2eZQHBkRY7BUZ+sJG9JH9nngFOmednoMG6Gxn7Trrq local marllus
>   - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDwklFtEWwv6frHY3RtzLzV4Cjk42Z6e+DoSSVo2Oc5BVRBUAgQ0TcjdJQKxQt5VxZs7inRlFcOJ/TU8PYveb0n7uPNDQRXHKKFFq/G6ByGORiMWsfGrA7CQe7lTLFmcDprqqxEuhfMNjC5wL9kegTS2boU+u8PyKFKnTT91ivPuaOCwhHM0k2N6Q+PIsuBj6UsrAt08XQHCwup2jvM+MrtQVRju+kHMhXHvUeOP19gCBkSFgU+jKJoHFQ14Fce4z9K+TvYqyBIlHJh/aTV9iTfsnbb3KHRCMu2hxD0V2TPCrlz24xajuKqQxAWieNbhcquRKPuU58jeegZ8VwzbaV3 marllus@peba

olivierlambert

@mlustosa said in Cloudinit does not work correctly:

ssh-ed25519

This is probably the issue. Are you sure you added correctly your key manually?

mlustosa

@olivierlambert
I found out where the error is. For some reason it doesn't load the user-data file (cloud-config) if the network-config has something written, even commented out (starting with #).

I removed the file out of the config drive and restarted the VM and then it worked! It managed to add the public key ssh and install some packages on the system. If in the act of creating the VM in the Xen Orchestra I leave the network-config blank also works.

Why doesn't he accept the commented network-config file?

As in the act of creation the file is commented (network-config), so I thought that keeping it that way would not influence anything.

mlustosa

@olivierlambert said in Cloudinit does not work correctly:

@mlustosa said in Cloudinit does not work correctly:

ssh-ed25519

This is probably the issue. Are you sure you added correctly your key manually?

it's correct. I have two keys, one id_ed25519 and the other id_rsa.