Can't for the life of me inject cloudinit config

p4-k4

I've configured it to use NoCloud using

dpkg-reconfigure cloud-init

However, no such luck again.

Should I be configuring it to use NoCloud in /etc/cloud/cloud.cfg rather than using the interactive setup?

If so, what should that look like? I see an entry there for EC2 as a datasource, not sure what NoCloud should look like this.

Cheers

olivierlambert

Re-check your cloudinit logs, maybe you'll spot something there

p4-k4

@olivierlambert Sorry to hound you mate.

Yeah not too much from logs other than it's loads in the user created during installation. Although it does look like it is loading in some metadata but not clear if it's from the Cloud config drive that XO creates or what. Either way, it's not the metadata specified in my cloud config.

Can you confirm if the interactive configuration using

dpkg-reconfigure cloud-init

Is the right way to go? Or should I be adding the datasource in the cloud.cfg file?

p4-k4

Is this a XOA registered feature only or? I'm almost certain I've done everything correct after probably 50 varied attempts.

olivierlambert

Cloudinit is available in XO from the sources too. If you have the correct file in the small extra disk, it's not an XO problem, it's a configuration problem in your VM.

p4-k4

Solved - At least in this case, network config must be specified otherwise nothing will happen.

olivierlambert

So the issue was to leave network config empty and then Cloudinit ignored the rest?

p4-k4

@olivierlambert

I haven't tried with an empty network config.

All I done was remove the # from the first two lines in the network config which then allowed the VM to utilize the cloud config drive.

olivierlambert

Thanks for the feedback. Maybe it was blocking cloud knit.

Ping @fohdeesha about this

fohdeesha

That's...strange. Can you try to reproduce the errors/failure to load in the VM again and grab the logs from the deployed VM? They should be under /var/log/cloud-init.log and /var/log/cloud-init-output.log - it should give us quite a bit of details

p4-k4

@fohdeesha

These are the logs from spinning up without configuring a network through cloud-init. As mentioned, all lines are commented out with # in the pre-populated text.

https://pastebin.ubuntu.com/p/4McRtgYR7f/

https://pastebin.ubuntu.com/p/q7BHQ7qW8k/

fohdeesha

That's very strange, it's finding the nocloud drive, mounting it, and pulling the config files off of it. Interestingly, it's finding a network-config file and trying to read it, then it looks like it decides it doesn't like the formatting, and decides to ignore the whole nocloud drive config source:

2020-08-24 12:36:22,816 - util.py[DEBUG]: Running command ['mount', '-o', 'ro', '-t', 'auto', '/dev/xvdb', '/run/cloud-init/tmp/tmpy_922uof'] with allowed return codes [0] (shell=False, capture=True)
2020-08-24 12:36:22,830 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpy_922uof//user-data (quiet=False)
2020-08-24 12:36:22,830 - util.py[DEBUG]: Read 109 bytes from /run/cloud-init/tmp/tmpy_922uof//user-data
2020-08-24 12:36:22,831 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpy_922uof//meta-data (quiet=False)
2020-08-24 12:36:22,831 - util.py[DEBUG]: Read 50 bytes from /run/cloud-init/tmp/tmpy_922uof//meta-data
2020-08-24 12:36:22,831 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpy_922uof//vendor-data (quiet=False)
2020-08-24 12:36:22,831 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpy_922uof//network-config (quiet=False)
2020-08-24 12:36:22,831 - util.py[DEBUG]: Read 104 bytes from /run/cloud-init/tmp/tmpy_922uof//network-config
2020-08-24 12:36:22,831 - util.py[DEBUG]: Running command ['umount', '/run/cloud-init/tmp/tmpy_922uof'] with allowed return codes [0] (shell=False, capture=True)
2020-08-24 12:36:22,838 - util.py[DEBUG]: Attempting to load yaml from string of length 50 with allowed root types (<class 'dict'>,)
2020-08-24 12:36:22,839 - util.py[DEBUG]: Attempting to load yaml from string of length 104 with allowed root types (<class 'dict'>,)
2020-08-24 12:36:22,839 - util.py[DEBUG]: loaded blob returned None, returning default.
2020-08-24 12:36:22,839 - handlers.py[DEBUG]: finish: init-local/search-NoCloud: FAIL: no local data found from DataSourceNoCloud
2020-08-24 12:36:22,839 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed
2020-08-24 12:36:22,844 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed

You said you were commenting out lines in the network config box, I might take a guess that that gets written to the config drive, and cloud-init tries to read it. Since it wants strict yaml, it's probably failing on those # characters. Can you try leaving the network config box in XOA COMPLETELY empty? (not even a space)

p4-k4

Hmm so it seems that clearing the box completely also leaves no errors and functions as expected.

Leaving it in with comment markers seems to trigger the error from what I can tell.

Maybe someone else can replicate the issue to confirm. I'm using a clean Ubuntu 18 image.

fohdeesha

@p4-k4 Indeed I believe this is expected behavior. It expects strict YAML formatted code in the network config file that is passed to it, and comments are unrecognized.

olivierlambert

You can check if it's Yaml valid with https://codebeautify.org/yaml-validator

# is a valid char for comments.

Anyway, sounds like valid Yaml to me, but if other can reproduce (ideally both on the same Ubuntu version but on CentOS and Debian to see if it's not a Cloudinit bug on a specific version)

p4-k4

@fohdeesha

I thought commenting every line with # would be valid YAML which is how you guys have it as default.

Which is cuasing the problem at least on my side of things.

mietek

Hey gents were you able to find the root cause for this issue?
I see the same problem on Debian 10.

I can see XO providing xvdb with all of the data. I can mount it and see all the info as expected.
I am able to start cloud init (systemctl start cloud-init) and all changes from user-data provided are done to the system not from network though (but that might be an expected behaviour as the system is running already?)
That being said - nothing happens on the VM boot.

I have checked systemd and cloud-init is enabled so all should run on the boot but for some reason it is not.

There are no log files as cloud-init is not able to find datasource:
[from /run/cloud-init/cloud-init-generator.log]

"cloud-init is enabled but no datasource found, disabling"

In my case there is no difference if the network text box in XO is empty or not or just first two lines are unhashaed.

I am using fully updated ver of debian 10 and XO 5.59.0
I assume Debian bug about not recognizing disk should be fixed already (?).
Cloud-init ver. 20.2-2.

Any tips on what am I missing?
Thanks

EDIT: looks like still bug was not fixed.
I have checked as mentioned in below manual and it's still marks /dev/xvdb wrong ...
https://www.gitmemory.com/issue/vatesfr/xen-orchestra/4449/614872988

ver: 20.2-2 is still the one with a bug. There is 20.3 awaiting to be delivered but still awaiting to be packaged:
https://tracker.debian.org/pkg/cloud-init

p4-k4

I had the same problem with Debian 10 also.

In my findings it seemed that the "commented out" network config that XO ships with was throwing it off.

The workaround in my case was...

Insert my cloud-config as per usual.
Remove everything from the network config box (if I didn't want cloud-config to set this up for me).

or

Insert a proper network config if I wanted network setup also.

Hope that helps

mietek

Hey @p4-k4, thank you for the clarification. Appreciate it.
I've tested that last week based on your post but it does not work for me with or withough network config present for some reason. (DS not found either way).
I am thinking about preparing a vm template with manually patched cloud-init. I will be trying that approach this week. I will post if I'll find a temp solution for this.

Cheers!

p4-k4

What does your cloud config logs show on the guest OS after boot?

Can you see if it has made any attempt to inject the config?
Only thing I can think of is that your cloud config might be wrong thus the cloud init service is refusing it.

Have you tried to use an ubuntu image? I used official images of both debian 10 and ubuntu 18.04 and worked fine once I got rid of the pre-populated network config or inserted a real network config.

I spent too long trying to figure out what was going on so let me know if there's anything else you need to know, it's super frustrating.