Kuberenets cluster recipe not happy

olivierlambert

Hi,

This is something for @BenjiReis

BenjiReis

Hi! The kube nodes communicate with the master with port 6433.

I don't understand the error message.

Can you try running on all nodes and master:

 - update-alternatives --set iptables /usr/sbin/iptables-legacy
 - update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

and reboot your Kubernetes nodes and master?

pnunn

@benjireis I did all of that... (with sudo first)... same result though.

Do you want me to start a ticket and a tunnel?

Peter.

BenjiReis

@pnunn Yeah please open a ticket and a tunnel, I'll have a look.

Please make sure to add the xoa ssh key to the VMs so I can ssh to the nodes and master.

mathiashedberg

Hi!

I had a similar issue, however i noticed that the k8s VMs created were based on a snapshot of a different VM i had running on the system. So in turn i ended up with a bunch of clones of a completely different VM.

This VM that the new k8s VMs were based on, was created based off the Debian 10 template. New VMs created with the debian 10 template work fine, however only the k8s recipe ends up basing the image on this existing VM.

BenjiReis

A new version has been released last week or so that solves issues in the recipe.
Did you encounter any issue with latest xoa?

mathiashedberg

@benjireis
Yes currently the issue is with version 5.54.0. Tested about an hour ago

BenjiReis

Latest XOA is 5.55.1, can you try with this version?

mathiashedberg

@benjireis
Looks like that fixed that issue. Now there is an issue with the cloud-config. Seems like gnupg2 is not installed

sudo systemctl status cloud-final.service
● cloud-final.service - Execute cloud user/final scripts
   Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-02-15 09:20:37 EST; 5min ago
  Process: 649 ExecStart=/usr/bin/cloud-init modules --mode=final (code=exited, status=1/FAILURE)
 Main PID: 649 (code=exited, status=1/FAILURE)

Relevant snippet: I could send you the whole snippet from sudo journalctl -u cloud-final.service if you would like

Feb 15 09:20:34 node-1 cloud-init[649]: 0 added, 0 removed; done.
Feb 15 09:20:34 node-1 cloud-init[649]: Running hooks in /etc/ca-certificates/update.d...
Feb 15 09:20:34 node-1 cloud-init[649]: done.
Feb 15 09:20:34 node-1 cloud-init[649]: Errors were encountered while processing:
Feb 15 09:20:34 node-1 cloud-init[649]:  linux-image-4.19.0-14-amd64
Feb 15 09:20:34 node-1 cloud-init[649]:  linux-image-amd64
Feb 15 09:20:34 node-1 cloud-init[649]: FATAL -> Failed to fork.
Feb 15 09:20:34 node-1 cloud-init[649]: Cloud-init v. 18.3 running 'modules:final' at Mon, 15 Feb 2021 14:19:26 +0000. Up 15.00 seconds.
Feb 15 09:20:34 node-1 cloud-init[649]: 2021-02-15 14:20:34,984 - util.py[WARNING]: Package upgrade failed
Feb 15 09:20:35 node-1 cloud-init[649]: Reading package lists...FATAL -> Failed to fork.
Feb 15 09:20:35 node-1 cloud-init[649]: 2021-02-15 14:20:35,124 - util.py[WARNING]: Failed to install packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg2', 'software-properties-common']
Feb 15 09:20:35 node-1 cloud-init[649]: 2021-02-15 14:20:35,127 - cc_package_update_upgrade_install.py[WARNING]: 2 failed with exceptions, re-raising the last one
Feb 15 09:20:35 node-1 cloud-init[649]: 2021-02-15 14:20:35,128 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_in
Feb 15 09:20:35 node-1 cloud-init[649]: /var/lib/cloud/instance/scripts/runcmd: 2: /var/lib/cloud/instance/scripts/runcmd: curl: not found
Feb 15 09:20:35 node-1 sudo[17486]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/apt-key add -
Feb 15 09:20:35 node-1 sudo[17486]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 15 09:20:35 node-1 cloud-init[649]: E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
Feb 15 09:20:35 node-1 cloud-init[649]: /var/lib/cloud/instance/scripts/runcmd: 3: /var/lib/cloud/instance/scripts/runcmd: curl: not found
Feb 15 09:20:35 node-1 sudo[17486]: pam_unix(sudo:session): session closed for user root
Feb 15 09:20:35 node-1 sudo[17505]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/apt-key add -
Feb 15 09:20:35 node-1 sudo[17505]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 15 09:20:35 node-1 cloud-init[649]: E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
Feb 15 09:20:35 node-1 cloud-init[649]: /var/lib/cloud/instance/scripts/runcmd: 4: /var/lib/cloud/instance/scripts/runcmd: add-apt-repository: not found
Feb 15 09:20:35 node-1 cloud-init[649]: /var/lib/cloud/instance/scripts/runcmd: 5: /var/lib/cloud/instance/scripts/runcmd: add-apt-repository: not found
Feb 15 09:20:35 node-1 sudo[17505]: pam_unix(sudo:session): session closed for user root
Feb 15 09:20:35 node-1 cloud-init[649]: Hit:1 http://security.debian.org buster/updates InRelease
Feb 15 09:20:35 node-1 cloud-init[649]: Hit:2 http://deb.debian.org/debian buster InRelease
Feb 15 09:20:35 node-1 cloud-init[649]: Hit:3 http://deb.debian.org/debian buster-updates InRelease
Feb 15 09:20:35 node-1 cloud-init[649]: Hit:4 http://deb.debian.org/debian buster-backports InRelease
Feb 15 09:20:36 node-1 cloud-init[649]: Reading package lists...

mathiashedberg

This post is deleted!

BenjiReis

That's weird, I've just ran the recipe and everything went fine.
Do you have any network issues explaining the installation problem?

debian@master:~$ sudo systemctl status cloud-final.service
● cloud-final.service - Execute cloud user/final scripts
   Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2021-02-15 09:47:38 EST; 14min ago
  Process: 558 ExecStart=/usr/bin/cloud-init modules --mode=final (code=exited, status=0/SUCCESS)
 Main PID: 558 (code=exited, status=0/SUCCESS)

olivierlambert

This or not enough RAM in the VM, explaining why fork failed to create a new thread…

mathiashedberg

@olivierlambert @BenjiReis
Im also guessing too little RAM. The VMs were started with only 500MB, which i found strange. Any way for me to change this on initialization?

Im guessing i could change the Debian 9 template

BenjiReis

Your VM should have 2G of RAM. Base on our Debian 10 template.
This is really weird.

mathiashedberg

@benjireis Mine says "Original Template Debian Stretch 9.0"

BenjiReis

Weird, which templates appear in your Hub view?

mathiashedberg

Just the standard ones:

Alpine 3.10
Centos 8.0
Debian 10
pfsense 2.4

My XOA has two pools

This issue is similar to the one i had earlier, where the image was being based on a different existing VM, now in this case its the debian 9 template.
I know the debian 10 template works fine as this is what i normally use

BenjiReis

I do not understand how the wrong templates can be used by the recipe.

mathiashedberg

Yes it is quite strange.

This XOA has been in use for some time with a couple of pools added and removed over time, so I may be an edge case here.

olivierlambert

Delete the template, maybe it's a previous version