Kubernetes cluster recipes not seeing nodes

GabrielG

@fred974 said in Kubernetes cluster recipes not seeing nodes:

Should it be debian?

No, only the /home/debian/.kube/config is meant to be owned by debian user.

Are you using kubectl with debian user or with the root user?

fred974

@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Are you using kubectl with debian user or with the root user?

I was using the root account I tried with the debian user and I now get something

debian@master:~$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   5d23h   v1.26.3
node-2   Ready    <none>          5d23h   v1.26.3

I have created a cluster with 1x master and 3x nodes. Should the output of the command above return 2 nodes?

GabrielG

Yes, you should have something like that:

debian@master:~$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   6m52s   v1.26.3
node-1   Ready    <none>          115s    v1.26.3
node-2   Ready    <none>          2m47s   v1.26.3
node-3   Ready    <none>          2m36s   v1.26.3

Are all worker nodes vm started? What's the output of kubectl get events?

fred974

@GabrielG Sorry for the late reply. Here is what I have.

debian@master:~$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   7d22h   v1.26.3
node-2   Ready    <none>          7d22h   v1.26.3

and

debian@master:~$ kubectl get events
No resources found in default namespace.

GabrielG

Thank you.

Are all VMs started?

What's the output of kubectl get pods --all-namespaces?

fred974

@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Are all VMs started?

Yes, all the VMs are up and running

@GabrielG said in Kubernetes cluster recipes not seeing nodes:

What's the output of kubectl get pods --all-namespaces?

debian@master:~$ kubectl get pods --all-namespaces
NAMESPACE      NAME                             READY   STATUS    RESTARTS        AGE
kube-flannel   kube-flannel-ds-mj4n6            1/1     Running   2 (3d ago)      8d
kube-flannel   kube-flannel-ds-vtd2k            1/1     Running   2 (6d19h ago)   8d
kube-system    coredns-787d4945fb-85867         1/1     Running   2 (6d19h ago)   8d
kube-system    coredns-787d4945fb-dn96g         1/1     Running   2 (6d19h ago)   8d
kube-system    etcd-master                      1/1     Running   2 (6d19h ago)   8d
kube-system    kube-apiserver-master            1/1     Running   2 (6d19h ago)   8d
kube-system    kube-controller-manager-master   1/1     Running   2 (6d19h ago)   8d
kube-system    kube-proxy-fmjnv                 1/1     Running   2 (6d19h ago)   8d
kube-system    kube-proxy-gxsrs                 1/1     Running   2 (3d ago)      8d
kube-system    kube-scheduler-master            1/1     Running   2 (6d19h ago)   8d

Thank you very much

fred974

@GabrielG Do you think I should delete all the VMs and reun the deploy recipe again? Also is it normal that I no longer have the option to set a network CIDR like before?

GabrielG

You can do that but it won't help us to understand what when wrong during the installation of the worker nodes 1 and 3.

Can you show me what's the output of sudo cat /var/log/messages for each nodes (master and workers)?

Concerning the CIDR, we are now using flannel as Container Network Interface, which uses a default CIDR (10.244.0.0/16) allocated to the pods network.

fred974

@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Can you show me what's the output of sudo cat /var/log/messages for each nodes (master and workers)?

From the master:

debian@master:~$ sudo cat /var/log/messages
Mar 26 00:10:18 master rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="572" x-info="https://www.rsyslog.com"] rsyslogd was HUPed

From node1:
https://pastebin.com/xrqPd88V

From node2:
https://pastebin.com/aJch3diH

From node3:
https://pastebin.com/Zc1y42NA

GabrielG

Thank you, I'll take a look tomorrow.

Is it the whole output for the master?

fred974

@GabrielG yes, all of it

fred974

@GabrielG did you get a chance to look at the log I provided? Any clues?

GabrielG

Hi,

Nothing useful. Maybe you can try to delete the VMs and redeploy the cluster.

fred974

@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Nothing useful. Maybe you can try to delete the VMs and redeploy the cluster.

Ok I will do that. Whilst I redeploy the cluster, what I am looking for? What log to monitor etc?

GabrielG

I'd say any error in the console during the cloud-init installation.

fred974

@GabrielG I deleted the VMs and redeployed it with3 nodes.
So far only the Master VM has been created and nothing else. I am missing the 3x nodes.
When I look at the console of the master VM, all I get is this:

So the master VM is created but nothing has been deployed

I have no error on Xen Orchestra screen or log

GabrielG

The cloud-init installation comes after the step on your screenshot.

Are the 3 nodes VMs started? Can you post the output of sudo cat /var/log/messages?

fred974

@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Are the 3 nodes VMs started? Can you post the output of sudo cat /var/log/messages?

No node come up. Do you need the message log from all the hosts ?

GabrielG

Yes please.

fred974

@GabrielG I just cleared allthe /var/log/messages logs from hosts and Xen Orchestra and started again:

Host1: /var/log/messages is empty
Host2: /var/log/messages is empty
Host3: /var/log/messages is empty
Host4: /var/log/messages is empty
XOA: /var/log/messages has the following message repeated 100's time:

Apr  4 07:00:40 xoa kernel: [56429.938470] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:cc:2d:e0:58:82:9c:86:dd SRC=fe80:0000:0000:0000:ce2d:e0ff:fe58:829c DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=205 TC=0 HOPLIMIT=1 FLOWLBL=211291 PROTO=UDP SPT=5678 DPT=5678 LEN=165
Apr  4 07:01:36 xoa kernel: [56485.646598] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:c4:ad:34:4a:8d:38:86:dd SRC=fe80:0000:0000:0000:c6ad:34ff:fe4a:8d38 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=206 TC=0 HOPLIMIT=1 FLOWLBL=467355 PROTO=UDP SPT=5678 DPT=5678 LEN=166

and this message in the middle of the mist

Apr  4 00:10:03 xoa rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="485" x-info="https://www.rsyslog.com"] rsyslogd was HUPed

As before, the master VM is created but nothing more

No nodes are created and the recipes page just have a spinning wheel with no error message

The XOA web log has the following:

sr.stats
{
  "id": "a20ee08c-40d0-9818-084f-282bbca1f217",
  "granularity": "seconds"
}
{
  "message": "Cannot read properties of undefined (reading 'statusCode')",
  "name": "TypeError",
  "stack": "TypeError: Cannot read properties of undefined (reading 'statusCode')
    at when (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:415:41)
    at matchError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/_matchError.js:17:103)
    at onError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:64:9)
    at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:471:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

I hope this is enough information to debug the issue