<![CDATA[Kubernetes cluster recipes not seeing nodes]]>Hi,

I just used XO ro deploy a 1x master, 3 node Kubernetes Recipe and I don't thing the deployment was successful.

Looking at the devblog, I was expecting to enter network CIDR but there was no option for that on my screen.

When I login and run kubectl get nodes I get the following:

E0323 10:59:45.750544   23576 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0323 10:59:45.752762   23576 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0323 10:59:45.753346   23576 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0323 10:59:45.755437   23576 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0323 10:59:45.757850   23576 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

I deployed this recipe in view of leaning it so I am really not sure if what I am seeing is correct or not but looking at other posts on this forum I was expected to see something like

$ kubectl get nodes
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   22m   v1.19.2
node-1   Ready    <none>   15m   v1.19.2
node-2   Ready    <none>   15m   v1.19.2
node-3   Ready    <none>   15m   v1.19.2

The Kubernetes Master can ping all the nodes.
XOA 5.78.0
xcp-ng 8.2
XOSTOR SR

Thank you

]]>https://xcp-ng.org/forum/topic/7115/kubernetes-cluster-recipes-not-seeing-nodesRSS for NodeWed, 11 Mar 2026 01:39:47 GMTThu, 23 Mar 2023 15:11:09 GMT60<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 09 May 2023 11:35:06 GMT]]>@Raymondello you right, the issue is from the storage and I am looking into that. Thank you for sharing the UAT Testing, I didn't know about it

]]>https://xcp-ng.org/forum/post/61989https://xcp-ng.org/forum/post/61989Tue, 09 May 2023 11:35:06 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 12:47:18 GMT]]>@olivierlambert Ok will do. Kubernetes is purely to learn .. so we good. Thank you

]]>https://xcp-ng.org/forum/post/60818https://xcp-ng.org/forum/post/60818Tue, 04 Apr 2023 12:47:18 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 12:20:59 GMT]]>I would advise to not combine problems, XOSTOR is not completely bug free yet, so if you have an issue to report, get rid of XOSTOR in the equation first.

]]>https://xcp-ng.org/forum/post/60816https://xcp-ng.org/forum/post/60816Tue, 04 Apr 2023 12:20:59 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 12:02:59 GMT]]>@olivierlambert ok so is it a matter to create K8S on local storage and moved them manually to XOSTOR untill XOSTOR issue get a fix?

]]>https://xcp-ng.org/forum/post/60815https://xcp-ng.org/forum/post/60815Tue, 04 Apr 2023 12:02:59 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 11:55:01 GMT]]>Because we identified some issues on XOSTOR recently, and that's exactly what causing your issue. If the VM doesn't boot, there's 0 chance the k8s recipe can work.

]]>https://xcp-ng.org/forum/post/60814https://xcp-ng.org/forum/post/60814Tue, 04 Apr 2023 11:55:01 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 11:54:18 GMT]]>@olivierlambert said in Kubernetes cluster recipes not seeing nodes:

What I mean is the fact that the issue with your k8s recipe is not related to the recipe itself, but a problem on your own XOSTOR cluster.

How can we be sure its not a bug and only and issue with my system? This is way i was asking if anyone else running XOSTOR could try to confirm the issue is with my system

]]>https://xcp-ng.org/forum/post/60813https://xcp-ng.org/forum/post/60813Tue, 04 Apr 2023 11:54:18 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 11:52:44 GMT]]>What I mean is the fact that the issue with your k8s recipe is not related to the recipe itself, but a problem on your own XOSTOR cluster.

]]>https://xcp-ng.org/forum/post/60812https://xcp-ng.org/forum/post/60812Tue, 04 Apr 2023 11:52:44 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 11:52:56 GMT]]>@olivierlambert said in Kubernetes cluster recipes not seeing nodes:

If you have issues to create the XO Hub Debian 11 VM, nothing else could work after that.

What do you mean? I can create XO Hub Debian 11 on XOSTOR with no issue. I am not sure I understand you ?

]]>https://xcp-ng.org/forum/post/60811https://xcp-ng.org/forum/post/60811Tue, 04 Apr 2023 11:52:56 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 11:06:05 GMT]]>I think it's totally unrelated. If you have issues to create the XO Hub Debian 11 VM, nothing else could work after that.

]]>https://xcp-ng.org/forum/post/60809https://xcp-ng.org/forum/post/60809Tue, 04 Apr 2023 11:06:05 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 10:34:46 GMT]]>@olivierlambert said in Kubernetes cluster recipes not seeing nodes:

Okay so your problem isn't k8s recipe related, but elsewhere (probably the storage).

Yup.. I just ran the K8S recipe on local storage and I have cluster..
33c5284d-e187-44a8-985f-839ec90ab3d4-image.png

debian@master:~$ kubectl get node
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   20m     v1.26.3
node-1   Ready    <none>          5m24s   v1.26.3
node-2   Ready    <none>          7m32s   v1.26.3
node-3   Ready    <none>          4m18s   v1.26.3

Issue seem to be with the XOSTOR SR. Any chance @ronan-a could test if the K8S recipe work on his XOSTOR SR? I just want to know if it is specific to my setup or XOSTOR in general (bug)

]]>https://xcp-ng.org/forum/post/60806https://xcp-ng.org/forum/post/60806Tue, 04 Apr 2023 10:34:46 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 10:00:52 GMT]]>Okay so your problem isn't k8s recipe related, but elsewhere (probably the storage).

]]>https://xcp-ng.org/forum/post/60805https://xcp-ng.org/forum/post/60805Tue, 04 Apr 2023 10:00:52 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 09:59:17 GMT]]>@olivierlambert

xoa.recipe.createKubernetesCluster
{
  "masterName": "K8S Master",
  "nbNodes": 3,
  "network": "37559e16-f83d-b1f4-a1ae-b9c3336dca22",
  "sr": "a20ee08c-40d0-9818-084f-282bbca1f217",
  "sshKey": "ssh-ed25519 AAAAC3NzaC1lZDI1Nhbilj;jojj'hufsdklk,kp;joFWlpk8BlpKpTO3T2R44YHP6b+g Kubernetes cluster"
}
{
  "code": "SR_BACKEND_FAILURE_46",
  "params": [
    "",
    "The VDI is not available [opterr=Failed to attach VDI during \"prepare thin\": The VDI is not available [opterr=Plugin linstor-manager failed]]",
    ""
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:6ae3475f-7455-4328-904d-168b35b8ce5f",
      false,
      false
    ]
  },
  "message": "SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Failed to attach VDI during \"prepare thin\": The VDI is not available [opterr=Plugin linstor-manager failed]], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Failed to attach VDI during \"prepare thin\": The VDI is not available [opterr=Plugin linstor-manager failed]], )
    at Function.wrap (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/_XapiError.js:16:12)
    at /usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/transports/json-rpc.js:35:21"
}
]]>https://xcp-ng.org/forum/post/60804https://xcp-ng.org/forum/post/60804Tue, 04 Apr 2023 09:59:17 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 09:55:49 GMT]]>@olivierlambert said in Kubernetes cluster recipes not seeing nodes:

Your first Debian VM isn't booting at all. Are you sure your host is working and supporting HVM guest?

Yes, the hosts has other VM running on them. Also on my original post, the master was created and also the nodes. They simply couldn't see each other

@olivierlambert said in Kubernetes cluster recipes not seeing nodes:

Can you boot the Debian 11 template from the XO Hub?

dab94744-84c6-4bff-a873-58b6ca1362a5-image.png
Yes, I can deploy Debian 11 from the XO Hub but it it won't stay up. The Vm does start and get stuck at the boot process
a4543a52-e92c-41f2-b1a7-a829b001f88e-image.png

vm.stats
{
  "id": "9082c067-e945-0ee0-2aca-4b46ab5c02be"
}
{
  "message": "Cannot read properties of undefined (reading 'statusCode')",
  "name": "TypeError",
  "stack": "TypeError: Cannot read properties of undefined (reading 'statusCode')
    at when (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:415:41)
    at matchError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/_matchError.js:17:103)
    at onError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:64:9)
    at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:471:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

I removed the template and re-installed it again. This time i successfully managed to deploy Debian 11 from XO Hub.
I then tried the Kubernetes recipe again and this time the master is created but is not powering on. Should I manually start the VM?

]]>https://xcp-ng.org/forum/post/60803https://xcp-ng.org/forum/post/60803Tue, 04 Apr 2023 09:55:49 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 08:49:14 GMT]]>Your first Debian VM isn't booting at all. Are you sure your host is working and supporting HVM guest?

Can you boot the Debian 11 template from the XO Hub?

]]>https://xcp-ng.org/forum/post/60798https://xcp-ng.org/forum/post/60798Tue, 04 Apr 2023 08:49:14 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 08:40:39 GMT]]>@GabrielG I just cleared allthe /var/log/messages logs from hosts and Xen Orchestra and started again:

bfbe92db-d559-477d-8b8c-c3fdd1604e71-image.png
Host1: /var/log/messages is empty
Host2: /var/log/messages is empty
Host3: /var/log/messages is empty
Host4: /var/log/messages is empty
XOA: /var/log/messages has the following message repeated 100's time:

Apr  4 07:00:40 xoa kernel: [56429.938470] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:cc:2d:e0:58:82:9c:86:dd SRC=fe80:0000:0000:0000:ce2d:e0ff:fe58:829c DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=205 TC=0 HOPLIMIT=1 FLOWLBL=211291 PROTO=UDP SPT=5678 DPT=5678 LEN=165
Apr  4 07:01:36 xoa kernel: [56485.646598] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:c4:ad:34:4a:8d:38:86:dd SRC=fe80:0000:0000:0000:c6ad:34ff:fe4a:8d38 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=206 TC=0 HOPLIMIT=1 FLOWLBL=467355 PROTO=UDP SPT=5678 DPT=5678 LEN=166

and this message in the middle of the mist

Apr  4 00:10:03 xoa rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="485" x-info="https://www.rsyslog.com"] rsyslogd was HUPed

As before, the master VM is created but nothing more
b6b60229-ab74-4425-93bf-b80c6b5af4b2-image.png

No nodes are created and the recipes page just have a spinning wheel with no error message
028f2f51-c3f3-47f6-b1f5-a6f0d8af1058-image.png

The XOA web log has the following:

sr.stats
{
  "id": "a20ee08c-40d0-9818-084f-282bbca1f217",
  "granularity": "seconds"
}
{
  "message": "Cannot read properties of undefined (reading 'statusCode')",
  "name": "TypeError",
  "stack": "TypeError: Cannot read properties of undefined (reading 'statusCode')
    at when (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:415:41)
    at matchError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/_matchError.js:17:103)
    at onError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:64:9)
    at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:471:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

I hope this is enough information to debug the issue

]]>https://xcp-ng.org/forum/post/60796https://xcp-ng.org/forum/post/60796Tue, 04 Apr 2023 08:40:39 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 08:23:21 GMT]]>Yes please.

]]>https://xcp-ng.org/forum/post/60795https://xcp-ng.org/forum/post/60795Tue, 04 Apr 2023 08:23:21 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Tue, 04 Apr 2023 08:19:33 GMT]]>@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Are the 3 nodes VMs started? Can you post the output of sudo cat /var/log/messages?

No node come up. Do you need the message log from all the hosts ?

]]>https://xcp-ng.org/forum/post/60794https://xcp-ng.org/forum/post/60794Tue, 04 Apr 2023 08:19:33 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Mon, 03 Apr 2023 16:22:12 GMT]]>The cloud-init installation comes after the step on your screenshot.

Are the 3 nodes VMs started? Can you post the output of sudo cat /var/log/messages?

]]>https://xcp-ng.org/forum/post/60766https://xcp-ng.org/forum/post/60766Mon, 03 Apr 2023 16:22:12 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Mon, 03 Apr 2023 14:15:25 GMT]]>@GabrielG I deleted the VMs and redeployed it with3 nodes.
So far only the Master VM has been created and nothing else. I am missing the 3x nodes.
When I look at the console of the master VM, all I get is this:

e59f3920-dbb2-473f-b5c2-ba793197a7b4-image.png

So the master VM is created but nothing has been deployed

I have no error on Xen Orchestra screen or log

]]>https://xcp-ng.org/forum/post/60760https://xcp-ng.org/forum/post/60760Mon, 03 Apr 2023 14:15:25 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Mon, 03 Apr 2023 11:50:09 GMT]]>I'd say any error in the console during the cloud-init installation.

]]>https://xcp-ng.org/forum/post/60753https://xcp-ng.org/forum/post/60753Mon, 03 Apr 2023 11:50:09 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Mon, 03 Apr 2023 10:10:12 GMT]]>@GabrielG said in Kubernetes cluster recipes not seeing nodes:

Nothing useful. Maybe you can try to delete the VMs and redeploy the cluster.

Ok I will do that. Whilst I redeploy the cluster, what I am looking for? What log to monitor etc?

]]>https://xcp-ng.org/forum/post/60751https://xcp-ng.org/forum/post/60751Mon, 03 Apr 2023 10:10:12 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Mon, 03 Apr 2023 09:19:12 GMT]]>Hi,

Nothing useful. Maybe you can try to delete the VMs and redeploy the cluster.

]]>https://xcp-ng.org/forum/post/60746https://xcp-ng.org/forum/post/60746Mon, 03 Apr 2023 09:19:12 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Mon, 03 Apr 2023 08:27:13 GMT]]>@GabrielG did you get a chance to look at the log I provided? Any clues?

]]>https://xcp-ng.org/forum/post/60731https://xcp-ng.org/forum/post/60731Mon, 03 Apr 2023 08:27:13 GMT<![CDATA[Reply to Kubernetes cluster recipes not seeing nodes on Thu, 30 Mar 2023 16:49:55 GMT]]>@GabrielG yes, all of it

]]>https://xcp-ng.org/forum/post/60548https://xcp-ng.org/forum/post/60548Thu, 30 Mar 2023 16:49:55 GMT