XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Kubernetes cluster recipes not seeing nodes

    Scheduled Pinned Locked Moved Xen Orchestra
    43 Posts 3 Posters 19.0k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F Offline
      fred974 @fred974
      last edited by

      @GabrielG did you get a chance to look at the log I provided? Any clues?

      G 1 Reply Last reply Reply Quote 0
      • G Offline
        GabrielG @fred974
        last edited by

        Hi,

        Nothing useful. Maybe you can try to delete the VMs and redeploy the cluster.

        F 1 Reply Last reply Reply Quote 0
        • F Offline
          fred974 @GabrielG
          last edited by

          @GabrielG said in Kubernetes cluster recipes not seeing nodes:

          Nothing useful. Maybe you can try to delete the VMs and redeploy the cluster.

          Ok I will do that. Whilst I redeploy the cluster, what I am looking for? What log to monitor etc?

          G F 2 Replies Last reply Reply Quote 0
          • G Offline
            GabrielG @fred974
            last edited by

            I'd say any error in the console during the cloud-init installation.

            1 Reply Last reply Reply Quote 0
            • F Offline
              fred974 @fred974
              last edited by

              @GabrielG I deleted the VMs and redeployed it with3 nodes.
              So far only the Master VM has been created and nothing else. I am missing the 3x nodes.
              When I look at the console of the master VM, all I get is this:

              e59f3920-dbb2-473f-b5c2-ba793197a7b4-image.png

              So the master VM is created but nothing has been deployed

              I have no error on Xen Orchestra screen or log

              G 1 Reply Last reply Reply Quote 0
              • G Offline
                GabrielG @fred974
                last edited by

                The cloud-init installation comes after the step on your screenshot.

                Are the 3 nodes VMs started? Can you post the output of sudo cat /var/log/messages?

                F 1 Reply Last reply Reply Quote 0
                • F Offline
                  fred974 @GabrielG
                  last edited by

                  @GabrielG said in Kubernetes cluster recipes not seeing nodes:

                  Are the 3 nodes VMs started? Can you post the output of sudo cat /var/log/messages?

                  No node come up. Do you need the message log from all the hosts ?

                  G F 2 Replies Last reply Reply Quote 0
                  • G Offline
                    GabrielG @fred974
                    last edited by

                    Yes please.

                    1 Reply Last reply Reply Quote 0
                    • F Offline
                      fred974 @fred974
                      last edited by

                      @GabrielG I just cleared allthe /var/log/messages logs from hosts and Xen Orchestra and started again:

                      bfbe92db-d559-477d-8b8c-c3fdd1604e71-image.png
                      Host1: /var/log/messages is empty
                      Host2: /var/log/messages is empty
                      Host3: /var/log/messages is empty
                      Host4: /var/log/messages is empty
                      XOA: /var/log/messages has the following message repeated 100's time:

                      Apr  4 07:00:40 xoa kernel: [56429.938470] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:cc:2d:e0:58:82:9c:86:dd SRC=fe80:0000:0000:0000:ce2d:e0ff:fe58:829c DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=205 TC=0 HOPLIMIT=1 FLOWLBL=211291 PROTO=UDP SPT=5678 DPT=5678 LEN=165
                      Apr  4 07:01:36 xoa kernel: [56485.646598] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:c4:ad:34:4a:8d:38:86:dd SRC=fe80:0000:0000:0000:c6ad:34ff:fe4a:8d38 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=206 TC=0 HOPLIMIT=1 FLOWLBL=467355 PROTO=UDP SPT=5678 DPT=5678 LEN=166
                      

                      and this message in the middle of the mist

                      Apr  4 00:10:03 xoa rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="485" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
                      

                      As before, the master VM is created but nothing more
                      b6b60229-ab74-4425-93bf-b80c6b5af4b2-image.png

                      No nodes are created and the recipes page just have a spinning wheel with no error message
                      028f2f51-c3f3-47f6-b1f5-a6f0d8af1058-image.png

                      The XOA web log has the following:

                      sr.stats
                      {
                        "id": "a20ee08c-40d0-9818-084f-282bbca1f217",
                        "granularity": "seconds"
                      }
                      {
                        "message": "Cannot read properties of undefined (reading 'statusCode')",
                        "name": "TypeError",
                        "stack": "TypeError: Cannot read properties of undefined (reading 'statusCode')
                          at when (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:415:41)
                          at matchError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/_matchError.js:17:103)
                          at onError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:64:9)
                          at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
                          at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)
                          at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)
                          at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)
                          at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)
                          at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)
                          at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)
                          at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)
                          at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)
                          at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)
                          at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)
                          at processImmediate (node:internal/timers:471:21)
                          at process.callbackTrampoline (node:internal/async_hooks:130:17)"
                      }
                      

                      I hope this is enough information to debug the issue

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Your first Debian VM isn't booting at all. Are you sure your host is working and supporting HVM guest?

                        Can you boot the Debian 11 template from the XO Hub?

                        F 1 Reply Last reply Reply Quote 0
                        • F Offline
                          fred974 @olivierlambert
                          last edited by

                          @olivierlambert said in Kubernetes cluster recipes not seeing nodes:

                          Your first Debian VM isn't booting at all. Are you sure your host is working and supporting HVM guest?

                          Yes, the hosts has other VM running on them. Also on my original post, the master was created and also the nodes. They simply couldn't see each other

                          @olivierlambert said in Kubernetes cluster recipes not seeing nodes:

                          Can you boot the Debian 11 template from the XO Hub?

                          dab94744-84c6-4bff-a873-58b6ca1362a5-image.png
                          Yes, I can deploy Debian 11 from the XO Hub but it it won't stay up. The Vm does start and get stuck at the boot process
                          a4543a52-e92c-41f2-b1a7-a829b001f88e-image.png

                          vm.stats
                          {
                            "id": "9082c067-e945-0ee0-2aca-4b46ab5c02be"
                          }
                          {
                            "message": "Cannot read properties of undefined (reading 'statusCode')",
                            "name": "TypeError",
                            "stack": "TypeError: Cannot read properties of undefined (reading 'statusCode')
                              at when (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:415:41)
                              at matchError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/_matchError.js:17:103)
                              at onError (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:64:9)
                              at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
                              at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)
                              at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)
                              at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)
                              at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)
                              at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)
                              at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)
                              at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)
                              at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)
                              at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)
                              at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)
                              at processImmediate (node:internal/timers:471:21)
                              at process.callbackTrampoline (node:internal/async_hooks:130:17)"
                          }
                          

                          I removed the template and re-installed it again. This time i successfully managed to deploy Debian 11 from XO Hub.
                          I then tried the Kubernetes recipe again and this time the master is created but is not powering on. Should I manually start the VM?

                          F 1 Reply Last reply Reply Quote 0
                          • F Offline
                            fred974 @fred974
                            last edited by fred974

                            @olivierlambert

                            xoa.recipe.createKubernetesCluster
                            {
                              "masterName": "K8S Master",
                              "nbNodes": 3,
                              "network": "37559e16-f83d-b1f4-a1ae-b9c3336dca22",
                              "sr": "a20ee08c-40d0-9818-084f-282bbca1f217",
                              "sshKey": "ssh-ed25519 AAAAC3NzaC1lZDI1Nhbilj;jojj'hufsdklk,kp;joFWlpk8BlpKpTO3T2R44YHP6b+g Kubernetes cluster"
                            }
                            {
                              "code": "SR_BACKEND_FAILURE_46",
                              "params": [
                                "",
                                "The VDI is not available [opterr=Failed to attach VDI during \"prepare thin\": The VDI is not available [opterr=Plugin linstor-manager failed]]",
                                ""
                              ],
                              "call": {
                                "method": "VM.start",
                                "params": [
                                  "OpaqueRef:6ae3475f-7455-4328-904d-168b35b8ce5f",
                                  false,
                                  false
                                ]
                              },
                              "message": "SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Failed to attach VDI during \"prepare thin\": The VDI is not available [opterr=Plugin linstor-manager failed]], )",
                              "name": "XapiError",
                              "stack": "XapiError: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Failed to attach VDI during \"prepare thin\": The VDI is not available [opterr=Plugin linstor-manager failed]], )
                                at Function.wrap (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/_XapiError.js:16:12)
                                at /usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/transports/json-rpc.js:35:21"
                            }
                            
                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Okay so your problem isn't k8s recipe related, but elsewhere (probably the storage).

                              F 1 Reply Last reply Reply Quote 0
                              • F Offline
                                fred974 @olivierlambert
                                last edited by

                                @olivierlambert said in Kubernetes cluster recipes not seeing nodes:

                                Okay so your problem isn't k8s recipe related, but elsewhere (probably the storage).

                                Yup.. I just ran the K8S recipe on local storage and I have cluster..
                                33c5284d-e187-44a8-985f-839ec90ab3d4-image.png

                                debian@master:~$ kubectl get node
                                NAME     STATUS   ROLES           AGE     VERSION
                                master   Ready    control-plane   20m     v1.26.3
                                node-1   Ready    <none>          5m24s   v1.26.3
                                node-2   Ready    <none>          7m32s   v1.26.3
                                node-3   Ready    <none>          4m18s   v1.26.3
                                

                                Issue seem to be with the XOSTOR SR. Any chance @ronan-a could test if the K8S recipe work on his XOSTOR SR? I just want to know if it is specific to my setup or XOSTOR in general (bug)

                                1 Reply Last reply Reply Quote 0
                                • F fred974 referenced this topic on
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  I think it's totally unrelated. If you have issues to create the XO Hub Debian 11 VM, nothing else could work after that.

                                  F 1 Reply Last reply Reply Quote 0
                                  • F Offline
                                    fred974 @olivierlambert
                                    last edited by fred974

                                    @olivierlambert said in Kubernetes cluster recipes not seeing nodes:

                                    If you have issues to create the XO Hub Debian 11 VM, nothing else could work after that.

                                    What do you mean? I can create XO Hub Debian 11 on XOSTOR with no issue. I am not sure I understand you ?

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      What I mean is the fact that the issue with your k8s recipe is not related to the recipe itself, but a problem on your own XOSTOR cluster.

                                      F 1 Reply Last reply Reply Quote 0
                                      • F Offline
                                        fred974 @olivierlambert
                                        last edited by

                                        @olivierlambert said in Kubernetes cluster recipes not seeing nodes:

                                        What I mean is the fact that the issue with your k8s recipe is not related to the recipe itself, but a problem on your own XOSTOR cluster.

                                        How can we be sure its not a bug and only and issue with my system? This is way i was asking if anyone else running XOSTOR could try to confirm the issue is with my system

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Offline
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Because we identified some issues on XOSTOR recently, and that's exactly what causing your issue. If the VM doesn't boot, there's 0 chance the k8s recipe can work.

                                          F 1 Reply Last reply Reply Quote 0
                                          • F Offline
                                            fred974 @olivierlambert
                                            last edited by

                                            @olivierlambert ok so is it a matter to create K8S on local storage and moved them manually to XOSTOR untill XOSTOR issue get a fix?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post