XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    How to kubernetes on xcp-ng (csi?)

    Scheduled Pinned Locked Moved Development
    44 Posts 9 Posters 16.3k Views 9 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Offline
      mohammadm @olivierlambert
      last edited by mohammadm

      @olivierlambert said in How to kubernetes on xcp-ng (csi?):

      Can you try on latest release channel?

      Samething, again apipa ip.

      Trying to login on the machine, is it the admin : admin?

      1 Reply Last reply Reply Quote 0
      • M Offline
        mohammadm
        last edited by

        On console I am getting "Failed to start Execute cloud user/final scripts."

        suddenly it has an ip address, but the installation has failed.

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          There's no default user/pass for obvious security reasons. Are you sure the VMs are deployed in the right network where you have a DHCP server (for example)?

          Also pinging @shinuza

          1 Reply Last reply Reply Quote 0
          • Theoi-MeteoroiT Offline
            Theoi-Meteoroi
            last edited by

            I've been using this with NVMe on 3 Dell 7920 boxen with PCI passthru.

            https://github.com/piraeusdatastore/piraeus-operator

            It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.

            Just one note on handing devices to the operator - I use raw NVMe disk.
            There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.

            I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.

            M 1 Reply Last reply Reply Quote 0
            • M Offline
              mohammadm @Theoi-Meteoroi
              last edited by

              @Theoi-Meteoroi said in How to kubernetes on xcp-ng (csi?):

              I've been using this with NVMe on 3 Dell 7920 boxen with PCI passthru.

              https://github.com/piraeusdatastore/piraeus-operator

              It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.

              Just one note on handing devices to the operator - I use raw NVMe disk.
              There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.

              I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.

              Did you use the built-in Recipes to create the kubernetes cluster? I tried NVMe, iCSCI, SSD, NFS Share. All the same thing.
              K8s.PNG

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                You have a network issue (well, a DNS one) inside your VM, are you using the right network?

                M 1 Reply Last reply Reply Quote 1
                • M Offline
                  mohammadm @olivierlambert
                  last edited by

                  @olivierlambert said in How to kubernetes on xcp-ng (csi?):

                  You have a network issue (well, a DNS one) inside your VM, are you using the right network?

                  Feel so dumb. When creating a VM, usually the top Network is the correct one. For Kubernetes, I had to scroll all the way down and select the correct network.

                  K8s1.PNG K8s2.PNG

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Is it working correctly now?

                    M 1 Reply Last reply Reply Quote 0
                    • M Offline
                      mohammadm @olivierlambert
                      last edited by mohammadm

                      @olivierlambert said in How to kubernetes on xcp-ng (csi?):

                      Is it working correctly now?

                      K8s.PNG

                      Currently it is stuck on this.

                      [FAILED] Failed to start Execute cloud user/final scripts.

                      cp-1 login:

                      I did not specify login credentials.

                      S 1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Still a network problem (Cloud init can't reach something, no route to host)

                        M 2 Replies Last reply Reply Quote 0
                        • M Offline
                          mohammadm @olivierlambert
                          last edited by

                          @olivierlambert said in How to kubernetes on xcp-ng (csi?):

                          Still a network problem (Cloud init can't reach something, no route to host)

                          This is the same error I used to get on my homelab when manually installing Ubuntu and trying to deploy k3s with rancheros and longhorn.

                          This try is on our datacenter and not my homelab. I'll do another setup with DHCP.

                          1 Reply Last reply Reply Quote 0
                          • M Offline
                            mohammadm @olivierlambert
                            last edited by

                            @olivierlambert Nope, still the same error.
                            I don't think our Sonicwall (in the datacenter) is blocking anything. Since at home, I am using Unifi.

                            1 Reply Last reply Reply Quote 0
                            • xyhhxX Offline
                              xyhhx
                              last edited by

                              fwiw, i'm about to set up a kubernetes cluster on xcp-ng. i'm still in the process, but i'm planning on just passing disks as hba storage to worker/storage nodes, then using openebs jiva (or maybe rook/ceph)

                              if anybody is interested in how that goes, i can post about it later on

                              S 1 Reply Last reply Reply Quote 1
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Adding @shinuza also in the convo

                                1 Reply Last reply Reply Quote 0
                                • S Offline
                                  shinuza @mohammadm
                                  last edited by

                                  @mohammadm said in How to kubernetes on xcp-ng (csi?):

                                  @olivierlambert said in How to kubernetes on xcp-ng (csi?):

                                  Is it working correctly now?

                                  K8s.PNG

                                  Currently it is stuck on this.

                                  [FAILED] Failed to start Execute cloud user/final scripts.

                                  cp-1 login:

                                  I did not specify login credentials.

                                  The error is probably earlier than that. You can see the full output of the cloudinit script in /var/log/cloud-init-output.log
                                  The error you are seeing is most likely due to the fact the kubernetes cluster could not be initialized. (kubeadm init failed)

                                  1 Reply Last reply Reply Quote 0
                                  • S Offline
                                    shinuza @xyhhx
                                    last edited by

                                    @xyhhx said in How to kubernetes on xcp-ng (csi?):

                                    fwiw, i'm about to set up a kubernetes cluster on xcp-ng. i'm still in the process, but i'm planning on just passing disks as hba storage to worker/storage nodes, then using openebs jiva (or maybe rook/ceph)

                                    if anybody is interested in how that goes, i can post about it later on

                                    Jiva is probably not the way to go, I believe the supported way is to use Mayastor (via nvme-tcp)

                                    xyhhxX 1 Reply Last reply Reply Quote 0
                                    • xyhhxX Offline
                                      xyhhx @shinuza
                                      last edited by

                                      @shinuza oh, yeah. they've deprecated it in the last couple of months. thanks for the heads up

                                      1 Reply Last reply Reply Quote 0
                                      • Theoi-MeteoroiT Offline
                                        Theoi-Meteoroi
                                        last edited by

                                        I should mention I use static IP for my VMs and I don't deploy using the hub recipes because my home setup is all Community bits and 8.3 XCP, updated regularly.
                                        I use kubespray. I'm at kubernetes 1.27.7

                                        So I can't provide any help with cloud init or hub/recipes.

                                        The working setup I have is built on CentOS7 ( need to replace ) VMs. I tried deploying on RockyOS 9.3 and spun my wheels for too long on issues there so I'm going to try 8.9.

                                        I just updated my XO-CE bits and see some nice new features in the UI that will be helpful with all the PCI passthru I want to do. I was putting add/remove instructions in the VM notes and tags on the VMs just to remind me of what they show below. Its like they read my mind.

                                        I need to learn KubeVirt.

                                        Screenshot 2024-05-03 at 11.34.42 PM.png

                                        Its christmas in may.

                                        xyhhxX 1 Reply Last reply Reply Quote 0
                                        • xyhhxX Offline
                                          xyhhx @Theoi-Meteoroi
                                          last edited by

                                          @Theoi-Meteoroi check out https://talos.dev 😎

                                          Theoi-MeteoroiT 1 Reply Last reply Reply Quote 0
                                          • Theoi-MeteoroiT Offline
                                            Theoi-Meteoroi @xyhhx
                                            last edited by

                                            @xyhhx Why thanks! I'll take a look at it. I've done coreos and flatcar for work but find immutable OS no fun at home. I'm used to doing all the admin already and most of my VMs live long enough to be classed as "pets".

                                            xyhhxX 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post