How to kubernetes on xcp-ng (csi?)
-
Next steps take a bit more time because it's about storing the cluster key safely to be able (then) to use XO to make basic queries on it (like current version and such), which is the first step to prepare the automated node upgrade/replacement
-
Trying to build a cluster from the hub, bit it is giving me "Err: http://deb.debian.org/debian bullseye/main amd65 ... ... Temporary failure resolvig deb.debian.org"
Probably because the VM gets an 169.254.0.2 apipa ip. Both setting up an static IP or DHCP is giving me the same issue. -
Can you try on
latestrelease channel? -
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
Can you try on
latestrelease channel?Samething, again apipa ip.
Trying to login on the machine, is it the admin : admin?
-
On console I am getting "Failed to start Execute cloud user/final scripts."
suddenly it has an ip address, but the installation has failed.
-
There's no default user/pass for obvious security reasons. Are you sure the VMs are deployed in the right network where you have a DHCP server (for example)?
Also pinging @shinuza
-
I've been using this with NVMe on 3 Dell 7920 boxen with PCI passthru.
https://github.com/piraeusdatastore/piraeus-operator
It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.
Just one note on handing devices to the operator - I use raw NVMe disk.
There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.
-
@Theoi-Meteoroi said in How to kubernetes on xcp-ng (csi?):
I've been using this with NVMe on 3 Dell 7920 boxen with PCI passthru.
https://github.com/piraeusdatastore/piraeus-operator
It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.
Just one note on handing devices to the operator - I use raw NVMe disk.
There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.
Did you use the built-in Recipes to create the kubernetes cluster? I tried NVMe, iCSCI, SSD, NFS Share. All the same thing.
-
You have a network issue (well, a DNS one) inside your VM, are you using the right network?
-
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
You have a network issue (well, a DNS one) inside your VM, are you using the right network?
Feel so dumb. When creating a VM, usually the top Network is the correct one. For Kubernetes, I had to scroll all the way down and select the correct network.
-
Is it working correctly now?
-
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
Is it working correctly now?
Currently it is stuck on this.
[FAILED] Failed to start Execute cloud user/final scripts.
cp-1 login:
I did not specify login credentials.
-
Still a network problem (Cloud init can't reach something, no route to host)
-
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
Still a network problem (Cloud init can't reach something, no route to host)
This is the same error I used to get on my homelab when manually installing Ubuntu and trying to deploy k3s with rancheros and longhorn.
This try is on our datacenter and not my homelab. I'll do another setup with DHCP.
-
@olivierlambert Nope, still the same error.
I don't think our Sonicwall (in the datacenter) is blocking anything. Since at home, I am using Unifi. -
fwiw, i'm about to set up a kubernetes cluster on xcp-ng. i'm still in the process, but i'm planning on just passing disks as hba storage to worker/storage nodes, then using openebs jiva (or maybe rook/ceph)
if anybody is interested in how that goes, i can post about it later on
-
Adding @shinuza also in the convo
-
@mohammadm said in How to kubernetes on xcp-ng (csi?):
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
Is it working correctly now?
Currently it is stuck on this.
[FAILED] Failed to start Execute cloud user/final scripts.
cp-1 login:
I did not specify login credentials.
The error is probably earlier than that. You can see the full output of the cloudinit script in /var/log/cloud-init-output.log
The error you are seeing is most likely due to the fact the kubernetes cluster could not be initialized. (kubeadm init failed) -
@xyhhx said in How to kubernetes on xcp-ng (csi?):
fwiw, i'm about to set up a kubernetes cluster on xcp-ng. i'm still in the process, but i'm planning on just passing disks as hba storage to worker/storage nodes, then using openebs jiva (or maybe rook/ceph)
if anybody is interested in how that goes, i can post about it later on
Jiva is probably not the way to go, I believe the supported way is to use Mayastor (via nvme-tcp)
-
@shinuza oh, yeah. they've deprecated it in the last couple of months. thanks for the heads up
