How to kubernetes on xcp-ng (csi?)
-
I've manually set up a few clusters on my xcp-ng server using Longhorn for my storage because I couldn't find a working storageclass which worked with xcp-ng.
Is there a csi way to do it? Arturo's solution doesn't seem to work anymore, and it has a few notes saying he moved to XenOrchestra.
I've installed XenOrchestra into one of my clusters using a docker image. Not sure what could be used there. Not sure if the container has all the features.
I'm building another cluster by hand, I'd really prefer to use a csi that can access the xcp-ng datastore.
-
Hi,
Sorry @karezza but since there's not a lot of k8s specialists here, we'd need to understand more clearly what do you actually need
-
@olivierlambert Allow me to share a quick story to help build a little momentum. A few years ago I built a few docker containers, one to add testing to a pipeline running robot framework then exiting... not much came of it, I found that running a couple containers that both used the same port immediately created some complexity. -- Few years later interviewed for a position that ended up being kubernetes related. The lead was absolutely brilliant. I was inspired and began looking into the technology. I've always self taught myself everything but this time, after a month, I took an udemy class. It was like back in college again having discovered linux and a whole new world to explore. I reinstalled my homelab of hyperv and 33 vms using kubernetes and I ended up using only half of my 128 gb of memory. Then I learned of helm and reinstalled everything again, then learned of argocd and reinstalled everything again. Now my whole configuration is in git and is reflected in my clusters automatically. The magic of kubernetes means I can stand up WordPress, the load balancer ip is registered in dns automatically, certificates are generated & maintained automatically and ... the storage needed is requested and setup automatically.
The dns is setup using a kubernetes add on called external dns, the certificates are setup using a kubernetes add on called cert manager, and the storage is setup well ... there is a storage class that defines a provisioner. It's the provisioner I need. On vcenter I can ask for a pvc (persistent volume) and the provisioner sets it up for me using the datastore configured in the vSAN. Kubernetes let's this happen using an api csi, cloud storage interface. The interface let's you setup a pvc, expand the pvc, delete the pvc etc ...
Just imagine never having to setup a cert again .... and the ability to upgrade an app by changing a version number... and it starting up the new version in parallel and then redirecting to the new instance automatically. To be able to change the replica count and have it spin up additional servers, to change the replica count based on cpu usage if you want, to have built in health checks that check a website is up and restart if it crashes, to restart the main process if it crashes automatically.
The people working with kubernetes know it will become the norm as its a better way to do many standard it operations... but I get that xcp-ng isn't responsible. Also, there are other container management systems besides kubernetes, it's just the current lead.
But, as folks make the move to kubernetes they'll be checking for a csi solution to use the datastore configured in the infrastructure. I'm loving xcp-ng and looking for a solution. An alternative could be to setup a truenas that I use for my pvcs instead of local storage, but local storage would have to be faster.
Right now I'm assigning virtual disks to worker nodes for storage but that ends up with idle unused space. Better to have a resource you can ask for some storage and give it back when you are done.
-
If you specify us exactly what do you need to access, we can help you to make a CSI provider for XO API. We need to know which "commands" do you need. It seems relatively simple if I understand correctly: create a virtual disk, give its size, then remove it or expand it. Right?
-
@olivierlambert here's a page at the kubernetes site talking about storage providers: https://kubernetes.io/docs/concepts/storage/storage-classes/
Kubernetes has a few built in but they can also be external providers, which I've used for nfs and cifs. I believe kubernetes speaks csi to interact with the providers.
There may already be something... in my first post I mentioned someone created one for xcp-ng a couple years ago. Tried it out and it no longer works. Hoping it may have evolved and become more integrated somehow... and someone might know who sees this post.
Getting something put together and added to kubernetes as a built in solution would be nice... could draw in business as xcp-ng would shine as a platform for kubernetes... but I'm ok with using a solution I have to install.
-
@olivierlambert ok, I can see about putting something together. First though, going to wait a bit with my fingers crossed to see if someone might already have something.
-
To help k8s integration with XCP-ng/XO we already have plans to write a Node driver for Rancher.
But if CSI driver is simple to make, why not. But having a real specification will be more helpful than telling us to learn it in the first place
-
Update in case someone comes across this posting looking for the same answer. I needed a solution sooner than later and am so far thrilled with truenas core, a free solution if you've got a box to spare. It has several providers which can be used with kubernetes.
Moved my large drives from my two xcp-ng servers over to truenas.
Working solution: Kubernetes nodes as vms on xcp-ng, Kubernetes storage provisioner hosted via truenas core.
-
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
To help k8s integration with XCP-ng/XO we already have plans to write a Node driver for Rancher.
OMG that would be amazing!
-
Stay tuned but work started
-
@olivierlambert Do you have any news regarding xo CSI storage provider for kubernetes?
-
Not yet sadly, the persons tasked to do it aren't giving any news since a while. Internally, we are working on our Project Pyrgos to help deploy easily k8s clusters.
-
@olivierlambert Thanks for the prompt reply! Fingers crossed for the Project Pyrgos!
-
@olivierlambert any update to this and the pyrgos project?
-
Yes, June's release came with new features, like selecting the Kubernetes version. We got multi control planes before, static IPs and such.
Take a look at our blog posts to see what's going on, there's not a new feature per month (yet) but it's moving forward
-
@olivierlambert ok seen there was another blog post about it here https://xen-orchestra.com/blog/xen-orchestra-5-84/
ok cool, thanks
-
Next steps take a bit more time because it's about storing the cluster key safely to be able (then) to use XO to make basic queries on it (like current version and such), which is the first step to prepare the automated node upgrade/replacement
-
Trying to build a cluster from the hub, bit it is giving me "Err: http://deb.debian.org/debian bullseye/main amd65 ... ... Temporary failure resolvig deb.debian.org"
Probably because the VM gets an 169.254.0.2 apipa ip. Both setting up an static IP or DHCP is giving me the same issue. -
Can you try on
latestrelease channel? -
@olivierlambert said in How to kubernetes on xcp-ng (csi?):
Can you try on
latestrelease channel?Samething, again apipa ip.
Trying to login on the machine, is it the admin : admin?
