Update in case someone comes across this posting looking for the same answer. I needed a solution sooner than later and am so far thrilled with truenas core, a free solution if you've got a box to spare. It has several providers which can be used with kubernetes.

Moved my large drives from my two xcp-ng servers over to truenas.

Working solution: Kubernetes nodes as vms on xcp-ng, Kubernetes storage provisioner hosted via truenas core.

@olivierlambert ok, I can see about putting something together. First though, going to wait a bit with my fingers crossed to see if someone might already have something.

@olivierlambert here's a page at the kubernetes site talking about storage providers: https://kubernetes.io/docs/concepts/storage/storage-classes/

Kubernetes has a few built in but they can also be external providers, which I've used for nfs and cifs. I believe kubernetes speaks csi to interact with the providers.

There may already be something... in my first post I mentioned someone created one for xcp-ng a couple years ago. Tried it out and it no longer works. Hoping it may have evolved and become more integrated somehow... and someone might know who sees this post.

Getting something put together and added to kubernetes as a built in solution would be nice... could draw in business as xcp-ng would shine as a platform for kubernetes... but I'm ok with using a solution I have to install.

@olivierlambert Allow me to share a quick story to help build a little momentum. A few years ago I built a few docker containers, one to add testing to a pipeline running robot framework then exiting... not much came of it, I found that running a couple containers that both used the same port immediately created some complexity. -- Few years later interviewed for a position that ended up being kubernetes related. The lead was absolutely brilliant. I was inspired and began looking into the technology. I've always self taught myself everything but this time, after a month, I took an udemy class. It was like back in college again having discovered linux and a whole new world to explore. I reinstalled my homelab of hyperv and 33 vms using kubernetes and I ended up using only half of my 128 gb of memory. Then I learned of helm and reinstalled everything again, then learned of argocd and reinstalled everything again. Now my whole configuration is in git and is reflected in my clusters automatically. The magic of kubernetes means I can stand up WordPress, the load balancer ip is registered in dns automatically, certificates are generated & maintained automatically and ... the storage needed is requested and setup automatically.

The dns is setup using a kubernetes add on called external dns, the certificates are setup using a kubernetes add on called cert manager, and the storage is setup well ... there is a storage class that defines a provisioner. It's the provisioner I need. On vcenter I can ask for a pvc (persistent volume) and the provisioner sets it up for me using the datastore configured in the vSAN. Kubernetes let's this happen using an api csi, cloud storage interface. The interface let's you setup a pvc, expand the pvc, delete the pvc etc ...

Just imagine never having to setup a cert again .... and the ability to upgrade an app by changing a version number... and it starting up the new version in parallel and then redirecting to the new instance automatically. To be able to change the replica count and have it spin up additional servers, to change the replica count based on cpu usage if you want, to have built in health checks that check a website is up and restart if it crashes, to restart the main process if it crashes automatically.

The people working with kubernetes know it will become the norm as its a better way to do many standard it operations... but I get that xcp-ng isn't responsible. Also, there are other container management systems besides kubernetes, it's just the current lead.

But, as folks make the move to kubernetes they'll be checking for a csi solution to use the datastore configured in the infrastructure. I'm loving xcp-ng and looking for a solution. An alternative could be to setup a truenas that I use for my pvcs instead of local storage, but local storage would have to be faster.

Right now I'm assigning virtual disks to worker nodes for storage but that ends up with idle unused space. Better to have a resource you can ask for some storage and give it back when you are done.

I've manually set up a few clusters on my xcp-ng server using Longhorn for my storage because I couldn't find a working storageclass which worked with xcp-ng.

Is there a csi way to do it? Arturo's solution doesn't seem to work anymore, and it has a few notes saying he moved to XenOrchestra.

I've installed XenOrchestra into one of my clusters using a docker image. Not sure what could be used there. Not sure if the container has all the features.

I'm building another cluster by hand, I'd really prefer to use a csi that can access the xcp-ng datastore.