not sure if anyone is still following this this thread, but just in case...

Over the last few months I’ve been working on a CSI driver for Kubernetes that integrates with Xen Orchestra. I’ve been running it in my own setup for a couple of weeks now and it seems to be working well.

I originally built it for my own needs, but I’ve since cleaned it up and added documentation so others can try it out. If you do, I’d love to hear your feedback, issues, or feature requests.

Here is the link:
github.com/m4rCsi/csi-xen-orchestra-driver

Features

• Dynamic provisioning (create disks on demand via PVCs)
• Migration of disks between storage repositories (meant for local SRs)
• Static provisioning (use an existing VDI by UUID)
• Offline volume expansion
• Topology aware (pool, and optionally host) (with the help of xenorchestra-cloud-controller-manager)

i think that's worth doing when the vms aren't being used as kubernetes nodes, but for kubernetes i'd prefer the vms be strictly for kubernetes

talos is purpose-built for this, too, so you get lots of great features, and you can be confident they're secure, and minimal. their support chat is very responsive too, if you're on matrix

ok i'm done shilling talos now

So I can't provide any help with cloud init or hub/recipes.

The working setup I have is built on CentOS7 ( need to replace ) VMs. I tried deploying on RockyOS 9.3 and spun my wheels for too long on issues there so I'm going to try 8.9.

I just updated my XO-CE bits and see some nice new features in the UI that will be helpful with all the PCI passthru I want to do. I was putting add/remove instructions in the VM notes and tags on the VMs just to remind me of what they show below. Its like they read my mind.

I need to learn KubeVirt.

Its christmas in may.

fwiw, i'm about to set up a kubernetes cluster on xcp-ng. i'm still in the process, but i'm planning on just passing disks as hba storage to worker/storage nodes, then using openebs jiva (or maybe rook/ceph)

if anybody is interested in how that goes, i can post about it later on

Jiva is probably not the way to go, I believe the supported way is to use Mayastor (via nvme-tcp)

@olivierlambert said in How to kubernetes on xcp-ng (csi?):

Is it working correctly now?

Currently it is stuck on this.

[FAILED] Failed to start Execute cloud user/final scripts.

cp-1 login:

I did not specify login credentials.

The error is probably earlier than that. You can see the full output of the cloudinit script in /var/log/cloud-init-output.log
The error you are seeing is most likely due to the fact the kubernetes cluster could not be initialized. (kubeadm init failed)

if anybody is interested in how that goes, i can post about it later on

Still a network problem (Cloud init can't reach something, no route to host)

This is the same error I used to get on my homelab when manually installing Ubuntu and trying to deploy k3s with rancheros and longhorn.

This try is on our datacenter and not my homelab. I'll do another setup with DHCP.

Is it working correctly now?

Currently it is stuck on this.

[FAILED] Failed to start Execute cloud user/final scripts.

cp-1 login:

I did not specify login credentials.

You have a network issue (well, a DNS one) inside your VM, are you using the right network?

Feel so dumb. When creating a VM, usually the top Network is the correct one. For Kubernetes, I had to scroll all the way down and select the correct network.

I've been using this with NVMe on 3 Dell 7920 boxen with PCI passthru.

https://github.com/piraeusdatastore/piraeus-operator

It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.

Just one note on handing devices to the operator - I use raw NVMe disk.
There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.

I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.

Did you use the built-in Recipes to create the kubernetes cluster? I tried NVMe, iCSCI, SSD, NFS Share. All the same thing.

https://github.com/piraeusdatastore/piraeus-operator

It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.

Just one note on handing devices to the operator - I use raw NVMe disk.
There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.

I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.