@stormi Nice work on the PCI passthru. Very helpful. I like the treeview as well. Thanks!

I forgot to mention - he also talks about Layer 3 features (BGP) and that he has to configure on his switch in the case of XCP. Most folks don't really have an L3 router beyond their firewall appliance. Not needed. And you can't configure BGP there. Boundary Gateway Protocol is for a more serious exposure to the internet and involves interacting with ASN routing tables. Lets just say beyond the scope of the forum. Most folks operate on a L2 network of some sort and don't configure VLANs. There are many exceptions here, I assume but they would tend to be pretty technical already. This stuff is fire and liability to advise about.

I see lots of questions here that probably would help with some simple configuration drawings to show a typical small network translated ( NAT ) environment for newbies. There are other tricks like router on a stick and such that mimick IP telephone connections to be able to use VLANs.

@olivierlambert He seems to be referring to some VMware dynamic networking I think. I looked at Tanzu a couple of years ago and see bits of it there. I try to forget about it because its a terribly expensive k8s platform and breaks the spirit of why I like k8s.

@xyhhx As I look further, it does seem to check a lot of boxes if I was building out a prod instance. I have enough exposure to dev-sec that I see the value. I feel like I sound like Linus and his comments about Debian but learning new ecosystems takes time and energy and sometimes is just distraction from building something. I kinda wore myself out learning all the details and quirks of DRBD and Linstor recently so I'll read some of the docs and hope the toolchain doesn't chase me away. Any product or project that genuinely and openly has a community like we have here would usually have my attention, just on those terms.

@xyhhx Why thanks! I'll take a look at it. I've done coreos and flatcar for work but find immutable OS no fun at home. I'm used to doing all the admin already and most of my VMs live long enough to be classed as "pets".

You lost quorum.

I would start looking at DRBD - that is the underlying part that isn't working at the moment. When I deployed this I wanted to understand the parts. Key to the Linstor layer - drbd stores the cluster state and membership.

I'd advise reading the DRBD docs as well as the Linstor docs to find the commands you need to stand this back up. I really don't advise using anything spinning for disk. SSD and NVMe is the ticket. You can make rust work but its terminally slow. I found 3TB disk was ok ( ~60MB/sec ) but 9.1 (10 ) TB were just awful at with 20-40MB/sec the best I saw. I removed all the XOSTOR stuff this week to maybe reinstall on some 4TB NVMe.

The upside of all that time learning drbd and linstor was helpful when I decided to use the Piraeus operator at the kubernetes level. Its basically all the same bits built from source on the nodes you deploy on and includes a CSI driver.

I should mention I use static IP for my VMs and I don't deploy using the hub recipes because my home setup is all Community bits and 8.3 XCP, updated regularly.
I use kubespray. I'm at kubernetes 1.27.7

So I can't provide any help with cloud init or hub/recipes.

The working setup I have is built on CentOS7 ( need to replace ) VMs. I tried deploying on RockyOS 9.3 and spun my wheels for too long on issues there so I'm going to try 8.9.

I just updated my XO-CE bits and see some nice new features in the UI that will be helpful with all the PCI passthru I want to do. I was putting add/remove instructions in the VM notes and tags on the VMs just to remind me of what they show below. Its like they read my mind.

I need to learn KubeVirt.

Its christmas in may.

I've been using this with NVMe on 3 Dell 7920 boxen with PCI passthru.

https://github.com/piraeusdatastore/piraeus-operator

It worked well enough that I installed the rest of the NVMe slots to have 7TB per node. I pin the master kubernetes nodes each to a physical node, I use 3 so I can roll updates and patches. The masters serve the storage out to containers - so the workers are basically "storage-less". Those worker nodes can move around. All the networking is 10G with 4 interfaces, so I have one specifically as the backend for this.

Just one note on handing devices to the operator - I use raw NVMe disk.
There can't be any partition or PV on the device. I put a PV on, then erase it so the disk is wiped. Then the operator finds the disk usable an initializes. It tries to not use a disk that seems in use already.

I also played a bit with XOSTOR but on spinning rust. Its really robust with the DRBD backend once you get used to working with it. Figuring out object relationships will have you maybe drink more than usual.

I would take a look at the disk-scheduler in use ( lsblk -t ) and change to perhaps deadline. CFQ is kind of a dog and a waste of time with SSD.

Also a MAC user - for Windoze guests - use Remote Desktop app. Then you get a good experience and cut and paste. I only use the guest console to turn on remote access.

The pass-through worked well in my case, attached to a Ubuntu 22.04 VM and the driver loaded ( had to blacklist it because I wanted to use the CUDA toolkit ) fine. The toolkit also installed fine, I kept the OpenGL libs so the GUI still works in XCP. I'd suppose this would also work just fine with Windows. I'll have to test that with a eval ISO.

I looked at desktop systems for some long while. None really do the needful very cheaply. I wanted to do PCI passthru at least and run a few more substantial vms. This meets the requirements and had pretty much everything

Looks like it was never powered on, not a spec of dust inside. Swapped out the supplied windoze disk with another SSD and so far, XCP likes it very much. I am very impressed with what I obtained for ~$1500 with tax.