Nas/Home/Lab heterogeneous pools and failure

crembz

I'm new to xcp-ng so please bear with me. I'm in the process of rebuilding my home network and lab and have been having a play with xc-png. I'm coming from over years of running proxmox and kvm before that.

I'm trying to understand whether I can achieve the following using xc-png:

I have a cluster of 7 hosts.

Virtual NAS with PCIE passthrough - threadripper 2950x
One host is for ciritical network infra, fw, dns, vpn, omada- i5 8500t
Virtual ESXi host - 5900x
Virtual Hyper V host - i5 10500t
Virtual Nutanix host - i7 6700
Virtual KVM host - i5 6500t
Docker host - i5 8500

I run everything virtual to easily snapshot and revert states, I'm testing a bunch of automation across hypervisors and public clouds.

Host 1,2,3,7 are connected with 10gb NICs. The others all have 1gb NICs. The NAS hosts VM drives for all but host #2

I'd love to have everything in one giant pool, and turn all bar the critical network host off at night (thus why host #2 I run VMs on local storage) to save on power. I've not found an elegant way to do this on pve. Are there any clever ideas on how I could achieve this on xcp-ng?

Also, I noticed VIFs are setup pool wide and mapped to a PIF. How do you handle hosts with different NIC counts? In the 10g boxes my PIF0 is mapped to 1g ports with is not being used (using the 10g).

NOTE - I've been able to have both AMD and Intel hosts in the same cluster in pve and live migrate guests between them. I don't think having mixed CPU vendors in the same pool works in xcp-ng. My understanding though on xcp-ng is that I can cold migrate between pools which I might be able to live with.

olivierlambert

Hi,

You can't have a pool between AMD and Intel CPUs, because a pool reason for being is to allow live migration between hosts with a shared storage (and also sharing the same configuration, eg NIC numbers) to provide a seamless experience when you add a new host into it (it will do all the configuration for you).

If your machines are having the same number of NICs but various CPU generations, it's OK since the pool will automatically mask the more recent CPU features to allow seamless live migrations.

As you can see, if your hardware is too heterogeneous, it doesn't make sense to pool those machines together. However, note that you CAN live migrate between pools (shared-nothing), but if the CPU on destination is too different or too old vs your current one, the VM will probably crash on destination. AMD vs Intel is guarantee to crash BTW. But for those cases, we have warm migration for example.

crembz

@olivierlambert interesting, I do like how xo manages all hosts in the one place. My understanding though is ha is not possible across clusters right?

I tend to pick up hosts over time, so having a homogenous cluster is practically impossible.

olivierlambert

If you can't have a homogeneous cluster, then so be it, you'll have multiple pools and that's it XO can manage them all anyway.

HA isn't possible across clusters, indeed.