@rfx77 Great information!

Considering we have a full HA Truenas SAN, we are most likely going to use https://github.com/democratic-csi/democratic-csi/blob/master/examples/freenas-nfs.yaml

If we go with a k8s distribution like rke2, is there xcp-ng guest tools support and if so when during the process would we install the tools on the nodes?

We are planning to move from VMs to Kubernetes for our websites. We are currently testing the hub feature in XOA, but need more information about the best way to leverage our current resources for kubernetes.

We have a highly available TrueNAS all flash m40 SAN that we currently use NFS shares hosted on the TrueNAS for storage SR's in xcp-ng.

Is there any guidance on how to configure StorageClasses or LoadBalancer Services for production?
I'm a little lost on storage, and we're looking at an IngressController exposed as a NodePort Service on 80/443 for now.

Also, it would be neat to have an IaC workflow (i.e. ekstcl) instead of a dialog. It might look something like this:

---
apiVersion: k8s.xcp-ng.org/v1alpha1
kind: KubernetesCluster
metadata:
  name: my-cluster
  labels:
    k8s.xcp-ng.org/cluster-name: my-cluster
spec:
  clusterVIP: 10.0.1.10
  gatewayIP: 10.0.1.1
  nameservers:
    - ip: 10.0.1.1
    - ip: 8.8.8.8
  searchDomains:
    - dev
  controlPlane:
    - name: control-plane-1
      nodeIP: 10.0.1.11
      subnetMask: 255.255.255.0
    - name: control-plane-2
      nodeIP: 10.0.1.12
      subnetMask: 255.255.255.0
    - name: control-plane-3
      nodeIP: 10.0.1.13
      subnetMask: 255.255.255.0
    
  nodeGroups:
    bootstrap: # bootstrap/system nodes
      resources:
        cpu: 2
        mem: 4096
      labels:
        k8s.xcp-ng.org/node-group: system
      nodes:
      - name: worker-node-1
        nodeIP: '10.0.1.101/24'
      - name: worker-node-2
        nodeIP: '10.0.1.102/24'
      - name: worker-node-3
        nodeIP: '10.0.100.103/24'

    app_cluster:
      # e.g., a clustered app with well-aligned, isolated resources
      minInstances: 3
      maxInstances: 15
      resources:
        cpu: 4
        mem: 8192
      network:
        name: my-app-network
        addressRange: 10.0.100.32/28
      nodeTemplate:
        labels:
          k8s.xcp-ng.org/node-group: my-clustered-app

    memoryBound:
      # (e.g.) VMs configured to scale with higher memory ratios
      resources:
        cpu: 2
        mem: 16384
      labels:
        k8s.xcp-ng.org/node-group: memory

    computeBound:
      # (e.g.) VMs configured to scale with higher processor limits
      resources:
        cpu: 4
        mem: 8192
      labels:
        k8s.xcp-ng.org/node-group: compute

We are also interested in running a production level kubernetes environment via XOA. Any updates on this?