XOA receipe not creating VIP address (balancer)
-
Hi,
I have tried to create k8s (v 1.30) cluster several times using Hub-Receipes automation in XOA, and if I chose only one control node, than configuration completes successfully and I can use that cluster.But if I choose scenario with fault tolerant control planes, than installation is failing and never completes, only first control plane is being created (only one VM for k8s is created), and in VM logs I can see that control node is trying to connect to VIP address (balancer) which does not exist. VM for VIP/balancer is never being created automatically.
Did I miss something? Should VIP (balancer) be automatically created/configured, or do I need to create it first manually?
I was trying to find more documentation on this subject but I was unlucky in finding it.XOA is on version : 5.93.1 - XOA build: 20240401
XCP-NG is on 8.3 beta 2Can you please advise how to proceed if I want to have fault tolerant k8s cluster?
Thank you in advance and best regards, Igor -
-
@igorf said in XOA receipe not creating VIP address (balancer):
Hi,
I have tried to create k8s (v 1.30) cluster several times using Hub-Receipes automation in XOA, and if I chose only one control node, than configuration completes successfully and I can use that cluster.But if I choose scenario with fault tolerant control planes, than installation is failing and never completes, only first control plane is being created (only one VM for k8s is created), and in VM logs I can see that control node is trying to connect to VIP address (balancer) which does not exist. VM for VIP/balancer is never being created automatically.
Did I miss something? Should VIP (balancer) be automatically created/configured, or do I need to create it first manually?
I was trying to find more documentation on this subject but I was unlucky in finding it.XOA is on version : 5.93.1 - XOA build: 20240401
XCP-NG is on 8.3 beta 2Can you please advise how to proceed if I want to have fault tolerant k8s cluster?
Thank you in advance and best regards, IgorHello,
Can you please send the output of
/var/log/cloud-init-output.log?With regards
-
@shinuza
Sorry, that environment don’t exist anymore.
BR, Igor -
Hi,
Long time user (xcp-ng) first time commentor. I've attempted to use the recipe to deploy k8s, having thought about having a cluster for a while but never had the motivation to look into creating one so the functionality of the recipe sounded awesome. Unfortunately I've hit the same problem by the sounds of it. I can create a single control plane node with workers but when attempting to deploy a more resilient configuration it stops at one node and the screen output reports that cloud-init failed and the logs report it's an issue connecting to the vip by the looks of it. Hopefully it's ok to upload my log in place of igorf's but looking at it it talks about checking the various containers so I did for the vip container and get:
root@cp-1:/home/debian# crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs 8f33bda832123
time="2024-07-16T09:16:08Z" level=info msg="Starting kube-vip.io [v0.8.1]"
time="2024-07-16T09:16:08Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2024-07-16T09:16:08Z" level=info msg="prometheus HTTP server started"
time="2024-07-16T09:16:08Z" level=info msg="Using node name [cp-1]"
time="2024-07-16T09:16:08Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2024-07-16T09:16:08Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [cp-1]"
I0716 09:16:08.494929 1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2024-07-16T09:16:08Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [cp-1]"
I0716 09:16:08.496428 1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-cp-lock...
E0716 09:16:10.511560 1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-svcs-lock: leases.coordination.k8s.io "plndr-svcs-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
E0716 09:16:10.511638 1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: leases.coordination.k8s.io "plndr-cp-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
........( message loops )....which, although I haven't really touched the stuff so can't be sure looks like it could possibly be to do with https://github.com/kube-vip/kube-vip/issues/684
-
That's more input for @shinuza , thanks !
-
@tuckertt said in XOA receipe not creating VIP address (balancer):
Hi,
Long time user (xcp-ng) first time commentor. I've attempted to use the recipe to deploy k8s, having thought about having a cluster for a while but never had the motivation to look into creating one so the functionality of the recipe sounded awesome. Unfortunately I've hit the same problem by the sounds of it. I can create a single control plane node with workers but when attempting to deploy a more resilient configuration it stops at one node and the screen output reports that cloud-init failed and the logs report it's an issue connecting to the vip by the looks of it. Hopefully it's ok to upload my log in place of igorf's but looking at it it talks about checking the various containers so I did for the vip container and get:
root@cp-1:/home/debian# crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs 8f33bda832123
time="2024-07-16T09:16:08Z" level=info msg="Starting kube-vip.io [v0.8.1]"
time="2024-07-16T09:16:08Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2024-07-16T09:16:08Z" level=info msg="prometheus HTTP server started"
time="2024-07-16T09:16:08Z" level=info msg="Using node name [cp-1]"
time="2024-07-16T09:16:08Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2024-07-16T09:16:08Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [cp-1]"
I0716 09:16:08.494929 1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2024-07-16T09:16:08Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [cp-1]"
I0716 09:16:08.496428 1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-cp-lock...
E0716 09:16:10.511560 1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-svcs-lock: leases.coordination.k8s.io "plndr-svcs-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
E0716 09:16:10.511638 1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: leases.coordination.k8s.io "plndr-cp-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
........( message loops )....which, although I haven't really touched the stuff so can't be sure looks like it could possibly be to do with https://github.com/kube-vip/kube-vip/issues/684
Hi,
Thank you for the report.
Can you tell us the version of your xoa-server plug-in? This is fixed in 0.29.1, you probably have version 0.29.0 or lower.With regards,
-
@shinuza said in XOA receipe not creating VIP address (balancer):
xoa-server plug-in
Thanks for that. Yup the plugin reports being at 0.29.0, although XOA reports being up to date too
-
Are you on XOA stable or latest?
edit: it's possible you are on stable, consider switching to latest and try again
-
@olivierlambert stable, which would probably explain it
-
Keep us posted on
latest, to confirm it's correctly fixed -
I can confirm that on latest, using XOA plugin v0.29.1 that all options of control planes 1 -> 7 deploys using DHCP and also tested a three control plane instance using static IP's. Pretty cool
kubectl get nodes
NAME STATUS ROLES AGE VERSION
cp-1 Ready control-plane 12m v1.30.2
cp-2 Ready control-plane 9m15s v1.30.2
cp-3 Ready control-plane 6m11s v1.30.2
worker-1 Ready <none> 3m30s v1.30.2
worker-2 Ready <none> 3m16s v1.30.2
worker-3 Ready <none> 3m23s v1.30.2 -
Yay!
