Kubernetes autoscaler xen api/XO api
-
Looking at Kubernetes these days, I guess we have to build a Cloud Controller Manager aimed at the XEN api.
This can be a standalone component which should be deployed to your Kubernetes cluster.
https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/
R.
-
We could build that if it makes sense. I have to admit I lack experience on k8s, so IDK what's needed from that Cloud Controller.
-
@olivierlambert that would be great. I'm willing to contribute on this. While I wouldn't call myself an expert on Kubernetes development, I do know a thing or two.
The link that @yctn mentioned is the old approach, with in-tree cloud provider implementations. For completeness, in-tree means that all the cloud provider implementations were part of the main git code tree. In the old approach, a cloud provider took care of nodes (VMs), network and storage.
As you can expect, the approach today is to have out-of-tree cloud provider implementations, residing in separate Git repositories. In this approach, the storage aspect has been separated and you have two different integrations to write:
- Cloud Provider Interface (CPI): lets Kubernetes manage worker nodes and handles the network interfaces to the VMs.
- Container Storage Interface (CSI): lets Kubernetes manage Persistent Volumes (PVs) which maps on underlying storage technology.
As an example, here are the implementations of the above 2 components for vSphere:
For XCP-ng, there is a CSI driver started by Arturro Guerra:
https://github.com/ArturoGuerra/xcpng-csi
It probably needs a refresher and some more testing but I guess we don't have to start from scratch here.
The cloud provider implementation for XCP-ng needs to be started. If you could create a repository named
cloud-provider-xcp-ng(or similar) on your Github organization, initialized with a README and a LICENSE file (preferrably Apache License 2.0), then I can clone it and provide the initial code.Ringo
-
Hi @ringods
Thanks for your feedback. To be fair, I would prefer a Xen Orchestra provider, to provide a better abstraction, as XO is becoming a real middleware for various things nowadays (and overcome some XAPI limitations). That's also why the Vmware provider is running on top of vSphere, not ESXi directly.
However, I'm not decided yet, this will require some thoughts internally here, between XCP-ng and XO team to discuss more deeply.
I'll come back here in next week with more ideas or questions
-
@olivierlambert I have no problem being it a Kubernetes component talking to Xen Orchestra. But that limits my contribution ability to the free features of XO. I only use XCP-ng for a personal homelab on a single machine setup.
-
Xen Orchestra is fully open source
You can build it with all XO features, it's even explaining in our doc: https://xen-orchestra.com/docs/installation.html#from-the-sourcesedit: do you know where could we have an overview on the kind of "calls" that would be needed to be done from Kubernetes?
-
@ringods maby you can join IRC or Discord if thats possible for you. cause i think there are still a few things that need to be discussed on how it should be implemented. for example. what about DHCP? since XO/xcp-ng to me knowledge has no DHCP component.
what about type loadBalancer call? since xcp-ng does not offer that.in my DIY k8s setup i just made 2 vm's with keepalived and haproxy that are able to send proxy protocol traffic to my haproxy ingress controller in k8s using nodePort.
but DHCP i have not found a sollution. but it would be great if xcp-ng can offer like a managed/ha DHCP sollution for the future.
-
That's why XO is relevant. All you ask can be managed at XO level, adding a simple abstraction layer
That's also why we got a load balanced in XO already BTW. We should probably study how it's done on VMware (as the link given) before taking any decision.
-
@yctn or @olivierlambert, which Discord is this? I don't see it mentioned in the Community section of the website.
-
-
@olivierlambert, coming back to this question:
edit: do you know where could we have an overview on the kind of "calls" that would be needed to be done from Kubernetes?
Every cloud provider is a Kubernetes controller. The basic initialisation creates a structure of type
cloudprovider.Interface. Here you can see the initialisation of the vSphere cloud provider:The Golang interface to implement has the following functions which needs to be implemented:
I hope this helps.
I understand that more investigation is needed, but can we do this work together? I can, and want to, help out to bootstrap the basic project layout, build setup, etc in a new Git repository. A repository can always be renamed if the name is not 100% correct.
Ringo
-
Sure.
- Do you know if it's our responsibility to create the repo or if it's within "upstream" in k8s?
- That will be probably named "cloud-provider-xo" or "cloud-provider-xenorchestra". What do you think?
-
- it would be our responsibility to create the repo and users would likely install it with Helm (from my reading of the cloud-provider-vsphere docs
- I think either is fine. My personal vote would be for the latter (cloud-provider-xenorchestra).
-
Great, let's do this then! On creating the repo, and will invite you Dom. @ringods what's your Github handle?
-
Done. Both of you added!
-
@olivierlambert thanks for that. Are you ok if I configure Github Actions as the CI for this repo?
-
Sure go ahead you should be able to do it in terms of permissions. If not, let me know
