Dynamically Pool Selection criteria
-
Hi,
I'm not sure to get it. Let me try to rephrase: you want to have a way to automatically decide where you'll create a VM (which pool) depending on some specific requirements?
-
@olivierlambert yes
-
It sounds to be something "on top" of XO or using XO API, because there's no universal rules on your own requirements. If you have some examples, that could be interesting though.
-
You are correct that there are no universal rules, and the decision logic should be implemented on top of XO or by using the XO API. My goal is to automatically decide which pool to use for VM creation based on available resources.
For this, I need to consider parameters such as?
-
Resources based is just one dimension. How about the geography and the latency of your multiple datacenters? Bandwidth available? It goes far beyond capacity.
But for capacity only, I would probably fetch the data per pool on the API and decide from this data.
-
IP address availability is also a criteria, but OK, it can be considered a "resource" like CPU, RAM and storage. There is also network connectivity issue. If you are using some private addressing, either any private subnet has to be addressable/connectable from any part of your whole infrastructure or you have to group your VMs per IP subnet availability (regions?) so that VMs belonging to the same logical group can have network connectivity between themselves.
Just out of curiosity, is this a hypothetical question or do you really have this huge infrastructure? Multiple pools with 64 hosts sounds like a considerable amount of resources. My experience is that if you go above 20 hosts per pool, you get all kinds of scalability issues.
The issues described here usually require some cloud platform deployed on top of XO/XCP-ng if you want to manage infrastructure as large as this and provide full flexibility to you users.
-
@bvitnik we are using VLAN based networking, and it's a big cidr block. also we have handle the case to check available of ip-space in the vlan. like when we assign an ip to the vm via cloud-init at that time we are keeping the record in our side a count. Assume if that VLAN gets a defined threadhold then we assign a new VLAN of PIF and it's handled programmatically to pick a new vlan. Also to clear network part our network connectivity gets properly tested before using it in the production env.
yes we have a large infra deployed. we are planning to expand it. We don't want to use any 3rd party provider that's why we are natively depending on the XO apis.
-
@olivierlambert got it, we have to think about it. How anycast will help us to achieve geography based decisions? or glbs or anything based on dns level magic?
-
@irtaza9 You have VLANs stretched across your whole infrastructure? That means your broadcast domain is your whole infra
? That's a bit risky. A broadcast storm can bring your whole infra down instead of being contained in a single DC for example. -
@bvitnik we provisioned VLANS with respect to DCs, like VLAN5 in KPK-DC1, VLAN5 in RWP-DC4 and these are private in its nature so we do firewall to route the traffic to the 0.0.0.0. I will be happy to hear more from your side to have better network architecture for this kind of work.
-
@irtaza9 OK so you have VLANs per DC. That means VLAN5 in KPK-DC1 and VLAN5 in RWP-DC4 are unrelated and machines in those VLANs cannot communicate directly without going trough some router? This goes back to my original answer. You have to take into account network connectivity when deciding what DC/pool to select for VM provisioning. If you have a few VMs in VLAN5 in KPK-DC1 and you decide to provision some more VMs at latter time that require direct connectivity with existing VMs, you have no choice but to provision them in VLAN5 in KPK-DC1.
Regarding VLANs in general, this is all fine except you can scale it up to a few thousand of VLANs per DC so it will impose a hard limit on the number of isolated network segments (i.e. users/customers). Some weapon of choice among most cloud providers is to use some kind of network overlay like VX-LANs, GRE tunnels and such but this has issues of it's own. For a start, these kinds of networks are software defined in nature (SDN), require an orchestrator and network nodes that run in software. You can leverage some hardware acceleration here and there but it will still be software oriented and have scaling and latency issues. There is no silver bullet and most providers hide their network implementation details.
All in all, a good, scalable and fast networking i probably the toughest thing to implement in any large scale cloud provider... and I'm definitely not a network engineer so I can't say much about it
.
