Racked today, entire hosting solution based on Vates stack

nikade

@Pilow That's a very nice setup, cool to see some real enterprise hardware for once in this thread
Also, thanks for informing me about Reunion Island, never heard of the place before and had to look it up on google. How many datacenters can you choose from on this island? Whats the connectivity like, is it very expensive with fiber and ip-transit?

Pilow

@nikade there are many local datacenter operators (ZEOP/OMEGA1/SFR/IDOM/CANAL+/FREE)

I chose SFR because they have connectivity also upto Mayotte Island (look it up too ) where we have clients that will profit our hosting solution on Reunion Island.

Many submarine cables reach us (oldest one is the SAFE : South Africa - Far East to Asia) and some new submarine cables to Africa.

Fiber connectivity exists, not cheap
for the x2paths 10Gb between the nodes you can count 3K€/month (no internet, just data)
100Mb symmetric internet connectivity from datacenter, with good SLAs, 500€/m

Real challenge to be in the middle of an ocean.

nikade

@Pilow Cool, you have now taught me something new, again
I find it very interesting that there is hosting business on these islands, I kind of expected everyone to use the cloud since it would be expensive to establish a datacenter precense on those islands.

Prices do seem expensive, im in Sweden and we have a lot of fiber and ip-transits here.
We pay about €300 per month for 10G CWDM between our datacenters, €550 for redundant (2 paths). Distance is about 10-20km.

IP-transit depends on the provider, we have 3 different ones, and we have different "deals" from each one of them. We mostly do 1G with 100-200Mbit/s but from our main provider which is 10G with a 1G traffic commit we pay about €375 per month. This price is mainly because I know one of the guys who works there + we're a big customer of theirs.

For comparison we pay about €300 for the other 1G with 100Mbit traffic commit...

Pilow

@nikade so our 10G WDM is ten times your price (but redundancy included :')

check here for a cool map
https://www.submarinecablemap.com/

olivierlambert

Yes but at least you have plenty of sun to enjoy during the whole year. Even in middle of winter, you can still eat lunch outside at 20°C

Pilow

@olivierlambert and hurricanes saeson from November to march

ha, we have an active volcano on the island too

olivierlambert

Obviously, you need some fun otherwise it would be boring

nikade

@olivierlambert said in Racked today, entire hosting solution based on Vates stack:

Obviously, you need some fun otherwise it would be boring

Haha yeah "fun"

nikade

@Pilow well you seem to have a nice setup, I am looking forward to more pictures!
I wish I could share ours, but im not allowed to

Pilow

@nikade will share some automation screenshots of our current developments as soon as they are proofed
we're building on top of APIs, all custom settings

nikade

@Pilow sounds good, i'll follow this thread!

Pilow

@nikade still early dev but here is what is actually working

defining a tenant VLAN, ip subnet is calculated with vlan
creating interfaces & VIP carp in netgate 8300 max cluster
creating bandwith limiters & default rules for the tenant
adding vlan to the switches clusters
creating the XCP networks on production pool in xoa
pushing documentation of new tenant to netbox

4 clicks !

work in progress : dhcp server/openvpn server by tenant, outbound nat dedicated IP of available pool per tenant

and XO like interface for resellers to manage their clients (this is a global admin view, all internal, clients, and resellers available)
pushing VMs in their reserved vlan
start/stopping vms
view only on their backup logs (not possible with XOA ACLs/self service resources without being an admin)
reseller can manage its own tenant and its clients tenants, firewall rules are made so that the reseller can access all its client tenants (if he wants to put up its own monitoring for exemple, or mutualised services for its clients)

work in progress : replicate XOA self-service like options, but with custom granularity. VMs deployement with pulumi is quite finished, need to better manage the available templates to each client/reseller
their will be a global admin view for us, reseller view for reseller tenant+its clients tenants, and client view on its own tenant

spinning up a tenant with zero-to-ping in less than 5 minutes is the goal !

nikade

@Pilow thats very impressive, you've done some really great work here.
I like the "private cloud" approach where you can have re-sellers under their own umbrella, but on your infrastructure.

Do you already have customers on another platform or why did you decide to make this big investment with time, hardware, colocation, fiber and all that is needed for this kind of project?

Pilow

@nikade we had a similar setup on VMWARE solution, OVH bare metal hosted in France.

but you know. broadcom

main company is Toolbox, we decided to migrate onprem and cloud clients to full Vates locally hosted on the island this time, and separated the hosting in Cloudbox, a sister company of Toolbox.

many clients do not want to be hosted externaly of the island because of the latency. 250ms to 10ms is quite an upgrade for some situations.

and disaster recovery for 10Tb of vm infrastructure from OVH to Reunion gets you a high RTO, many clients had their external backups on our OVH servers. from days to hours now if needed.

nikade

@Pilow yeah I know, we're a vmware customer as well and the pricing increase was pretty rough.

I can imagine this is a big upgrade for your company, and it seems like you already have the customers so this will probably be a very nice upgrade for them.
I can also guess, that you were able to completely rebuild everything now exactly how you wanted it, which is also a big plus.