Racked today, entire hosting solution based on Vates stack
-
Hey all,
We are proud of our new setup, full XCPng hosting solution we racked in a datacenter today.
This is the production node, tomorrow i'll post the replica node !XCPng 8.3, HPE hardware obviously, and we are preparing full automation of clients by API (from switch vlans to firewall public IP, and automatic VM deployment).
This needs a sticker "Vates Inside"
#vent -
Nice setup!
Feel free to share more details if you can, thanks for posting in any case
-
@olivierlambert here is the replica node, on a 10km distant datacenter (two ways 10Gb network between the two nodes)
smaller setup here, just to host backup copies of production node and replica/DR VMs.the whole setup consist of 7 VMS Enterprise hosts
Netgate 8300 max firewalls top of rack.we are a MSP (and Vates partner
Tier2 soon to be Tier4 !) providing full hosting or hybrid onprem/oncloud to our clients.
some other services like web hosting with Plesk platform, Veeam Cloud Connect, security/firewalling services, centralised monitoring with Centreon and as soon as we manage to connect onprem XCP to our cloud xoproxies, full replica solution of onprem XCP servers.on the bare metal, XCP 8.3, value added with full automation of tenant creation/administration/documentation with diverse APIs (check CRM for clients tagged to be admin of their tenant, get VLANs from there, create them in XCP pools/switchs/pfsense firewalls, create firewall rules and limiters, create openvpn server, spin up VMs in client tenant, automatic netbox documentation on top of xoa plugin, enjoy !)
we are on pre-production and should be on the market in november, currently migrating OVH VMs to these servers.
VATES stack is the best solution to be fully integrated in our vision of providing VMs and services to clients in an efficient way.
we left vmware, had ESXs hosted in OVH datacenters in France, but they were 10 000km and 250ms away from our end users.
for those who wonder, we are located at Reunion Island, indian ocean, french overseas territory.
-
Nice! We also have a pretty big partner in your island. Glad to see XCP-ng and the Vates stack is booming in there
(I've been visiting last March, for an entire week, it was wonderful) -
@Pilow nice! what are the individuell server specs and storage? please share more
-
@Henrik 1Tb RAM per host on production and replication, storage is 25Gb fiber channel, fully multipathed iSCSI (so yeah... thick provisionning lvmoiscsi...
SANs are thin storage backend, its reliable and redundant but lvmoiscsi is storage hungry, need to have good monitoring !).one host on each node have local raid5 SSD storage, where we put our own management&automation vms, clients are on shared storage
S3 minios on iSCSI as remotes, cross backuped between the two nodes with xoproxies on each end.
designed to be fully resilient with the less SPOF inside
-
@Pilow That's a very nice setup, cool to see some real enterprise hardware for once in this thread
Also, thanks for informing me about Reunion Island, never heard of the place before and had to look it up on google. How many datacenters can you choose from on this island? Whats the connectivity like, is it very expensive with fiber and ip-transit? -
@nikade there are many local datacenter operators (ZEOP/OMEGA1/SFR/IDOM/CANAL+/FREE)
I chose SFR because they have connectivity also upto Mayotte Island (look it up too
) where we have clients that will profit our hosting solution on Reunion Island.Many submarine cables reach us (oldest one is the SAFE : South Africa - Far East to Asia) and some new submarine cables to Africa.
Fiber connectivity exists, not cheap
for the x2paths 10Gb between the nodes you can count 3K€/month (no internet, just data)
100Mb symmetric internet connectivity from datacenter, with good SLAs, 500€/mReal challenge to be in the middle of an ocean.
-
@Pilow Cool, you have now taught me something new, again
I find it very interesting that there is hosting business on these islands, I kind of expected everyone to use the cloud since it would be expensive to establish a datacenter precense on those islands.Prices do seem expensive, im in Sweden and we have a lot of fiber and ip-transits here.
We pay about €300 per month for 10G CWDM between our datacenters, €550 for redundant (2 paths). Distance is about 10-20km.IP-transit depends on the provider, we have 3 different ones, and we have different "deals" from each one of them. We mostly do 1G with 100-200Mbit/s but from our main provider which is 10G with a 1G traffic commit we pay about €375 per month. This price is mainly because I know one of the guys who works there + we're a big customer of theirs.
For comparison we pay about €300 for the other 1G with 100Mbit traffic commit...
-
@nikade so our 10G WDM is ten times your price (but redundancy included :')
check here for a cool map
https://www.submarinecablemap.com/ -
Yes but at least you have plenty of sun to enjoy during the whole year. Even in middle of winter, you can still eat lunch outside at 20°C
-
@olivierlambert and hurricanes saeson from November to march
ha, we have an active volcano on the island too
-
Obviously, you need some fun otherwise it would be boring
-
@olivierlambert said in Racked today, entire hosting solution based on Vates stack:
Obviously, you need some fun otherwise it would be boring
Haha yeah "fun"
-
@Pilow well you seem to have a nice setup, I am looking forward to more pictures!
I wish I could share ours, but im not allowed to -
@nikade will share some automation screenshots of our current developments as soon as they are proofed
we're building on top of APIs, all custom settings -
@Pilow sounds good, i'll follow this thread!
-
@nikade still early dev but here is what is actually working
- defining a tenant VLAN, ip subnet is calculated with vlan
- creating interfaces & VIP carp in netgate 8300 max cluster
- creating bandwith limiters & default rules for the tenant
- adding vlan to the switches clusters
- creating the XCP networks on production pool in xoa
- pushing documentation of new tenant to netbox
4 clicks !
work in progress : dhcp server/openvpn server by tenant, outbound nat dedicated IP of available pool per tenant
and XO like interface for resellers to manage their clients (this is a global admin view, all internal, clients, and resellers available)
pushing VMs in their reserved vlan
start/stopping vms
view only on their backup logs (not possible with XOA ACLs/self service resources without being an admin)
reseller can manage its own tenant and its clients tenants, firewall rules are made so that the reseller can access all its client tenants (if he wants to put up its own monitoring for exemple, or mutualised services for its clients)work in progress : replicate XOA self-service like options, but with custom granularity. VMs deployement with pulumi is quite finished, need to better manage the available templates to each client/reseller
their will be a global admin view for us, reseller view for reseller tenant+its clients tenants, and client view on its own tenantspinning up a tenant with zero-to-ping in less than 5 minutes is the goal !
-
@Pilow thats very impressive, you've done some really great work here.
I like the "private cloud" approach where you can have re-sellers under their own umbrella, but on your infrastructure.Do you already have customers on another platform or why did you decide to make this big investment with time, hardware, colocation, fiber and all that is needed for this kind of project?
-
@nikade we had a similar setup on VMWARE solution, OVH bare metal hosted in France.
but you know. broadcom
main company is Toolbox, we decided to migrate onprem and cloud clients to full Vates locally hosted on the island this time, and separated the hosting in Cloudbox, a sister company of Toolbox.
many clients do not want to be hosted externaly of the island because of the latency. 250ms to 10ms is quite an upgrade for some situations.
and disaster recovery for 10Tb of vm infrastructure from OVH to Reunion gets you a high RTO, many clients had their external backups on our OVH servers. from days to hours now if needed.
