So I woke up and saw this today:

https://xcp-ng.org/blog/2020/09/09/edge-case-2-protectli-hardware/

And a massive smile came across my face, because this is the solution I have cut my teeth on XCP-NG just over 2 years ago now, and is still reliably running my home and two person cybersecurity consultancy business with 9 VM's running 24/7, and another 3-4 VMs that we spin up whenever we need them.

Two years ago this month, I purchased a new Protectli model, the i5 7200u as a replacement system for my old and finally failed (blown caps) Firewall. While waiting for the shipment to arrive, I saw @olivierlambert give his presentation on forking Xen and the launch of XCP-NG. Already familiar with Xen through my previous employer (T-Mobile), I thought when my Protectli unit arrives, and before I launch it as a bare metal firewall on my network, I thought I'd try XCP-NG on it first. When I did, my mind was blown almost instantaneously for a few reasons.

First, there was a bit of a learning curve. I'd never actually installed / administrated a Xen server at my former employer - that wasn't our job. Specifying security requirements and performing pen-testing and audits of implementations was more our thing. Thanks in part to the members of the community that were here, and even Oliver himself (who graciously extended my XOA demo license & answered further technical questions I had) that learning curve was just weeks rather than months. And after some initial configuration experimentation and performance testing, I ended up launching our XCP-NG Protectli unit with a Firewall (pfSense) VM and our first completely virtualised DMZ. Fuller disclosure: I've run my own Email, Web and DNS servers from home since 1994. But this was the first time I've attempted to virtualise many of my servers running on bare metal. And I was so impressed, within a couple of months - I was virtualising nearly everything onto this box apart from my NAS units onto XCP-NG, including two RaspberryPi's. In fact, at the time, I made many jokes about how consolidating two very heavily loaded RaspberryPi 3B+ Units (sucking a total of 10+ watts) into XCP-NG on Protectli was resulting in electrical bill savings & huge performance gains on the apps that previously ran on those RPI's were now realised.

What was on those two RaspberryPi's? The master node of the first High Availability implementation of Home Assistant which I specifically built for monitoring, managing and automating my Nano Coral Reef and Marine Fish breeding farm, composed of 12 interconnected aquariums. It also controls our house too, but the aquarium aspect is really why I needed 99.7%+ uptime service availability. So I architected a high availability solution using RaspberryPi's that works in practice, not unlike the Space Shuttle or Dragon Crew Flight Systems computer. Each node is sanity checking the other nodes and ensuring it's still online and operating as expected. If any node drops off, freezes, locks up, or operates outside of designed parameters, one of the remaining 4 nodes on the network (each even on their own circuit breaker) will take over. Additionally each RaspberryPi functions as a GPIO / USB / I2C / Dallas 1-wire sensor input/output device - except the master Home Assistant node and a dedicated RPI for the SQL server, which was located in my office. And migrating this master node and related SQL to a VM on XCP-NG was easy-peezy, and required no changes to my High Availability design of Home Assistant. In fact, it brought new features even better assuring more resilient high availability features.

I'm not 100% sure of this, but via the Home Assistant forums, we are under the impression we have been operating one of the largest domestic installations of Home Assistant that exists out there. It controls over 80 electrical sockets, and another 68+ network controlled lights, over 20 different temp probes, two dozen+ electrical pumps (including 8 aquarium dosing pumps) , displays 10 residential security cameras, and streams another 10 public webcams at the local Dutch beaches (which my wife calls "Sunset Cams"), tracks overhead airplanes (as far as the coast of the UK) using ADS-B, tracks the International Space Station & crew members onboard (my wife works for the ISS), weather conditions, and much much more.

This solution also monitors my home-office network, once waking me up at 5am when one of my Cisco network switch's power supply failed. 15 minutes later, I'd had dropped in a cold-standby Cisco switch in it's place and was crawling back into bed. But the real life saver has been with the aquariums, whether it was reacting to overflow situations to Ground Fault Circuit failures (think ageing aquarium heaters that die / leak in the middle of the night) that result in unexpected power cuts (the aquariums have their own dedicated GFCI protected electrical circuits).

And of the 3 years we've had this Home Assistant solution (May 2017), 2 of those years the primary node has been running on XCP-NG on the Protectli hardware.

Originally, we started running XCP-NG on the Protectli with just 16GB of RAM and a 512GB SSD. Today, it's got 32GB of RAM and a 2TB SSD. The Protectli unit is wall mounted in my office and serves as a complete 100% replacement for all our former Cloud experiences, including Apple's iCloud / Office365 / GoogleDocs / WeTransfer / Dropbox / Maps and more. And we couldn't be more happy with this solution - so much so - my business partner and I are looking to launch a Edge based service using a similar combination of tools for the consumer / commercial market.

In March of this year, the Home Assistant Blog officially recognised my build and featured my implementation in the community spotlight.

https://www.home-assistant.io/blog/2020/03/15/community-highlights/

For those who are interested in my Home Assistant portion of the build, you can find full details on this epically long thread (TL;DR) here:
https://community.home-assistant.io/t/going-to-next-level-of-aquarium-automation-whos-with-me/18486
(There is a discrepancy in regards to the total number of aquariums - In Jan/Feb, we started shutting down several of my breeding and farming aquariums in the dedicated fish room because we are preparing for renovations in our house. Once those renovations are complete later this year, we'll be scaling back up to even more than 12 aquariums.)

If anyone has any questions about the Protectli hardware, let me know and I'll answer as best as I can.

And here's a couple of images about what I describe above, including 2 of my several aquariums - the 2.5 meter long Reef tank and a smaller dedicated anemone tank.

Finally, and again, many, many, many thanks to Oliver, the Vates Team and everyone in the XCP-NG community who has contributed to a most excellent OpenSource project. None of what I did here, would have been possible without all you contributing to this most epic effort.

PS - Oliver - if you're ever looking for enthusiastic and skilled XCP-NG resources, please let me know. Both my business partner (@bill-gertz - who's already contributed to the XCP-NG project with acme.sh improvements for OPNSense / pfSense implementations) and myself are more than capable and willing.

Well, it's been more than a week now and have only good news to report with running 7.6.

In fact, I'm rather stunned - as CPU usage / Load logging clearly demonstrates even significant performance gains on my my HomeLab production system (an i5 7200 Intel NUC-Like system with 6 Intel NICs & 24GB RAM / 500GB Samsung EVO SSD).

Running 4 Linux VMs / a BSD VM / a Windows10 VM on XCP-NG 7.5 - CPU perf was averaging around 25% & Load was around 0.4-5 average when under light load.

Now, running all those VMs on 7.6, CPU perf average has dropped to 10% and Load averages have also dropped to 0.1 - 0.15 under the same VM load as before.

My two other XCP-NG - a Disaster Recovery box and a Test Bed (a laptop from 2009) also appear to have realised similar performance gains as well.

I didn't see any mention of "performance gains" in this maintenance release, but I'm amazed and very impressed.

I did do complete fresh Installs on my production system & DR Box, and only did an upgrade on my Test Bed laptop. All 3 migrations to 7.6 went without a hitch.

Well done contributors and builders - well done indeed!

Absolutely, please be my guest.

BTW, if you're ever in the Netherlands and want to drop by The Hague for beer and a personal visit to see the whole setup, you have an invitation.

@badrAZ w00t! Thank you! I look forward to the update.

BTW, I went ahead and threw my updated XO CE into "production" here at home where it runs 10 VMs. So far no other issues seen by me. XCP-NG/XO.

@errellion Have you ever considered or looked into using a USB over Ethernet utility like VirtualHere or one of the other alternatives as a work around?

In a nutshell, you could deploy something like a RPI2 or RPI3 and run VirtualHere on it & plug your dongles into that and have your VM's connect to it with a USB over Ethernet client. Then you wouldn't have issues with snapshots or migrations between hosts.

I use this for Wave, Software Defined Radio & ADSB dongles and USB printer - just forward the raw USB connections over Ethernet to my VM's and it's worked great for me for years.

Just spitballing the idea if it helps you.

@Andrew I flipped it off in BIOS, worked like a charm. BTW, I'm only on a 1G switch with my 226's. Looking forward to getting a faster network switching at home now.

Looking forward to eventually test driving XOSTOR on this cluster.

@xerxist It's good you commented. I finally figured this out and had intended to come back to update this post but forgot. Yes, that was the magic secret that solved it all.

Now, if only we could get Thunderbolt Networking support in XCP-NG, it would be perfect, but I can imagine that's not a high priority for the Vates team.

None the less, my 3 X MS-01 cluster (not setup as a pool.... yet) continues to work very well for it's first month in service.

Anyone else seeing Intel 226 (I226-LM / I226-V) network performance issues? It seems the receive performance is fine, but transmission speeds seem kneecapped at 200Mb speeds on a Gigabit switch.

My old 2012 MacMini's running XCPNG are able to migrate VM's in 1/4 - 1/2 the time that it takes my new MS-01's between nodes.

I see the same thing on XCP-NG 8.2.1 and 8.3 beta 2.

@ChristianL Well, good news. I have my MS-01s.

I've been futzing with them now for 36 hours and started off with 8.2.1 XCP-NG on it but was disturbed by the fact I'm not getting full 1 GB in both directions on my NICs.

Currently I don't have my SFP ports populated; I'm only using my 2 x 2.5Gbe NICs at the moment. Down seems to pull fine in at 1GBe (my switch is the choke) but going back out of my box suffers at approx 200Mb speeds.

I just now upgraded all three MS-01 nodes to 8.3 beta2 and still see the same limitation on sending stuff back out and up the Ethernet NICs.

Do you still see your VLAN performance issue? Have you checked to confirm that issue is specific to VLAN and not LAN?

Also, have you managed to get the Ethernet NIC feature of the Thunderbolt ports working?

@ChristianL Cool to hear about your report and successes.

Sadly, Amazon lost my 3 X MS-01 computers in their distribution center in Witen Germany, and I have spent the last 10 days struggling with Amazon to give me a solution - either resend me 3 new ones (which they refused to do) or refund my money to a gift-card so I could immediately buy 3 new ones and get them shipped to me by tomorrow - which they said they cannot do because it was a high value ticket item. So now I'm stuck waiting another 3-5 days for my CC to get the refund, so I can order 3 new ones to replace the ones that were lost 15 days ago by Amazon.

I'll just source them via another channel now, but I'll eventually get them, and get started on mine.

@ChristianL Yeah, the learning curve can be a tad difficult at the start. I came to XCP-NG on my own in 2018 (just as the project got off the ground) after having to work on securing Citrix XenServer environments at my former employer (circa 2008-2015) and even for me then, I spent a few late nights trying to wrap my head around how it all went together. But since then, I've relied on XCP-NG to run / host my Home Assistant instance (among many other things) to automate and maintain my multiple marine fish & coral reef aquariums at home for 6 years now.

What's the issue with VLAN routing you see? You disabled hardware checksumming, right? Or are you doing NIC pass through?

https://docs.xcp-ng.org/guides/pfsense/

@ChristianL Just saying hi as a soon to be fellow user of the Minisforum MS-01s.

I'm migrating away from running XCP-NG on one of the original 6 port Protectli model of NUC-like computers that I've run XCP on for 5 years, because one of them finally died approx a month ago.

Come the end of next week, I should have 3 MS-01 units to build my own XCP-NG HA cluster with & am excited about getting started with those little beasts.

Have you seen any issues with your own unit / install?

I hope to get a Thunderbolt mesh network going for the clusters; am hoping it works as well on XCP-NG as others have reported on Proxmox.

Anyways, just wanted to say Hi to a fellow MS-01 user.

Best regards

Absolutely, please be my guest.

BTW, if you're ever in the Netherlands and want to drop by The Hague for beer and a personal visit to see the whole setup, you have an invitation.

So I woke up and saw this today:

https://xcp-ng.org/blog/2020/09/09/edge-case-2-protectli-hardware/

And a massive smile came across my face, because this is the solution I have cut my teeth on XCP-NG just over 2 years ago now, and is still reliably running my home and two person cybersecurity consultancy business with 9 VM's running 24/7, and another 3-4 VMs that we spin up whenever we need them.

Two years ago this month, I purchased a new Protectli model, the i5 7200u as a replacement system for my old and finally failed (blown caps) Firewall. While waiting for the shipment to arrive, I saw @olivierlambert give his presentation on forking Xen and the launch of XCP-NG. Already familiar with Xen through my previous employer (T-Mobile), I thought when my Protectli unit arrives, and before I launch it as a bare metal firewall on my network, I thought I'd try XCP-NG on it first. When I did, my mind was blown almost instantaneously for a few reasons.

First, there was a bit of a learning curve. I'd never actually installed / administrated a Xen server at my former employer - that wasn't our job. Specifying security requirements and performing pen-testing and audits of implementations was more our thing. Thanks in part to the members of the community that were here, and even Oliver himself (who graciously extended my XOA demo license & answered further technical questions I had) that learning curve was just weeks rather than months. And after some initial configuration experimentation and performance testing, I ended up launching our XCP-NG Protectli unit with a Firewall (pfSense) VM and our first completely virtualised DMZ. Fuller disclosure: I've run my own Email, Web and DNS servers from home since 1994. But this was the first time I've attempted to virtualise many of my servers running on bare metal. And I was so impressed, within a couple of months - I was virtualising nearly everything onto this box apart from my NAS units onto XCP-NG, including two RaspberryPi's. In fact, at the time, I made many jokes about how consolidating two very heavily loaded RaspberryPi 3B+ Units (sucking a total of 10+ watts) into XCP-NG on Protectli was resulting in electrical bill savings & huge performance gains on the apps that previously ran on those RPI's were now realised.

What was on those two RaspberryPi's? The master node of the first High Availability implementation of Home Assistant which I specifically built for monitoring, managing and automating my Nano Coral Reef and Marine Fish breeding farm, composed of 12 interconnected aquariums. It also controls our house too, but the aquarium aspect is really why I needed 99.7%+ uptime service availability. So I architected a high availability solution using RaspberryPi's that works in practice, not unlike the Space Shuttle or Dragon Crew Flight Systems computer. Each node is sanity checking the other nodes and ensuring it's still online and operating as expected. If any node drops off, freezes, locks up, or operates outside of designed parameters, one of the remaining 4 nodes on the network (each even on their own circuit breaker) will take over. Additionally each RaspberryPi functions as a GPIO / USB / I2C / Dallas 1-wire sensor input/output device - except the master Home Assistant node and a dedicated RPI for the SQL server, which was located in my office. And migrating this master node and related SQL to a VM on XCP-NG was easy-peezy, and required no changes to my High Availability design of Home Assistant. In fact, it brought new features even better assuring more resilient high availability features.

I'm not 100% sure of this, but via the Home Assistant forums, we are under the impression we have been operating one of the largest domestic installations of Home Assistant that exists out there. It controls over 80 electrical sockets, and another 68+ network controlled lights, over 20 different temp probes, two dozen+ electrical pumps (including 8 aquarium dosing pumps) , displays 10 residential security cameras, and streams another 10 public webcams at the local Dutch beaches (which my wife calls "Sunset Cams"), tracks overhead airplanes (as far as the coast of the UK) using ADS-B, tracks the International Space Station & crew members onboard (my wife works for the ISS), weather conditions, and much much more.

This solution also monitors my home-office network, once waking me up at 5am when one of my Cisco network switch's power supply failed. 15 minutes later, I'd had dropped in a cold-standby Cisco switch in it's place and was crawling back into bed. But the real life saver has been with the aquariums, whether it was reacting to overflow situations to Ground Fault Circuit failures (think ageing aquarium heaters that die / leak in the middle of the night) that result in unexpected power cuts (the aquariums have their own dedicated GFCI protected electrical circuits).

And of the 3 years we've had this Home Assistant solution (May 2017), 2 of those years the primary node has been running on XCP-NG on the Protectli hardware.

Originally, we started running XCP-NG on the Protectli with just 16GB of RAM and a 512GB SSD. Today, it's got 32GB of RAM and a 2TB SSD. The Protectli unit is wall mounted in my office and serves as a complete 100% replacement for all our former Cloud experiences, including Apple's iCloud / Office365 / GoogleDocs / WeTransfer / Dropbox / Maps and more. And we couldn't be more happy with this solution - so much so - my business partner and I are looking to launch a Edge based service using a similar combination of tools for the consumer / commercial market.

In March of this year, the Home Assistant Blog officially recognised my build and featured my implementation in the community spotlight.

https://www.home-assistant.io/blog/2020/03/15/community-highlights/

For those who are interested in my Home Assistant portion of the build, you can find full details on this epically long thread (TL;DR) here:
https://community.home-assistant.io/t/going-to-next-level-of-aquarium-automation-whos-with-me/18486
(There is a discrepancy in regards to the total number of aquariums - In Jan/Feb, we started shutting down several of my breeding and farming aquariums in the dedicated fish room because we are preparing for renovations in our house. Once those renovations are complete later this year, we'll be scaling back up to even more than 12 aquariums.)

If anyone has any questions about the Protectli hardware, let me know and I'll answer as best as I can.

And here's a couple of images about what I describe above, including 2 of my several aquariums - the 2.5 meter long Reef tank and a smaller dedicated anemone tank.

Finally, and again, many, many, many thanks to Oliver, the Vates Team and everyone in the XCP-NG community who has contributed to a most excellent OpenSource project. None of what I did here, would have been possible without all you contributing to this most epic effort.

PS - Oliver - if you're ever looking for enthusiastic and skilled XCP-NG resources, please let me know. Both my business partner (@bill-gertz - who's already contributed to the XCP-NG project with acme.sh improvements for OPNSense / pfSense implementations) and myself are more than capable and willing.