Empowering Nor-Cal Controls: A Success Story in the Power Generation Sector

For this new user story, we had the opportunity to speak with Chuck Colby, IT/OT Solutions Architect at Nor-Cal Controls that recently moved their on-premises infrastructure on Vates Virtualization Management Stack (XCP-ng and Xen Orchestra).

👷 My name is Chuck Colby, and I suffer delusions of grandeur as the IT/OT Solutions Architect for Nor-Cal Controls ES., Inc. I started as an IT Specialist in Silicon Valley approximately 1 million years ago, (or 35 years ago) and became a small business consultant, then a network engineer, and now I get paid to figure out solutions to interesting challenges. This is a great fit for my natural curiosity.  At the moment, I’m splitting my time between re-engineering Nor-Cal’s internal IT landscape and assisting our customers with their own infrastructure challenges. I hope to be able to hand off a well-engineered network to an IT Director and focus my time on customer challenges.

Who is Nor-Cal

Nor-Cal Controls ES, Inc. is a professional, customer‐focused company offering consulting, engineering, and training services to the United States power generation utility sector. We specialize in power plant automation, solar and process engineering solutions, providing customized SCADA (Supervisory Control and Data Acquisition), MET Stations, and DAS Systems.

Our team of qualified Engineers have over 50 years of combined progressive and working experience serving the solar energy utilities, marking Nor-Cal as the industry’s most trusted leader.

Our ‘motto’ is "Your project, our passion".

The core values of the company are :

• Family –We treat everyone like our family. We are supportive, respectful and accepting.
• Trust – We do what is right. We are honest, transparent and ethical.
• Hard Work- We perform to the best of our abilities. We are problem solvers who will do whatever it takes to get the job done right.
• Dependability – We are customer-focused, responsive, adaptive and flexible. You can count on us when you need us the most.
• Knowledge- We promote a culture of learning. We empower our customers to take ownership, but are always here if you need us.

We asked Chuck about the current and future project at Nor-Cal

Chuck👷: Nor-Cal Controls ES., Inc has experienced what can only be described as meteoric growth. This, of course, places a huge strain on the infrastructure – so we are taking the system that made a lot of sense 5 years ago and engineering it to begin to make sense for the next 5 or 10 years. With one of our core values being trust (and with it, transparency) we have been really focusing on the open source community and products to help solve interesting puzzles.

This has been said many times by people smarter than I, but open source is not about free (as in beer) software, it’s about having a more interesting and effective array of tools with which to tackle your challenges. If it doesn’t quite fit, you can ask the developer (or hire your own) and take it in a new direction.

Nor-Cal Controls infrastructure

Chuck also shared his thoughts on Nor-Cal Controls' infrastructure:

👷 As noted, our infrastructure is rapidly evolving. I inherited a network that made a lot of sense for a company of 25-40 people ; fortunately/unfortunately, we were already over 110 people strong in an incredibly short period.

We use another virtualization provider with our customers, but I’m not certain that the other provider is providing much added value along with their ever increasing licensing costs.  I thought it would be a worthwhile exercise to cast the net a little wider, and in the process find a competing product for interested customers.

We started with augmenting the aging physical Windows Domain Controller.  We had some very capable Dell servers that were severely underutilized, so I combined those with a TrueNAS Scale deployment (also built from repurposed hardware) tied together with 10G switching to create a very capable resource pool using only XO From Source. The intent was always to purchase the product provided I didn’t find an insurmountable issue along the way. The value in the ability to lab this using repurposed hardware and zero software cost into a capable proof of concept – in fact, far more capable than the production system it was proposed to replace - can not be adequately expressed.

We ran the test for many months, and in that process found other potential uses for the products, such as a working lab for our SCADA engineers.

What led you to choose our solutions for your infrastructure?

👷 I’ve been in and around IT for many years ; I’ve seen virtualization when it was the « new hotness ». I’ve seen it transition from being a tool for only the largest of enterprises to a tool that I’d put into a 5-user network if only for the ability to snapshot and backup. For many of my previous small business clients, that was exactly why virtualization was purchased – better block level backups.

There are some things that just make sense as being part of the offering. For instance, when I looked at XCP-NG and XOA, the inherent value is so much higher than the competition. Competitors charge for features that are provided in XOA’s free version. Organizations that are more serious about how the virtualization infrastructure works can opt to expand XOA’s capabilities through licensing as we did. It’s very much a pay-for-what-you-use model, rather than the feeling that you’re being charged for the most basic of features.

For instance, when I looked at XCP-NG and XOA, the inherent value is so much higher than the competition.

Initially I simply wanted an inexpensive way to use surplus hardware to:

• a) augment existing infrastructure
• b) build a proof of concept lab for an initiative that ultimately we ended up not pursuing ; the initiative was deemed to not be in our core competency so was – after being well represented on the XOA / XCP-NG platform – moved to the side.

Virtualization, however, is not a one trick pony. We have a number of initiatives to improve automation internally. We have found that we are able to spin up and configure VMs very easily via Ansible and the XOA API, we could deploy test servers for a very wide variety of uses very quickly.

We use primarily Dell R3xx, R4xx and R6xx servers for both ourselves and our customers. Our 10G core/distribution switches are a pair of MikroTik CRS326-24S+2Q+ (in case one has an issue – which has not been a thing, thankfully), VLANed out for a number of different functions – Lab, internal network, server network, voice network, wireless networks. We are steadily replacing aging Netgear access switches with Cisco switches. We have an un-routed NAS (really, almost a SAN) network that ties the XCP-NG hosts to the VM disk pool.

Primary storage is an iXSystems TrueNAS Core X10 which provides SMB sharing on the client network as well as an NFS pool for VM Disks on the unrouted network. We have a small handful of TrueNAS Scale boxes for various backup and/or ZFS sync duties.

How does this architecture allow you to meet your business needs?

👷 As technical people, we are too often put in the position of determining whether or not a technology is a value at its price point. I’ve seen rapidly increasing instances where it is not. Value is always determined by the customer, not by the provider.

With this solution, we are able to pay for what we need rather than having to purchase higher subscriptions for what we believe to be basic functionality. This reduces R&D costs significantly, resulting in both an operational internal IT environment and a lab environment that easily meet the needs of a rapidly growing company.

What were the important things that convinced you to use XCP-ng and Xen Orchestra rather than another solution?

👷 More than anything, philosophy. I do not want to take anything away from a company that charges a fair price for a good product ; that’s what our economies should be built on. Craftsmanship should be rewarded. I’m not necessarily interested in a product that sets pricing to keep investors happy.

Aside from philosophy, execution. This is a very well executed project with a strong community around it. You’re not going to be left out to dry with whatever issues you may be experiencing.

I also love the methodology of using pools so that like resources have like configurations. This makes high availability (HA) amazingly simple to deploy, and makes pools actually usable.

Lastly (on this list) but certainly not least, support. The response time and expertise has been quite good. All too often when contacting a support organization you wonder if the person receiving the ticket actually has the skills to carry through. I have had no such concerns with this solution. In our most difficult support issue we found that our own engineers caused the issue that was making myself and Vates’ support scratch our collective heads. The response was quick, effective, and thorough ; unfortunately, our team was more thorough in breaking the system.

For those that want to know more about that last statement ; when your engineering team goes through a Windows Server and disables a bunch of core services, you’re going to have a bad time installing virtualization agents.

With this solution, I never doubt the support team's abilities.

Which features do you like the most and do you feel that they enable you to accomplish your daily mission?

👷 I always work my way backwards from backups. If backups are not solid, what are you even doing ? As mentioned above, this is baked into the product ; various stragegies and abilities for various licensing levels, but backups are part of the deal.

Once you’re confident with your backup solution, for us, ease of VM deployment is important. While we have scripts that will build a VM in moments, I can equally train a reasonably competent technician how to deploy a server in minutes.

Lastly, the metrics that you can see on your deployment are on point. The more you oversubscribe your infrastructure (as we do for our lab deployments), the more you want to keep an eye on what’s actually being utilized. The metrics you get from the system help immensely.

Any additional thoughts?

👷 I initially learned about this project through a very prolific YouTube channel ; comparisons were made against other virtualization platforms and a strong case was built for standardizing on XCP-NG/Xen Orchestra virtual Appliance. I took the challenge and I feel that it’s paid off quite well.

This is a very strong product with excellent potential for a wide variety of uses. Based on very promising internal testing I will be recommending this solution to our proposal coordinators as a virtualization option for our customers.