@nikade

BTW, the ram increase made zero difference in the VDI migration times.

I may still change the ram for each host in production, but I only have like 12 VMs right now and probably not many more to create in my little system. Might be useful if I ever get down to a single host because of a catastrophe. Currently running what it chose for default at 7.41 MiB, roughly doubled that in my lab to see what might happen.

Just got my VMware lab started so all this other stuff is going to back burner for a while. I will say I like the basic ESXi interface that they give you, I can see why smaller systems may not bother buying VCenter. I hope XO-lite gives us the same amount of function when it gets done (or more).

@olivierlambert

I was looking for a way to mark this solved, can't find it.

I haven't moved things, but after migrating my big lab to my mini-lab, I'm confident that the warm migration is the way to go. It was fast and seamless as long as you have the right network adapters set up. I had to fool with one of my networks to make a VM function, but that was certainly something I overlooked while setting up the mini-lab. A little testing before moving the VMs should make this go easily if using the old servers is the option for this project.

Having what is essentially XCP-NG Center from a browser would be nice, no more need to install something on a Windows machine just to get some basic stuff set up until you get a full XOA instance running. Yes I know you can just grab the demo with a single command, but sometimes you want to configure a few things first.

@manilx

It is not cheaper, but it draws about half the power when idle. When doing work the power still gets up there, they are 60-90 watt computers, which is still less than the 300 watt capable HP servers at full go.

If you really stress your lab a lot, real servers are a win as they will draw less power for more performance, but if yours is like mine and it idles a lot, cutting the power is nice. I'm about 200 watts at idle with the above, and that's two labs worth. I was 400 with the single old lab system.

Mounts are cheap book ends, drilled and screwed to that specific shelf, and drilled for short m4 thumbscrews to get into the VESA mount on the t740.

If anyone reading this wants to duplicate, and they see HP T755 cheap enough, I recommend these because it has 6 cores / 12 threads over the t740 with 4 cores / 8 threads. Both are DDR4 sodimm and top out at 64GB. Both have a PCI 3.0 x8 (x16 slot) that takes half height cards. And if the BIOS is locked, you can write over them with a programmer and enter the data needed to function (see badcaps forum for my guide).

Here is mine

HP DL360e Gen 8 for Truenas, 96GB of RAM, more processors than it needs, and 8x500 drives, NFS share for the VMs

3x HP DL360p gen 8 for XCP-NG with 128GB of RAM and 20c40t worth of processors.

10gbps networking to a Mikrotik switch, also an old Cisco 2960s for a second gigabit network (with POE+)

There is also a cheap GPS/GNSS NTP clock at the top of the rack, just too far to show these other items.

I like the spinning lights on the drives, just wish it was a little faster.

@olivierlambert

Thank you, that's what I thought and why I was willing to buy the lower end CPU in order to get the project through. At the time going for the larger Gold would have doubled the cost of each server.

Will this installer remove the Xenserver drivers if they are already installed? Or should I use the cleaner first?

The instructions say to remove secure boot, then run a powershell script, isn't this going to cause a problem with Windows 11 and maybe Server 2025? Can secure boot be turned back on after the powershell script has been run?

I'm trying to slice out some time to work with Server 2025 in my XCP-ng lab so I can move forward in my production system when I'm comfortable with the upgrade. I'm on 8.3 stable if that matters, I could move to 8.3 testing if needed for these drivers/management agent. I'd really like to move away from Xenserver drivers because I just don't trust them to remain available in the long term.

I'm also trying to keep my Server 2025 in UEFI, Secureboot, and vTPM going forward. We are going to be forced sooner or later so I might as well get familiar with any problems that might appear now while I can test.

@olivierlambert

The only place running XCP-ng on things like an RK3588 make sense is for home lab stuff, I probably wouldn't even build work lab stuff on these. Never enough ram (so far) and limited cores, plus almost everything will need to be running in virtual processor architecture.

All that said, it might be neat to simulate these ARM processors in x86 so that we could create a VM of the Pi, etc.

@olivierlambert

I was looking for a way to mark this solved, can't find it.

I haven't moved things, but after migrating my big lab to my mini-lab, I'm confident that the warm migration is the way to go. It was fast and seamless as long as you have the right network adapters set up. I had to fool with one of my networks to make a VM function, but that was certainly something I overlooked while setting up the mini-lab. A little testing before moving the VMs should make this go easily if using the old servers is the option for this project.

@manilx

It is not cheaper, but it draws about half the power when idle. When doing work the power still gets up there, they are 60-90 watt computers, which is still less than the 300 watt capable HP servers at full go.

If you really stress your lab a lot, real servers are a win as they will draw less power for more performance, but if yours is like mine and it idles a lot, cutting the power is nice. I'm about 200 watts at idle with the above, and that's two labs worth. I was 400 with the single old lab system.

Mounts are cheap book ends, drilled and screwed to that specific shelf, and drilled for short m4 thumbscrews to get into the VESA mount on the t740.

If anyone reading this wants to duplicate, and they see HP T755 cheap enough, I recommend these because it has 6 cores / 12 threads over the t740 with 4 cores / 8 threads. Both are DDR4 sodimm and top out at 64GB. Both have a PCI 3.0 x8 (x16 slot) that takes half height cards. And if the BIOS is locked, you can write over them with a programmer and enter the data needed to function (see badcaps forum for my guide).

@CJ

Bringing this back up as I've changed things due to space limitations that are going to be happening:

Top shelf is three VMware hosts and a Truenas Scale storage.
Bottom shelf is three XCP-ng hosts and two extra.

I think one of the extra are going to be another XCP-ng host that I keep as a separate pool with internal storage and use for backups and backup verification. The other will probably end up being a general purpose Linux desktop and probably run XO from sources as a backup (gigabit).

VM hosts are HP T740 (AMD V1756B processor) with 64GB or ram and a Supermicro dual 10gbps card (X520 based), one extra also has 10gbps but only 16GB of RAM. All of these have an extra a+e m.2 slot Intel i226-v card installed, and all have the built in Realtek gigabit which is working good enough for what I need (with XCP-ng, not VMware).

Switching is now Mikrotik CRS326-24s+2q+rm with most of the hosts broken out from the QSFP+ ports (8 connections) and a few DAC cables.

@raider600

Move the shared file to a NAS/SAN and map the drive letter to the clients?

An 8TB drive size is going to be all kinds of unhappy if you want to snapshot or backup that VM.

Exception to this might be a large database that needs fast access to the data, but even there I would expect some kind of shared storage to be at least as fast as a giant VM.

@dinhngtu

It is probably Trellix, but hard to say since Trellix operates on top of Defender. It blocks the download from starting and I didn't have time to dig into it to see what was happening.

Browser was Chrome if it matters.

Chris Titus has similar problems with the exe version of his windows tool, and he paid to get that tool signed.

@dinhngtu

Hmmm... I'm going to need to download on another computer, I'm getting a virus detected error. If you have a .ps1 script that elevates privileges, this is probably the cause. I've seen this before with Windows debloat tools (even signed ones). I'll log into a linux machine after I go set something up in another building and take a look.

Secure boot signing is probably a big hurdle, Chris Titus mentioned how much of a battle it was for him to get his Windows debloat tool signed.

I've also read a bunch on secure boot signing with the Fog server project, it sounded like it was a huge issue to deal with and a non-trivial amount of money. There is a workaround, but not for a production level thing like your drivers/agent would need to have.

I'll have to find some time next week to give this a try on one of my test VMs, if they crash, no big deal and I'll build them again.

I will make one suggestion... Can you make these available as an ISO? That way we can drop them right into our ISO SR and mount just like inserting a CD. I've been doing this with the Xenserver drivers and it works out really nice, any other tools that might be useful can also go in there.

Will this installer remove the Xenserver drivers if they are already installed? Or should I use the cleaner first?

The instructions say to remove secure boot, then run a powershell script, isn't this going to cause a problem with Windows 11 and maybe Server 2025? Can secure boot be turned back on after the powershell script has been run?

I'm trying to slice out some time to work with Server 2025 in my XCP-ng lab so I can move forward in my production system when I'm comfortable with the upgrade. I'm on 8.3 stable if that matters, I could move to 8.3 testing if needed for these drivers/management agent. I'd really like to move away from Xenserver drivers because I just don't trust them to remain available in the long term.

I'm also trying to keep my Server 2025 in UEFI, Secureboot, and vTPM going forward. We are going to be forced sooner or later so I might as well get familiar with any problems that might appear now while I can test.

@Greg_E

I have another question:

If I was going to move my production system from Intel Scalable Silver to Intel Xeon v2 to get my servers shut down for some construction, would warm migrate be the best way to move them?

We might be doing some physical construction in my server room, this would involve knocking out a concrete wall, and adding another concrete wall splitting the room in half. HVAC will also be shut down while this is happening. My choices are physically move my production servers, or rent/sell work my old lab servers. I have a room in another building where I'm going to be putting the servers that are really needed to keep things running.

What I'm thinking is that since there is a pretty big age gap in the processors, that a warm migration might be the best way forward to make sure that the network interfaces, and the processor architecture don't get in the way. Yes new shiny tool gets used for everything.

Also the Lab migrations are going really well, seeing some speed increase now that I have an XO on the new hosts and this XO has a 10gbps connection. Still only getting around 1-2gbps, but that's WAY better than the 400mbps I was getting before. Making me rethink how I have my production system running, and making me think about what I'm going to do with that fourth T740 in the picture (think it will be a standalone XOsources on 10gbps). I'm down to a Windows server of around 60GB disk was about 4 minutes, around 20 minutes the old way. Even my KASMweb server was around 4 minutes and it had around 500gb sized disk (not too much in use). Over all a big step up with small sized equipment. Even the Realtek gigabit NICs are working fine, can't say that with vmware!

@olivierlambert

Well... That worked just like you would think.

Set up a new NFS share on the new mini lab, set up a new pool made up of the three T740, selected my test machine and hit the warm migrate button to migrate to that new NFS share. After 22 minutes the VM popped up on the new system and the old one is waiting to be destroyed, just like planned. My only request would be to add the selection of the receiving host (if possible) this one migrated to the AMD master and I probably would have sent it to #2 or #3 since the master does more work than the other hosts.

Word or warning: Do not mix AMD and Intel on the same pool, and move the master function between them. I had to do a full shutdown to get control back so I could remove the AMD machines from the pool.

Also of note: When a host joins a pool, it will inherit the password from the master of the pool. All my AMD had a different password when I started this yesterday, now they share the same password as my old Intel master.

Went from this:

To This:

The first three on the bottom shelf are XCP-ng, the top three are vmware. The extra 2 are undecided, one might be a general Linux machine and probably XO from sources, the last might be a firewall since it has 6 network connections. Truenas Scale on the storage machine, 6 SATA spinning disks. All of this on 10 gigabit connections with a CRS326-24s+2q+ switch.