Having what is essentially XCP-NG Center from a browser would be nice, no more need to install something on a Windows machine just to get some basic stuff set up until you get a full XOA instance running. Yes I know you can just grab the demo with a single command, but sometimes you want to configure a few things first.

Here is mine

HP DL360e Gen 8 for Truenas, 96GB of RAM, more processors than it needs, and 8x500 drives, NFS share for the VMs

3x HP DL360p gen 8 for XCP-NG with 128GB of RAM and 20c40t worth of processors.

10gbps networking to a Mikrotik switch, also an old Cisco 2960s for a second gigabit network (with POE+)

There is also a cheap GPS/GNSS NTP clock at the top of the rack, just too far to show these other items.

I like the spinning lights on the drives, just wish it was a little faster.

@olivierlambert

Thank you, that's what I thought and why I was willing to buy the lower end CPU in order to get the project through. At the time going for the larger Gold would have doubled the cost of each server.

@olivierlambert

Bringing this back up because I'm thinking of retiring old servers in exchange for new mini-PC.

Looking around in XO, I don't see anywhere to click to promote a server in a pool to controller (master), am I missing something?

My plan would be to join all new mini-pc to the existing pool, migrate VMs to these new servers, then take controller (master) onto one of the mini-pc. After that I would remove the older servers from the pool and shut them down.

Is the proper way to use the above command, or go to the local console and just force promote one of the new mini-pc to be the new controller?

[edit] found it, been a while since I've set up a pool. Click where the arrow is and you get choices on which server in the pool you want to control things.

I'm making a plan to cover a situation and it brings up what might be a stupid question. Migrate from newer hardware to older harder or "smaller" AMD hardware?

In a stroke a genius, we are possibly cutting my tech room in half, cutting out concrete at the end of one room and building an extension into my tech area (TV studio equipment and server equipment). This will of course generate massive amounts of concrete dust. It will also probably shut down the cooling. And will give us no room for expanding any tech equipment.

My choices are shut down production servers one by one and move them to another room where I have connections. Then reverse this when the construction is done. A big job that I'm not looking forward to doing.

Pull my lab apart and put that in the other room, two parts to this. Each option has 3 hosts and 1 storage.

Part one is pull my "big lab" apart which is HP DL360 gen8 servers, move those, join the pool, and migrate vms and then migrate XCP controller. This would be going from Xeon Silver (10c/20t) to Xeon E5 v2 processors (20c/40t).

Part two would be pulling my "mini lab" down and moving it to the other room. This is HP T740 thin clients, core count is my concern here with AMD v1756b 4c/8t processors and half the ram at 64gb. This mini lab all fits on a single rack shelf and much easier to move.

The work load from the VMs is pretty light, so I think I can over provision the T740 and it will be OK. I could also add more T740 if needed ( or T755 if I'm lucky, 6c/12t).

I'm not sure which of these choices is going to make the most sense for 3-4 months of use. The other room is in another building and an extra 3 switches away from my core switching stack.

@mattnj

Thanks for the update, I didn't realize you could select the kernel in the ipxe boot, this might help me down the road.

@mattnj

[edit] I think you should make a second FOG server for testing the secure boot stuff to prove it out, I'm not going to try this on my production system until I know I'm not going to mess it up [/edit]

I think this is the string of posts:

https://forums.fogproject.org/topic/15888/imaging-with-fog-and-secure-boot-poc/6

If you get this working, I'd really like to know because I'm going to need to got through this for summer refresh on my desktops and want to turn secure boot back on. Sure would have been nice if Microsoft updated WDS to work with win11 so smaller facilities still had a Microsoft method of doing this and not jumping through hoops or buying something expensive.

Do you still have the physical machine? I've had some luck with disk2vhd:
https://learn.microsoft.com/en-us/sysinternals/downloads/disk2vhd

Remember to create a VHD not a VHDx.

If you have the physical machine and it still works, you could also try using Clonezilla on both the physical and virtual machines to transfer the image over the network.

Make the VM but do not boot, simulate as many aspects of the physical machine as possible (ram, drive size, MAC address, etc). This will let you import the VHD into the UUID of the disk you just created, start it up and see what happens. I'm a little foggy on the details, I'd need to walk through this again, but I did get it to work on one of my physical servers when I moved to virtual, one other failed because an application had too many things tied to physical bits of the server and I had to go through support to update it's license on a fresh VM.

All that said, problems with your secure boot are concerning. Are you saying that even a fresh install with secure boot is failing? I've been using the Eval versions of Windows for most of my testing, they should be close enough to the release versions that this should all be the same. Just for fun, I'd suggest downloading the win11 Eval and giving that a try to see if you can create a new VM that works with vTPM, vSecureBoot, vUEFI

@cairoti

I think this may be like my benchmarks, the benchmarks show decent speed to disk, but migration from server to server to local to server are just SLOW.

@cairoti

The Broadcom cards can be a problem, is it possible to swap out for Intel cards?

@cairoti

For me, it is normal to only see 400mbps when migrating from one NFS storage server to another NFS storage server. This is also on a 10gbe network and the drives are fast enough to benchmark a Windows VM up to 6gbps. MTU only 1500.

Under the same storage servers and ESXi8.02, I get faster speeds and I think this is because they use nconnect=4 as the default for NFS connections. I need to do more work with ESXi and the whole vSphere system before rendering firm conclusions, but this might be a thing.

Truenas Scale 24.10.x on both storage servers, both with spinning SATA drives for the array.

@DustinB

Secure Boot may depend on the version of Windows you are using. Education and LTSC don't care right now (could change). As you note, vTPM would be required and vUEFI might be needed too.

The only testing I've done in this area is Windows Server 2025 which is essentially 24h2 win11. But that is only 1 machine, and a fresh install. I did use secure boot, tpm, and uefi to install it.

What I would suggest for the OP is this:

Install the same version of Windows 11 as a clean install into a VM, see if anything stops you from doing this. Install it without secure boot to make sure that works, else if it requires secure boot, you have a bit of a process to get the FOG PXE boot working with secure boot. This is something I know I'm going to have to deal with in the near future, there is a procedure built in a forum post that's hard to find, it can be done but not a simple task.

@Mefosheez

The only experience that I have is Server 2025 Eval with an older Xeon, vtpm, vsecure boot, vuefi and no problems with default configuration using win2022 template and turning on the above mentioned settings.

The only real difference is that I always uncheck automatic boot during creation, go into advanced and change the Intel NIC, save and then initial boot.

I mention that all of the settings are virtual because my old servers do not support hardware uefi or hardware tpm 2.0 so all of that must be synthetic for my lab.

I'm also using up to date XCP 8.3 with this VM.

I have never deleted snapshots before migrating disk from one NFS to a different NFS, this might be different shares on the same physical NAS or shares between two physical NAS.