@olivierlambert

Bringing this back up because I'm thinking of retiring old servers in exchange for new mini-PC.

Looking around in XO, I don't see anywhere to click to promote a server in a pool to controller (master), am I missing something?

My plan would be to join all new mini-pc to the existing pool, migrate VMs to these new servers, then take controller (master) onto one of the mini-pc. After that I would remove the older servers from the pool and shut them down.

Is the proper way to use the above command, or go to the local console and just force promote one of the new mini-pc to be the new controller?

[edit] found it, been a while since I've set up a pool. Click where the arrow is and you get choices on which server in the pool you want to control things.

I'm making a plan to cover a situation and it brings up what might be a stupid question. Migrate from newer hardware to older harder or "smaller" AMD hardware?

In a stroke a genius, we are possibly cutting my tech room in half, cutting out concrete at the end of one room and building an extension into my tech area (TV studio equipment and server equipment). This will of course generate massive amounts of concrete dust. It will also probably shut down the cooling. And will give us no room for expanding any tech equipment.

My choices are shut down production servers one by one and move them to another room where I have connections. Then reverse this when the construction is done. A big job that I'm not looking forward to doing.

Pull my lab apart and put that in the other room, two parts to this. Each option has 3 hosts and 1 storage.

Part one is pull my "big lab" apart which is HP DL360 gen8 servers, move those, join the pool, and migrate vms and then migrate XCP controller. This would be going from Xeon Silver (10c/20t) to Xeon E5 v2 processors (20c/40t).

Part two would be pulling my "mini lab" down and moving it to the other room. This is HP T740 thin clients, core count is my concern here with AMD v1756b 4c/8t processors and half the ram at 64gb. This mini lab all fits on a single rack shelf and much easier to move.

The work load from the VMs is pretty light, so I think I can over provision the T740 and it will be OK. I could also add more T740 if needed ( or T755 if I'm lucky, 6c/12t).

I'm not sure which of these choices is going to make the most sense for 3-4 months of use. The other room is in another building and an extra 3 switches away from my core switching stack.

@mattnj

Thanks for the update, I didn't realize you could select the kernel in the ipxe boot, this might help me down the road.

@mattnj

[edit] I think you should make a second FOG server for testing the secure boot stuff to prove it out, I'm not going to try this on my production system until I know I'm not going to mess it up [/edit]

I think this is the string of posts:

https://forums.fogproject.org/topic/15888/imaging-with-fog-and-secure-boot-poc/6

If you get this working, I'd really like to know because I'm going to need to got through this for summer refresh on my desktops and want to turn secure boot back on. Sure would have been nice if Microsoft updated WDS to work with win11 so smaller facilities still had a Microsoft method of doing this and not jumping through hoops or buying something expensive.

Do you still have the physical machine? I've had some luck with disk2vhd:
https://learn.microsoft.com/en-us/sysinternals/downloads/disk2vhd

Remember to create a VHD not a VHDx.

If you have the physical machine and it still works, you could also try using Clonezilla on both the physical and virtual machines to transfer the image over the network.

Make the VM but do not boot, simulate as many aspects of the physical machine as possible (ram, drive size, MAC address, etc). This will let you import the VHD into the UUID of the disk you just created, start it up and see what happens. I'm a little foggy on the details, I'd need to walk through this again, but I did get it to work on one of my physical servers when I moved to virtual, one other failed because an application had too many things tied to physical bits of the server and I had to go through support to update it's license on a fresh VM.

All that said, problems with your secure boot are concerning. Are you saying that even a fresh install with secure boot is failing? I've been using the Eval versions of Windows for most of my testing, they should be close enough to the release versions that this should all be the same. Just for fun, I'd suggest downloading the win11 Eval and giving that a try to see if you can create a new VM that works with vTPM, vSecureBoot, vUEFI

@cairoti

I think this may be like my benchmarks, the benchmarks show decent speed to disk, but migration from server to server to local to server are just SLOW.

@cairoti

The Broadcom cards can be a problem, is it possible to swap out for Intel cards?

@cairoti

For me, it is normal to only see 400mbps when migrating from one NFS storage server to another NFS storage server. This is also on a 10gbe network and the drives are fast enough to benchmark a Windows VM up to 6gbps. MTU only 1500.

Under the same storage servers and ESXi8.02, I get faster speeds and I think this is because they use nconnect=4 as the default for NFS connections. I need to do more work with ESXi and the whole vSphere system before rendering firm conclusions, but this might be a thing.

Truenas Scale 24.10.x on both storage servers, both with spinning SATA drives for the array.

@DustinB

Secure Boot may depend on the version of Windows you are using. Education and LTSC don't care right now (could change). As you note, vTPM would be required and vUEFI might be needed too.

The only testing I've done in this area is Windows Server 2025 which is essentially 24h2 win11. But that is only 1 machine, and a fresh install. I did use secure boot, tpm, and uefi to install it.

What I would suggest for the OP is this:

Install the same version of Windows 11 as a clean install into a VM, see if anything stops you from doing this. Install it without secure boot to make sure that works, else if it requires secure boot, you have a bit of a process to get the FOG PXE boot working with secure boot. This is something I know I'm going to have to deal with in the near future, there is a procedure built in a forum post that's hard to find, it can be done but not a simple task.

@Mefosheez

The only experience that I have is Server 2025 Eval with an older Xeon, vtpm, vsecure boot, vuefi and no problems with default configuration using win2022 template and turning on the above mentioned settings.

The only real difference is that I always uncheck automatic boot during creation, go into advanced and change the Intel NIC, save and then initial boot.

I mention that all of the settings are virtual because my old servers do not support hardware uefi or hardware tpm 2.0 so all of that must be synthetic for my lab.

I'm also using up to date XCP 8.3 with this VM.

I have never deleted snapshots before migrating disk from one NFS to a different NFS, this might be different shares on the same physical NAS or shares between two physical NAS.

@flakpyro

Thanks, that will probably save me an hour this summer if I decide to upgrade my production system. By then it might all be fixed.

Well... This is good news and I look forward to when a (stable enough) Alpha or Beta is available for us to test. I might even treat my "big lab" to some newer/larger SSD system drives for this since it will probably need to be a fresh install for the first rounds of testing.

As far as old hardware support, my lab is really the only place I worry about this, my production system is new enough that it should be supported for a while. Made sure that it was good enough for "modern" Windows requirements which is UEFI, Secureboot, and TPM 2.0 hardware.

@flakpyro

I haven't done much testing with it, but delayed start doesn't seem to cause any negative side effects. If you have a large number of services, maybe this could become an issue, but I doubt it.

It was fully supported when I installed it (I think).

If you need it for production, it seems stable enough with the delayed start change. Not sure if other roles or features need this change so I would just be aware of it and proceed.

Also of note, this only delays the management agent from reporting back to the Xen system, it does not delay the drivers from loading. The only complication I could see is booting the VM and immediately trying to migrate it or shut it back down. Normally the MA doesn't do a lot of work so again, delayed is probably fine as I'm seeing delayed is faster than I can log in to look at it.

@olivierlambert

Thanks. I've been doing some digging this morning and what I was thinking of testing isn't going to work. It's interesting reading but not going to help me.

@mauzilla

I just read this thread and it might help

https://xcp-ng.org/forum/topic/9924/vm-vcpu-allocation

I've only ever run my systems with the same processor and configuration in each host, so not sure about your system. The biggest issue with a mixed environment is to keep the processors in the same family/generation, and mixing Intel and AMD in different hosts could cause issues after a migration.

@mauzilla

I tend to set almost all of my VMs to 1 socket with 8 cores. My production system only has one socket per host, so no big deal there. But my lab has 2 sockets per host. I have seen mist of the cores on a host active with multiple VMs configured this way, so the hyper visor must be balancing this in one way or another.

One of these days I'll have to try setting a VM for 8 sockets with 1 core each and see what happens.

Why not just set the VM to 1 processor with X cores?

I didn't really find an answer with web search or forum search... Is managing the Xen Project servers the same in XOA or XO-CE as managing XCP-NG? I want to test something for comparison to XCP-NG and it seems the best way for me to test this is to build a Xen Project hypervisor (plus a small pile of money for a new host that I will need to order).

Any known issues, I need really basic control of the host, no pools, remote storage, and would be nice to have a console into a Windows VM.

If not I'll figure this out the hard way and RDP into the VM.

Any known issues with using an HP T740 for this task? It's an AMD V1756b processor (4c/8t) and I'll probably jam a Supermicro 10gbps card in it and 64GB of ram, plus some local storage. I have some of these running vSphere8 that I'm learning on, and they seem to be fine for light lab use, I'm guessing XCP-NG will be fine on them, but never messed with the Xen Project version.

Should I go with a RHEL based OS like Rocky or Alma instead of Debian?

@archw Yes,

I'm seeing the network changing too. But no rhyme or reason behind it. I know I changed this once last week, and had to change it this morning. Did a reboot and it is still OK so I'll have to monitor this as well.

So far after moving the MA service to delayed start, things are working fine in this regard. The only service I've added is the Zabbix Agent2 and connected it to my Zabbix server. So far, so good.

I'm going to say that I won't be upgrading any of my production servers to 2025 until summer of 2025, just to be safe and hope all these little oddities get worked out. I'm guessing most of these are Windows changes that don't really play nice with the PV drivers. Have to see how quickly these drivers get updated.

@olivierlambert

It is looking good, and like many people I am eagerly awaiting the roll out of the complete XO-Lite. But we need to remember that there are other ways to get your system up and going. You can grab the official XOA right from the XO-Lite pages, you can command line from a host and download XOA or one of the XO from sources. You can also take a completely separate Linux computer, in my case this is an old HP T630, install Debian, and build the sources on it.

No matter how you build your system, and whether you are paying for XO or using sources, I advocate to having at least one separate computer that has XO from sources running. Could be a laptop with Windows WSL, could be just a "terminal" in your rack room. Some cheap little computer that has Debian with a desktop for a few tools like web browser, and XO from sources that you can hit from the loop back or from a computer out of the room. There are times when having that local computer up to monitor things is really handy, especially if your XOA is crashed for some odd reason (host crash, etc.)

Now the above may change once they finish XO-Lite because the basic functions we need will again be present, but until then, there are ways to get going. XCP-NG Center had a lot of good stuff in it, but after teaching myself what I needed to do with XOA or XO-CE (sources), I don't really miss it. But set up from zero will be much easier with a completed XO-Lite, or at least can be different.