Having what is essentially XCP-NG Center from a browser would be nice, no more need to install something on a Windows machine just to get some basic stuff set up until you get a full XOA instance running. Yes I know you can just grab the demo with a single command, but sometimes you want to configure a few things first.

Here is mine

HP DL360e Gen 8 for Truenas, 96GB of RAM, more processors than it needs, and 8x500 drives, NFS share for the VMs

3x HP DL360p gen 8 for XCP-NG with 128GB of RAM and 20c40t worth of processors.

10gbps networking to a Mikrotik switch, also an old Cisco 2960s for a second gigabit network (with POE+)

There is also a cheap GPS/GNSS NTP clock at the top of the rack, just too far to show these other items.

I like the spinning lights on the drives, just wish it was a little faster.

@olivierlambert

Thank you, that's what I thought and why I was willing to buy the lower end CPU in order to get the project through. At the time going for the larger Gold would have doubled the cost of each server.

@stormi said in Nested Virtualization of Windows Hyper-V on XCP-ng:

Actually, Xen never officially supported Nested Virtualization. It was experimental, and broke when other needed changes were made to Xen. Now there's work to be done to make it fully supported, and this won't happen before the final release of XCP-ng 8.3. This will be documented in the release notes.

This is also an issue for us internally as we create a lot of virtual pools for our tests.

I read through a lot of the earlier posts and finally started scrolling to find this, which is the answer I was looking for. Why do I care? There is a Microsoft evaluation learning lab for things like Intune that runs in Hyper-V, basically a bunch of VHD (x) that get spawned as needed. Applications I need to teach myself. Running XCP-NG 8.3 current updates for this lab.

If it doesn't happen, then I'll just need to throw an eval version of Windows Server on something else like an HP T740 to run these labs, not the biggest issue for me.

Link for the labs if anyone is curious (free with an email registration like all the evals):

https://www.microsoft.com/en-us/evalcenter/evaluate-mem-evaluation-lab-kit

I'd think direct Docker support would be a higher priority than nested virtualization with a focus on Hyper-V. But that's just me.

@mattnj

[edit] I think you should make a second FOG server for testing the secure boot stuff to prove it out, I'm not going to try this on my production system until I know I'm not going to mess it up [/edit]

I think this is the string of posts:

https://forums.fogproject.org/topic/15888/imaging-with-fog-and-secure-boot-poc/6

If you get this working, I'd really like to know because I'm going to need to got through this for summer refresh on my desktops and want to turn secure boot back on. Sure would have been nice if Microsoft updated WDS to work with win11 so smaller facilities still had a Microsoft method of doing this and not jumping through hoops or buying something expensive.

Do you still have the physical machine? I've had some luck with disk2vhd:
https://learn.microsoft.com/en-us/sysinternals/downloads/disk2vhd

Remember to create a VHD not a VHDx.

If you have the physical machine and it still works, you could also try using Clonezilla on both the physical and virtual machines to transfer the image over the network.

Make the VM but do not boot, simulate as many aspects of the physical machine as possible (ram, drive size, MAC address, etc). This will let you import the VHD into the UUID of the disk you just created, start it up and see what happens. I'm a little foggy on the details, I'd need to walk through this again, but I did get it to work on one of my physical servers when I moved to virtual, one other failed because an application had too many things tied to physical bits of the server and I had to go through support to update it's license on a fresh VM.

All that said, problems with your secure boot are concerning. Are you saying that even a fresh install with secure boot is failing? I've been using the Eval versions of Windows for most of my testing, they should be close enough to the release versions that this should all be the same. Just for fun, I'd suggest downloading the win11 Eval and giving that a try to see if you can create a new VM that works with vTPM, vSecureBoot, vUEFI

@cairoti

I think this may be like my benchmarks, the benchmarks show decent speed to disk, but migration from server to server to local to server are just SLOW.

@cairoti

The Broadcom cards can be a problem, is it possible to swap out for Intel cards?

@cairoti

For me, it is normal to only see 400mbps when migrating from one NFS storage server to another NFS storage server. This is also on a 10gbe network and the drives are fast enough to benchmark a Windows VM up to 6gbps. MTU only 1500.

Under the same storage servers and ESXi8.02, I get faster speeds and I think this is because they use nconnect=4 as the default for NFS connections. I need to do more work with ESXi and the whole vSphere system before rendering firm conclusions, but this might be a thing.

Truenas Scale 24.10.x on both storage servers, both with spinning SATA drives for the array.

@DustinB

Secure Boot may depend on the version of Windows you are using. Education and LTSC don't care right now (could change). As you note, vTPM would be required and vUEFI might be needed too.

The only testing I've done in this area is Windows Server 2025 which is essentially 24h2 win11. But that is only 1 machine, and a fresh install. I did use secure boot, tpm, and uefi to install it.

What I would suggest for the OP is this:

Install the same version of Windows 11 as a clean install into a VM, see if anything stops you from doing this. Install it without secure boot to make sure that works, else if it requires secure boot, you have a bit of a process to get the FOG PXE boot working with secure boot. This is something I know I'm going to have to deal with in the near future, there is a procedure built in a forum post that's hard to find, it can be done but not a simple task.

@Mefosheez

The only experience that I have is Server 2025 Eval with an older Xeon, vtpm, vsecure boot, vuefi and no problems with default configuration using win2022 template and turning on the above mentioned settings.

The only real difference is that I always uncheck automatic boot during creation, go into advanced and change the Intel NIC, save and then initial boot.

I mention that all of the settings are virtual because my old servers do not support hardware uefi or hardware tpm 2.0 so all of that must be synthetic for my lab.

I'm also using up to date XCP 8.3 with this VM.

I have never deleted snapshots before migrating disk from one NFS to a different NFS, this might be different shares on the same physical NAS or shares between two physical NAS.

@flakpyro

Thanks, that will probably save me an hour this summer if I decide to upgrade my production system. By then it might all be fixed.

Well... This is good news and I look forward to when a (stable enough) Alpha or Beta is available for us to test. I might even treat my "big lab" to some newer/larger SSD system drives for this since it will probably need to be a fresh install for the first rounds of testing.

As far as old hardware support, my lab is really the only place I worry about this, my production system is new enough that it should be supported for a while. Made sure that it was good enough for "modern" Windows requirements which is UEFI, Secureboot, and TPM 2.0 hardware.

@flakpyro

I haven't done much testing with it, but delayed start doesn't seem to cause any negative side effects. If you have a large number of services, maybe this could become an issue, but I doubt it.

It was fully supported when I installed it (I think).

If you need it for production, it seems stable enough with the delayed start change. Not sure if other roles or features need this change so I would just be aware of it and proceed.

Also of note, this only delays the management agent from reporting back to the Xen system, it does not delay the drivers from loading. The only complication I could see is booting the VM and immediately trying to migrate it or shut it back down. Normally the MA doesn't do a lot of work so again, delayed is probably fine as I'm seeing delayed is faster than I can log in to look at it.