Applied latest candidate test updates to 3 x Dell OptiPlex 7040 (i7-6700, 48GB, 10Gbps-attached TrueNAS shared-storage) pool. Update process was error-free and successful. Everything appears to be working normally following the update.

Latest version 8.3 candidate updates installed and are working fine on three-host home lab pool. Received a couple of repo errors for a certain mirror, but yum tried another mirror and it completed successfully. After updates were applied, performed live migrations between hosts with no problems and updated a Windows 11 Version 24H2 VM to the November 2024 cumulative update without problems. (VM is currently running Citrix Tools 9.3.2 without issues.)

Applying the latest XCP-ng 8.3 RC2 "xen-*" and intel-microcode updates from the candidate repository worked great here as well. Everything appears to be running well.

Latest updates over ISO-installed 8.3 RC2 worked fine for me. I did experience one host in my three-host pool to which no VMs could be migrated. After looking at the networking from bash in DOM0, it showed that both 10G ports for the storage and migration networks were DOWN. These ports are on a genuine IBM-branded Intel X540-T2 card I bought used on eBay so it might have gone bad. Since the card has worked well for some time, I figured it couldn't hurt to re-seat it in the PCIe slot. Sure enough, that fixed it. Moral of the story: check the mundane stuff first; it's not always the fault of new updates.

Upgraded 3-node pool (home lab: Dell OptiPlex 7040 SFF x 3) from 8.3 RC1 to 8.3 RC2 using bootable ISO. It worked perfectly for me. As others have noted, it does ask you to select the management interface when upgrading the slave nodes. Once you do that, it automatically populates all of the previously configured network parameters for that host so you are really only confirming the existing values. The OptiPlex 7040's (i7-6700) all have Intel VPro AMT so they are running headless. The MeshCommander program is used to access the VPro console on each host. A DisplayPort display emulator dongle is needed to keep the integrated-GPU active in order to be able to see the console and firmware setup screens with this configuration. It's effectively a poor man's iDRAC. So far, everything is working well on 8.3 RC2.

Serious movement appears to be happening with respect to NV. See videos below cross-posted from this forum thread:

https://xcp-ng.org/forum/topic/8932/hardware-assisted-virtualization-is-not-enabled-on-this-host-even-though-platform-exp-nested-hvm-true-is-set/52

Nested Virtualization (X86) Part I - George Dunlap, Xen Server:
https://www.youtube.com/watch?v=8jKGYY1Bi_o

Nested Virtualization (X86) Part II - George Dunlap, Xen Server:
https://www.youtube.com/watch?v=3MxWvVTmY1s

@abudef Thank you for providing these links to George Dunlap's Xen Summit nested virtualization talk. It was very informative and also demonstrates a strong commitment to bringing NV to Xen Hypervisor and its derivatives. Particularly in light of Broadcom's acquisition of VMware and the resulting customer exodus, adopting XCP-ng and Vates looks to be an increasingly smart play. I will cross-post the provided links to the big NV thread on here.

Applied recent 87 updates to 3-node home-lab pool running XCP-ng 8.3 using XO from source on the latest commit. The update worked perfectly and a mix of existing Linux and Windows VMs are running normally after the update.

@doogie06 Thank you too! I also have XO setup for self-signed TLS access. I knew about the xo-cli task deletion command, but always had to disable TLS in my /root/.config/xo-server/config.toml file and then restart xo-server.service to delete the orphaned tasks. The allowUnauthorized parameter was just what I needed. That's why this community is really wonderful. Thanks again.

@julien-f @olivierlambert I've said it before and I'll say it again: "Wow, that was fast!" You guys do more to foster international cooperation than most of our planet's governments. Keep up the great work. Thank you Vates for creating, maintaining and improving this fantastic software!

@Andrew Sorry, I should have added that it remains broken in the latest master commit 8b7e1.

Using XO from source code. Commit bfb8d3b29e4f9531dda368f6624652479682b69d dated 12 March 2024 @ 11:22:12 broke it. Commit immediately before that 51f95b3c8590492164be38a77ad2c7bf5dc42451 dated 12 March 2024 @ 11:18:20 exporting VM works. Seems to be something related to undici code.

@ajpri1998 Don't despair. Also, don't go crazy trying to make the processor look like a newer one in order to be acceptable to Windows 11. The registry bypasses work. I am personally running an Intel Haswell era laptop: UEFI secure boot, but no TPM, i7-4700HQ, 16GB RAM; it runs Windows 11 Pro perfectly. Believe me, as a long time Microsoft user and enterprise customer, if Satya Nadella didn't want your old box to run Windows 11, it wouldn't. Microsoft provides this workaround so technical users can run the latest Windows until THEY are ready to upgrade to new hardware. My home computing lab is 3 x Dell OptiPlex 7040 SFF eBay used bargains (i7-6700 CPUs) running XCP-ng 8.3 with Xen Kernel 4.17 and a diverse mix of Linux and Windows VMs including Windows 11. It's an evolution to a more secure computing future. We're all on the journey at our own pace. Relax and enjoy it. Use the hardware you have. It's "new enough."

Stats are back in XO after latest XCP-ng 8.3 updates and compiling XO from latest source. That was fast!

@john-c @stormi @olivierlambert All, given the complexities involved in providing an automated solution for this issue, it may make the most sense at this time to mitigate any negative outcomes using an installation/update warning. For instance, throw a message dialog issued by both the ISO installer and the pool update feature in XO such as: WARNING: One or more of the pending updates may alter your VM's firmware. If you are using Microsoft BitLocker to encrypt virtual drives, you should exit now and suspend BitLocker prior to performing this update. (Provide customer with Exit or Continue buttons with Exit selected by default.)

Re: XCP-ng 8.3 beta

Had a couple of Windows 11 VMs setup previously on XCP-ng Beta 8.3 with UEFI Secure Boot, vTPM and BitLocker encryption active. After applying the latest updates, unencrypted UEFI Secure Boot Windows VMs still boot fine, but BitLocker encrypted Windows VMs boot to blue screen and prompt for the BitLocker recovery key. Normally with Windows, an OEM firmware update will trigger this behavior if BitLocker is not suspended prior to flashing the system firmware. As a result, OEM firmware installers generally check for active encryption and suspend it automatically prior to flashing the firmware. Not sure which of the latest updates changed the VM firmware state values, but this could potentially be a huge issue for a production system. In my case, these were just test VMs so no damage was incurred.

@olivierlambert You and the rest of the Vates team are already my Xen heros! I've been running XCP-ng 8.3 Beta on my home lab since August. I was impressed to see the new virtual TPM option in XO this weekend after pulling and compiling the latest source code. (BTW: Windows 11-23H2 BitLocker works flawlessly with the new vTPM support.) XCP-ng and XO truly just keep getting better and better! Hopefully, nested-virtualization of Windows Hyper-V on Xen will get solved before too long since a good variety of capable hypervisor options is important for a healthy and competitive virtualization ecosystem. This is especially true now as we all look to see what Broadcom will do as the new owner of VMware.

@donileo Sadly, yes. No apparent forward movement to date. From the testing I was able to do and also from information passed along by Xen guru Andrew Cooper of Citrix, the problem lies partially with the Xen hypervisor code itself. It therefore requires the applied focus of an expert Xen developer in cooperation with, I think, the XenServer Windows Tools (drivers and management agent) developers. The guest would often hang with Xen drivers installed. The boot hang seemed to get worse with newer versions of Windows. It sometimes would boot and work in a flakey way with a really old Windows version e.g. Server 2008 SP2. This makes some sense intuitively since the Xen bus driver, Hyper-V bus driver and all the rest have to coexist and work together harmoniously. I simply don't have the skills to debug that. My sense is that there is a conflict among the various Windows guest drivers and also more work to be done on nested virtualization in Xen itself. I continue to hold out for a Xen hero that will bring nested-virtualization functional parity to Xen and its derivatives matching that of VMware, Hyper-V and KVM. The recent addition of nascent vTPM support in XCP-ng 8.3 gives me hope that the talent required to do this exists.

@warriorcookie Your characterization is basically correct, but perhaps it should be "closer but no cigar." Masking the hypervisor's presence from the guest is required in all of the other hypervisors to successfully run a Windows guest with nested virtualization enabled. Prior to the discovery of the cited technique, nobody in the community knew how to do it on XenServer/XCP-ng using the xe API. However, the upstream Xen code itself and likely the guest drivers need more work in order for nested virtualization of a Windows guest to work reliably the way it does on ESXi, Hyper-V, etc. With the advent of Windows 11 and Server 2022, a virtualized TPM is also a required feature for full Windows compliance, so Xen has quite a bit on its "to do" list with respect to nested virtualization of Windows.