XCP-ng 8.3 betas and RCs feedback 🚀
-
@stormi @ThierryC01 For me my PFSense VMs it get stuck at splash screen. Going from 8.3b to new batch update on 12/21. It does boot up with no error for me just get stuck at autoboot_delay="X". Where X is the time it takes to count down.
Reference to my other post link
Sadly no error so cant tell. Will see how I can get these files never done this before so will be good to know if I can just get it from cmd or XOA and hopefully it is more useful.
/var/log/xensource.log OR /var/log/daemon.log -
@wilsonqanda So your issue is different from that of @archw, whose VMs do not even attempt to boot and fail with a triple fault.
Hopefully you're not stuck. Most of the team will be away for one or two weeks and the remaining ones will only handle urgent support requests
I've created an issue internally so that we investigate this UEFI + pfSense/FreeBSD issue.
-
@stormi Thanks just notice that his situation is more serious than my as I can still move forward by pressing enter on boot screen. His does sound quite different from my situation and really thanks for the update.
HAPPY HOLIDAYS!!! MERRY CHRISTMAS AND HAPPY NEW YEAR
-
Could you try
yum downgrade edk2and then try again to start the VM? -
@stormi I believe this reply is for me as well. Just finish moving the copy of a VM over to a new XCP-ng system and fresh installed 8.3b with latest patch updates. Still fail. Right afterward i added the suggested code below and the countdown work instantly. Thanks!!!
yum downgrade edk2Note: The "edk2" package is related to the UEFI firmware used by the virtualization platform.
This work great for now. Once official release 8.3 non-beta hopefully it will be resolved. A happy early Christmas present for me thanks!
-
So despite dissimilar symptoms, this worked for both of you. We'll investigate this in January then!
-
@stormi I've hold back my fingers, and didn't press the "reboot button".
But, if I need to, will downgrading edk2 make anything break? I guess my pfSense instance will fail after a reboot (like described), so... Hopefully I don't need to reboot my host in the next weeks
? Maybe it's a good idea to note, to not push a update like this, so close to Christmas.Happy Christmas, and Merry New Year
. -
@exetico
edk2was initially updated by XenServer to fix a VNC console corruption when vGPU was in use. As we don't support vgpu at the moment (it requires proprietary components), I doubt going back to the previous known to work version will cause any regressions. But there's always some risk. -
FYI for anyone doing this the snippet of code on a 1 host 1 pool configuration no issue. If you run this code on a multiple hosts in 1 pool configuration make sure to run it all on each host that is affected by the update.
-
@exetico it didnt break for me lol and all 4 of my xcp-ng and multiple pfsense VMs
-
Though as an FYI it is annoying but it show that single upgrade needed in the pool lol
that you just downgrade lol. So best not to upgrade until it is resolved in the 1st place. FYI cant move the vm even to another version of 8.3b (original iso version) that doesn't have the updated 12/21 patches as it's not same version.Obviously after the HOSTs in the pool are updated (patch 12/21) VMs can be migrated but they all need to be on same version.
I thought 8.3b can migrate between all 8.3b of various updates but definitely not the case here. Which I think is a pretty big issues but oh well.
-
@wilsonqanda Stormi found the issue with edk2 and got me running!
-
@archw Thanks for the update yep he got me working as well. It works like a charm
-
New batch of updates for XCP-ng 8.3 beta
So, as several already found out, I pushed many updated packages to XCP-ng 8.3's RPM repositories at the same time. This is the result of the last six months of work on XCP-ng 8.3.
Changes coming from XenServer
This update includes everything published by XenServer between the release of XCP-ng 8.3 Beta1 and October 2023. Newer changes will be brought in after Beta2.
I am a bit late and sadly don't have time now to detail all their changes, but the most important one is maybe completion of vTPM support (and thus compatibility with Windows 11). Previously, it would work, but many features were still missing. Now, snapshots, VM export/import and live migration all are supported for UEFI VMs with a vTPM. A Windows 11 template is available, which will transparently create the required vTPM when you create a new VM.
Changes coming from XCP-ng
On XCP-ng's side, in no particular order:
- We rebased all packages on XenServer 8 preview + all updates up to early October.
- Installer
- rebased on latest upstream release.
- better granularity in error messages displayed to users when install fails due to wrong system date, signature issues, etc.
- re-allow interactive installation of driver disks on host during installation.
- Avoid failed services in the installer (there were benign "failed" messages for services the installer doesn't actually use)
- IPv6 support continued
- IPv6 testing / automated tests. Not run automatically on our internal CI yet due to setup difficulties.
- Fixed live migration failure due to upstream bug in ocaml-uri. Fix contributed upstream to
ocaml-uri. - Allow to use a CIDR for VIFs IPv4 and IPv6 allowed IPs. Feature contributed upstream to XAPI project.
- xsconsole : allow to configure IPv6 via autoconf for the management interface. Feature contributed upstream to XAPI project.
- Making XOSTOR available in XCP-ng 8.3. Another update is pending, so better wait for it.
- Installer image generation: fixes and improvements.
- VLAN display in xsconsole. Feature contributed upstream to XAPI project.
- smartmontools updated to version 7.
- Plugin added to use the data from smartmontools 7 in JSON format (initially contributed by Cécile, then improved by one of our developers)
- Debian 12 template added.
- Security fixes. However, no security fix was made to XCP-ng 8.3 in the last weeks, while we were busy testing and stabilizing.
- Added new tests to our test suite. For example this new test which exercises the vTPM features. Windows 11 is now also tested in our CI/CD.
- Deleted the old, unsupported since XCP-ng 8.1, experimental EXT4 driver. We're talking about an old experimental driver that you never used unless you installed the experimental packages in the XCP-ng 7.x era.
- As in 8.2.1, lsscsi added to our repositories.
- As in 8.2.1, more alternate drivers packaged.
- As in 8.2.1, drivers updated and added to default installation:
mpi3mr,igc,r8125(+firmware) - xo-lite RPM added and installed by default.
- UEFI and SecureBoot support:
- The version of XAPI included in XCP-ng 8.3 after Beta1 provides a new way to handle UEFI certificates at the pool level, implemented by XenServer developers while trying to take both their and our needs into account.
- We then adapted the automated tests for XCP-ng 8.3, based on new XAPI behaviour.
- Running the adapted tests revealed issues (one bug, but also more importantly that we did not fully understand each other regarding XCP-ng's needs). So we proposed a new design to the XAPI project, then implemented it. This hopefully concludes more than a year of intermittent work on this very topic.
- From a user standpoint, despite all the work behind the scene, there aren't many changes. Mostly retaining the ability we had in XCP-ng 8.2.1 for users to download and install the default certificates from Microsoft (must run
secureboot-certs installat least once on the pool after this update), or install their own.
- Various diagnoses and fixes.
Known issues
Several users reported issues with UEFI support. UEFI, again! Just when we thought we were done with it. According to what we found out together, the issue is not related to the changes I was mentioning above, which are limited to certificate management. Here, it's apparently a piece of software emulating a UEFI firmware for VMs,
edk2, which regressed in some situations.So, the current situation, as I know it today:
- For various users, it works, but it looks like pfSense (and maybe FreeBSD) in a UEFI VM has trouble starting.
- In our internal tests, UEFI works for all the OSes we tested. But currently FreeBSD is only tested in BIOS mode, so we wouldn't have seen it. We'll improve our test coverage.
- For one user, it manifested in a way that no UEFI VM would ever start, whatever the OS.
For now, the workaround is to downgrade the component which seems faulty, on every host:
yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3. Note that Xen Orchestra, oryum update, will attempt to reinstall the more recent version of the package, so better avoid updating again until we've solved these issues.Updated installer (XCP-ng 8.3 Beta2)
As soon as any remaining blockers are lifted, this batch of updates will also be released in a XCP-ng 8.3 Beta2 installation ISO image.
You may download a test build of the installation ISOs: https://mirrors.xcp-ng.org/tmp/xcp-ng-8.3.0-beta2-test2.iso
SHA256SUM:
11ed1ca5d757177e0edab6c7a7a371306e3ddcdfab7df62b6f076e527a7a5015/!\ The netinstall repository won't be updated until Beta2 is released, so any network installation using the main mirrors will actually install XCP-ng Beta1, not Beta2.
-
@stormi Is it possible to pull the bad edk2 package or at least for problematic systems send a package update which excludes in yum the bad update package of edk2?
Any way I have suggested to the developers of dnf to implement some kind of compatibility hold system, or possibility of a bad packages list file for repository meta-data. This info would be used by default (though overridable) when using dnf install and dnf update, so bad packages are automatically excluded from installation and/or system updates.
In principal it would operate in a similar fashion to how dandelions hold onto bad seeds, to keep them from spreading. So a bad packages list would be used by rpm repositories and dnf to keep from spreading the bad packages to more systems, thus improving system stability, reliability and/or preventing regressions.
If you wish to give update or assistance on implementing this request upstream you can find the issue here (https://github.com/rpm-software-management/dnf/issues/2029). Also may be worth considering and/or working on transitioning to dnf for XCP-ng as well as updating the appropriate parts of the software stack in the hypervisor, and Xen Orchestra.
-
A Andrew referenced this topic on
-
We already retain a lot of packages until they pass our internal tests successfully, but at some point we need to get feedback from users, and that's what happened here, revealing at the same time that we need to add pfSense to the OSes we test internally. Better find issues now than after the final release, and retaining packages would limit the amount of testing they receive. There's a known workaround for people who are really stuck, so I didn't pull the package back.
-
There is a new build of
edk2available for testing. We don't know yet if it fixes all the issues users met, so we need your feedback.Install with:
yum update edk2 --enablerepo=xcp-ng-testing,xcp-ng-ciIf you need to rollback:
yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3I'm especially interested in @archw's feedback, since their issue was not just with pfSense but with any UEFI VM instead. But I'm also interested in feedback from UEFI pfSense users.
-
@stormi New update version
edk2-20220801-1.7.3.xcpng8.3.x86_64fixed my FreeBSD UEFI boot problem (stuck on boot countdown). XCP reboot NOT required after install. Windows 10 and Ubuntu boot correctly also (but did before too). I did not test secure boot or bitlocker. -
Great news
-
Hey gents,
Has anyone else noticed that if you have a Win 11 guest (pro or enterprise) with the vtpm, that's Entra (Azure) joined, that when migrating to a diff host it breaks the Azure AD membership?
I'll try to dig up some more specifics and run a more thorough test, but I've run into it twice now on a VM I've been running on my test lab with 8.3 beta. Relatively simple solution, leave Azure domain, rejoin, but I suspect that that the token gets saved in vtpm and is either lost or invalidated by migrating hosts.
