XCP-ng 8.3 betas and RCs feedback 🚀

[14wkinnersley]

Needing some assistance on where to start with the upgrade. I want to upgrade my 3-node cluster. As of right now all 3 nodes are on 8.2.1 but are BIOS boot, and it appears you need UEFI for 8.3 RC1. What is the best way to go about upgrading to where I wont lose data/vm’s? Understand I will most likely need to do a fresh install on each host.. What’s recommended? Should I start with the node that holds the master first on doing the fresh install?

[stormi]

@exetico said in XCP-ng 8.3 betas and RCs feedback :

Question : What's the "upgrade path" for us starting on 8.3 alpha. Is it just a question about updating the system, and nothing else?
Or, should I plan some kind of operation, so I can "switch" to a stable 8.3 release at some time? Like switching from unstable to a stable branch in Manjaro.

There are instructions in the blog post of the release candidate: https://xcp-ng.org/blog/2024/07/12/xcp-ng-8-3-release-candidate-1/

Currently, updating normally is enough. However, the blog post describes manual changes you have to do (deleting lines in SSH configuration files).

In the end, when the final release is published, then I advise that everyone upgrades using the installation ISO, to clean the system of leftovers of the intermediary test packages. Avoid doing it now, though, because we found a last minute issue specifically on upgrades from 8.3 pre-releases: TLS certificates of the pool are not retained, and this may cut the communications between members of a pool.

[stormi]

@14wkinnersley

1. For any upgrade (and in any case), make sure you have good backups, just in case.
2. The upgrade process doesn't touch your storage repositories nor VM metadata, so your VMs will still be there (but still apply 1.)
3. You can upgrade even if your boot mode is Legacy BIOS, but the installer will warn you that sooner or later, this won't work anymore. It's easier when you have enough resources to move your VMs around while reinstalling. I'm leaving the details on how to achieve this to others because I'm on vacation :). There's also our technical support to help on this kind of migration.

[manilx]

@stormi reading https://xcp-ng.org/blog/2024/07/12/xcp-ng-8-3-release-candidate-1/
I found this:

"Current XCP-ng 8.3 users will need to review these configuration files. Make sure to remove any lines starting with Ciphers, MACs, KexAlgorithms, and HostKeyAlgorithms if they are present. This ensures that future changes to the defaults made by our security team will be applied."

Checking /etc/ssh/ssh_config I find these lines at the end:

        Ciphers aes128-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
        MACs hmac-sha2-256,hmac-sha2-512,hmac-sha1
        KexAlgorithms curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
        HostKeyAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

Checking /etc/ssh/sshd_config I find the following in the middle of the file:

# Ciphers, MACs, KEX Algorithms & HostKeyAlgorithms
Ciphers aes128-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-256,hmac-sha2-512,hmac-sha1
KexAlgorithms curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
HostKeyAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

So just to confirm/double check:

I'm supposed to delete those lines as per RC1 info, right?

I don't like to edit system files without having to do so
Thx in advance!

[eb-xcp]

I'm having issues updating via ISO on supermicro motherboard. I have a netinstall iso mounted (though I've also tried a full ISO via usb), and can get to the initial "install" bootup prompt from 8.3 RC1 image.
Issue is that once "install" option is chosen, "Booting install" shows for about a minute, screen blanks and normal bootup occurs instead. No sign that booting the install has ever occurred.
Do I need to wipe the SSD with the 8.2 and just do a clean 8.3 install? Is there anything I can do to get a more detailed view of logs / or what else could have gone wrong?
If this has been covered, I apologize, I tried reading through 500+ messages in this tread, but nothing stood out to me describing a similar issue.
Supermicro X10SRM-TF motherboard with E5-2699 V4 processor, latest bios, UEFI boot.

If I choose install, or let it run automatically, same results.

This is shown for about a minute

Than ready to boot prompt flashes for a sec.

And it goes to a normal xcp-ng bootup...

[stormi]

@manilx said in XCP-ng 8.3 betas and RCs feedback :

I'm supposed to delete those lines as per RC1 info, right?

Yes

[manilx]

@stormi thx! reboot afterwards needed?

[14wkinnersley]

@stormi Follow up on the steps I took to possibly help others that may have similar scenarios.

1. Migrated all VM's off of my Master (Node1) and rebooted, upgraded to 8.3 from ISO still on BIOS mode.
2. Once Node1 was back online, I went ahead and upgraded Node2, and Node3 one at a time. I would just move my current VM's around to the host I wasn't upgrading. Finally had all 3 hosts on 8.3.
3. Started the UEFI transition. I started with Node1 which was the master, but I changed the master to be on Node3. I went ahead and migrated all the VM's off the Node1, shutdown, pull the HDD (So I would have a backup) and put a new HDD in. Changed my boot mode to UEFI and did a fresh install. I removed Node1 from the cluster in XO. Once back online, I went ahead and just added Node1 back into my cluster and let XO do its magic of bringing everything back online.

Only issue I ran into was a POOL_JOINING_HOST_TLS_VERIFICATION_MISMATCH when I attempted to add the Node1 to the cluster. I followed the steps listed here: https://xcp-ng.org/forum/post/63470

Running xe pool-enable-tls-verification and xe host-emergency-reenable-tls-verification on Node3 which was still the master and was then able to get the new server added to the pool.

4. Moved the master back to Node1 and started the process over again on Node2 and 3.

[yann]

@eb-xcp is it something that also happens with the previous alpha/beta ISOs ?

[eb-xcp]

@yann, No seems Beta 2 works as it should. Though I'm hesitant to go that route since RC1 is available. Is update from Beta 2 to RC1 save / reliable to do via internal updates without ISO?

[eb-xcp]

Decided to try updating to beta 2 and... install went smooth, but system now refuses to boot.
Seems issue is similar to RC1 install. Boot fails silently, similar to install on RC1, and next boot option is executed. But while the RC1 failed into a working 8.2 install, 8.3 boot fails into EFI shell.

[eb-xcp]

I've tried to do a "clean" beta 2 install, same issue (efi shell).
I've changed bios to legacy boot and was able to run a RC1 installer (nice bootup graphic and all), but it warns you that such method will not be supported:

So I ended up using said installer to recover 8.2 install from backup and I'm back up and running at 8.2 (via EFI). Not sure what part of my config could cause issues with 8.3.

[Andrew]

@stormi The CentOS Base yum repo need to be changed to the baseurl vault as the mirrorlist no longer works...

[Houbsi]

@Andrew said in XCP-ng 8.3 betas and RCs feedback :

@stormi The CentOS Base yum repo need to be changed to the baseurl vault as the mirrorlist no longer works...

Oh yeah, i wanted to give that info to.

Is there any plan/information on a distro upgrade / distro rebase?

There is from plesk a centos2alma 8 script on the internet, if there is some ressource needed.

[olivierlambert]

We are aware of that, we discussed the best approach recently.

[ha_tu_su]

Is XOSTOR supported on 8.3 rc1?

[olivierlambert]

No yet, we prioritize 8.2 so far. But it will come, don't worry It's mostly packaging work and some subtle difference, but it's not that big. Just a matter of what to do first

[ha_tu_su]

@olivierlambert
Thanks for the info. No hurry. Better to do it right than fast.

[olivierlambert]

[eb-xcp]

Ok, update on the update issues:
I could not migrate VMs from the older backup server which was upgraded to 8.3 RC1 back to 8.2 install, so I'm forced with having to continue troubleshooting it.
Today I tried to start a CD install from efi shell and I finally got an error / output as to why install fails silently:

"This build of Xen requires NX support"

I guess I had it disabled in the BIOS, just wish the install would have output the error with a wait prompt or something, rather than a silent failure.

Edit: Confirmed; after enabling execution disable option within bios, installer booted without issues and the install is currently ongoing.