XCP-ng 8.3 betas and RCs feedback 🚀

manilx

@exetico said in XCP-ng 8.3 betas and RCs feedback :

Remember to update your Xen Orchestra from time to time. I can't really explain why, but I've not thought about that at all. So yesterday I updated from a march 2023 commit, to the most recent one. Not much was changed, but I guess some kind of incompatibilities could be expected, after more than one year without a single update of Xen Orchestra .

Actually there have been HUGE changes in 1 year! You should check them out (only the last weeks big changes to how backups/snapshots work).
Keep it updated and follow Vates' blog to have the news and changes.

olivierlambert

March 2023 means almost 16 releases since, and you can see the amount of change done each release, multiply it by 16…

Here is all the blog posts with the tag release: https://xen-orchestra.com/blog/tag/release/

You missed a LOT

exetico

@olivierlambert I clearly didn't spend much time in the updated interface, yesterday evening ! It was not meant as a "Well, no changes, boring and lazy devs", or anything like that. I really enjoy XO.

I normally update all other things quite often. But I simply didn't think of the fact, that XO was isolated in it's own VM, and living it's own dreams.

@manilx said in XCP-ng 8.3 betas and RCs feedback :

Keep it updated and follow Vates' blog to have the news and changes

Thank you. I'll start following that for news. My limited use of XO does mean that I'm not really digging into the deeper levels. This is a great shortcut for that .

vectr0n

Updated 2 hosts to 8.3 RC1 without any issues and all VM's are functioning correctly. I've been running 8.3 since I moved my personal stuff away from ESXi. Only ran into a few issues in the past with graphs not appearing in XO/XOA but the team was already working on fixes and came thru as always. Keep up the good work XCP-NG/XO teams!

14wkinnersley

Needing some assistance on where to start with the upgrade. I want to upgrade my 3-node cluster. As of right now all 3 nodes are on 8.2.1 but are BIOS boot, and it appears you need UEFI for 8.3 RC1. What is the best way to go about upgrading to where I wont lose data/vm’s? Understand I will most likely need to do a fresh install on each host.. What’s recommended? Should I start with the node that holds the master first on doing the fresh install?

stormi

@exetico said in XCP-ng 8.3 betas and RCs feedback :

Question : What's the "upgrade path" for us starting on 8.3 alpha. Is it just a question about updating the system, and nothing else?
Or, should I plan some kind of operation, so I can "switch" to a stable 8.3 release at some time? Like switching from unstable to a stable branch in Manjaro.

There are instructions in the blog post of the release candidate: https://xcp-ng.org/blog/2024/07/12/xcp-ng-8-3-release-candidate-1/

Currently, updating normally is enough. However, the blog post describes manual changes you have to do (deleting lines in SSH configuration files).

In the end, when the final release is published, then I advise that everyone upgrades using the installation ISO, to clean the system of leftovers of the intermediary test packages. Avoid doing it now, though, because we found a last minute issue specifically on upgrades from 8.3 pre-releases: TLS certificates of the pool are not retained, and this may cut the communications between members of a pool.

stormi

@14wkinnersley

For any upgrade (and in any case), make sure you have good backups, just in case.
The upgrade process doesn't touch your storage repositories nor VM metadata, so your VMs will still be there (but still apply 1.)
You can upgrade even if your boot mode is Legacy BIOS, but the installer will warn you that sooner or later, this won't work anymore. It's easier when you have enough resources to move your VMs around while reinstalling. I'm leaving the details on how to achieve this to others because I'm on vacation :). There's also our technical support to help on this kind of migration.

manilx

@stormi reading https://xcp-ng.org/blog/2024/07/12/xcp-ng-8-3-release-candidate-1/
I found this:

"Current XCP-ng 8.3 users will need to review these configuration files. Make sure to remove any lines starting with Ciphers, MACs, KexAlgorithms, and HostKeyAlgorithms if they are present. This ensures that future changes to the defaults made by our security team will be applied."

Checking /etc/ssh/ssh_config I find these lines at the end:

        Ciphers aes128-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
        MACs hmac-sha2-256,hmac-sha2-512,hmac-sha1
        KexAlgorithms curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
        HostKeyAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

Checking /etc/ssh/sshd_config I find the following in the middle of the file:

# Ciphers, MACs, KEX Algorithms & HostKeyAlgorithms
Ciphers aes128-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-256,hmac-sha2-512,hmac-sha1
KexAlgorithms curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
HostKeyAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

So just to confirm/double check:

I'm supposed to delete those lines as per RC1 info, right?

I don't like to edit system files without having to do so
Thx in advance!

eb-xcp

I'm having issues updating via ISO on supermicro motherboard. I have a netinstall iso mounted (though I've also tried a full ISO via usb), and can get to the initial "install" bootup prompt from 8.3 RC1 image.
Issue is that once "install" option is chosen, "Booting install" shows for about a minute, screen blanks and normal bootup occurs instead. No sign that booting the install has ever occurred.
Do I need to wipe the SSD with the 8.2 and just do a clean 8.3 install? Is there anything I can do to get a more detailed view of logs / or what else could have gone wrong?
If this has been covered, I apologize, I tried reading through 500+ messages in this tread, but nothing stood out to me describing a similar issue.
Supermicro X10SRM-TF motherboard with E5-2699 V4 processor, latest bios, UEFI boot.

If I choose install, or let it run automatically, same results.
iKVM_capture (1).jpg

This is shown for about a minute
iKVM_capture (2).jpg

Than ready to boot prompt flashes for a sec.

And it goes to a normal xcp-ng bootup...
iKVM_capture (3).jpg

stormi

@manilx said in XCP-ng 8.3 betas and RCs feedback :

I'm supposed to delete those lines as per RC1 info, right?

Yes

manilx

@stormi thx! reboot afterwards needed?

14wkinnersley

@stormi Follow up on the steps I took to possibly help others that may have similar scenarios.

Migrated all VM's off of my Master (Node1) and rebooted, upgraded to 8.3 from ISO still on BIOS mode.
Once Node1 was back online, I went ahead and upgraded Node2, and Node3 one at a time. I would just move my current VM's around to the host I wasn't upgrading. Finally had all 3 hosts on 8.3.
Started the UEFI transition. I started with Node1 which was the master, but I changed the master to be on Node3. I went ahead and migrated all the VM's off the Node1, shutdown, pull the HDD (So I would have a backup) and put a new HDD in. Changed my boot mode to UEFI and did a fresh install. I removed Node1 from the cluster in XO. Once back online, I went ahead and just added Node1 back into my cluster and let XO do its magic of bringing everything back online.

Only issue I ran into was a POOL_JOINING_HOST_TLS_VERIFICATION_MISMATCH when I attempted to add the Node1 to the cluster. I followed the steps listed here: https://xcp-ng.org/forum/post/63470

Running xe pool-enable-tls-verification and xe host-emergency-reenable-tls-verification on Node3 which was still the master and was then able to get the new server added to the pool.

Moved the master back to Node1 and started the process over again on Node2 and 3.

yann

@eb-xcp is it something that also happens with the previous alpha/beta ISOs ?

eb-xcp

@yann, No seems Beta 2 works as it should. Though I'm hesitant to go that route since RC1 is available. Is update from Beta 2 to RC1 save / reliable to do via internal updates without ISO?

iKVM_capture (4).jpg

iKVM_capture (5).jpg

eb-xcp

Decided to try updating to beta 2 and... install went smooth, but system now refuses to boot.
Seems issue is similar to RC1 install. Boot fails silently, similar to install on RC1, and next boot option is executed. But while the RC1 failed into a working 8.2 install, 8.3 boot fails into EFI shell.

iKVM_capture (10).jpg

iKVM_capture (12).jpg

eb-xcp

I've tried to do a "clean" beta 2 install, same issue (efi shell).
I've changed bios to legacy boot and was able to run a RC1 installer (nice bootup graphic and all), but it warns you that such method will not be supported:

iKVM_capture (13).jpg

So I ended up using said installer to recover 8.2 install from backup and I'm back up and running at 8.2 (via EFI). Not sure what part of my config could cause issues with 8.3.

Andrew

@stormi The CentOS Base yum repo need to be changed to the baseurl vault as the mirrorlist no longer works...

Houbsi

@Andrew said in XCP-ng 8.3 betas and RCs feedback :

@stormi The CentOS Base yum repo need to be changed to the baseurl vault as the mirrorlist no longer works...

Oh yeah, i wanted to give that info to.

Is there any plan/information on a distro upgrade / distro rebase?

There is from plesk a centos2alma 8 script on the internet, if there is some ressource needed.

olivierlambert

We are aware of that, we discussed the best approach recently.

ha_tu_su

Is XOSTOR supported on 8.3 rc1?