XCP-ng 8.1 Release Candidate now available!

[stormi]

Hello to everyone.

The Release Candidate (RC) release of XCP-ng 8.1 is available right now .

This post is a mostly a copy-paste from the beta release announcement, with additions regarding the RC release.

Stability status

Good as far as we call tell from our tests and beta feedback. I would personally use it in production (actually we probably will very soon at Vates). The main unknown is how well the new coalesce algorithm will perform. Now is the time for as large testing as possible before the final release.

What changed

Here's what changes from XCP-ng 8.0. Changes since the 8.1 beta are in bold characters.

Non-exhaustive list:

• Based on Citrix Hypervisor 8.1 (https://docs.citrix.com/en-us/citrix-hypervisor/whats-new.html).
• Same base version of CentOS (7.5)
• Same base version of kernel (4.19, with more patches of course)
  • Latest kernel hotfix from CH 8.1 included in the RC release.
• Xen 4.13
• UEFI support for guests is not experimental anymore
• Support for Secure Boot is NOT INCLUDED because it relies on Proprietary packages. We're raising the issue with the Xen community to see how to provide a FOSS implementation.
• Citrix announces:
  • "Improved performance for VM imports and exports that use the XVA format" thanks to the use of a very efficient hash algorithm
  • "Storage performance improvements"
  • "New Windows I/O drivers with improved performance"
  • (From https://bugs.xenserver.org/browse/XSO-951) The next update (apparently not published yet as of 2020-01-27) of the Windows Drivers through Windows Update should at last be compatible with non-english locales!
• chrony replaces ntp
• Support for PV guests is now deprecated
  • templates for creating PV guests have been removed
  • existing guests will still run
  • it is advised to convert them to HVM guests
  • a compatibility layer should be provided in the future for PV guests that really can't be converted. But really anyone who can convert to HVM, should
  • due to how 32-bit PV guests work, keeping it functioning on newer hardware with newer features comes with an increasing performance cost, and the linux kernel is about to drop support for 32-bit PV guests
• Support for AMD EPYC 7xx2(P) added
• Dynamic Memory Control (DMC) is deprecated and will be removed in the future.
• VSS and quiesced snapshots support is removed, because it never worked correctly and caused more harm than good.
  • For backups, we are working on support for backups that include the guest RAM to replace the need for quiesced snapshots.
  • Note that Windows guest tools version 9 (the default for recent versions of Windows if you install through Windows Update) already removed VSS support, even for older versions of CH / XCP-ng
• For new local storage repositories using the EXT filesystem, it now defaults to ext4.
  • The transition from our experimental ext4 storage driver is not automatic.
  • Instructions for transitioning written in specific section below.
  • sm-additional-drivers package updated to refuse to create new experimental ext4 SRs since ext4 is already the default with the ext driver.
• Status of our specific packages:
  • Experimental support for XFS in local storage repository still available through the sm-additional-drivers package.
  • ZFS still available as an additional package and updated to 0.8.3.
  • Alternate kernel now available, version 4.19.102 (probable update to 4.19.108 before final release). Installing it now automatically adds a new boot entry in grub's configuration, to make testing easier. Default entry remains that of the main kernel.
  • netdata RPMs are now ready
    • @r1 contributed a fix to netdata project to bring support for Xen 4.13
    • @stormi made netdata cache be RAM-only to workaround an upstream bug that could make the disk cache grow forever
• Plans are laid out for simpler installation and maintenance of Windows guest tools. Unfortunately, we haven't found resources yet to implement them so the current state remains that of 8.0. If you're a developer on the Windows platforms, we're hiring! (full time or part time, contracts or hires) - Contact us.
• Installer improvements in 8.1 RC: see section below.
• New leaf coalesce logic of VDIs with dynamic limits: see section below.
• Fixed VM Autostart issues in 8.1 beta when updated from 8.0 through yum (upstream bug https://bugs.xenserver.org/browse/XSO-978)
• Fixed netxtreme drivers (bnx2x module) that crashed with some models
• Updated zstd to 1.4.4. Possibly even better perfs according to upstream. Still no import/export comparative benchmarks to 8.0. Does anyone volunteer?

Installer improvements in 8.1 RC

Our installer now offers two new installation options. In BIOS mode, access them with F2 when offered the choice. In UEFI mode, see the added boot menu entries.

• First new option: boot the installer with a 2G RAM limit instead of the 8G default. This is a workaround for installation issues on hardware with Ryzen CPUs. Though those are Desktop-class CPUs and not supported officially in the HCL, we tried to make it easier to workaround the infamous "installer crashes on Ryzen" issue.
• Second new option: boot the installer with our alternate kernel (kernel-alt). That kernel, built and maintained by @r1 for the team, is based on the main kernel, with all upstream kernel.org patches from the LTS 4.19 branch applied. It should be very stable by construction but it receives less testing. That option is there for cases when the main kernel and drivers have issues, so that you can quickly test if kernel.org patches have fixed it already. It will also install the alternate kernel in addition to the main kernel as a convenience. If kernel-alt fixes issues for you, the most important thing to do is to tell us so that we may fix the main kernel!

Announcement about our former experimental ext4 SR driver

It is now deprecated in 8.1. For a good reason: in XCP-ng 8.1 and above, following upstream changes, the ext driver now formats new SRs as EXT4. Existing SRs are untouched (so remain formatted to EXT3).

There is no easy way to convert an existing SR created with our driver, so those using it will need to move the VDIs out (to another SR or to export them), destroy the SR and create an EXT SR instead. Make sure to do this on XCP-ng 8.1.

The sm-additional-drivers package remains available in XCP-ng 8.1 in order to ease the transition. However I've broken the sr-create command on purpose. Any attempt to create a SR of type ext4 will result in an error with a message that explains that you need to use the ext type instead.

Our experimental driver will be completely removed in a later release, possibly XCP-ng 8.2. Unless someone convinces me to delay the removal for a good reason. I will accept reasons such as "I know I shouldn't have used the experimental driver in production, but I did and need more time to convert my SR while at the same time I really need feature xxx from XCP-ng 8.2", but I really would prefer to drop it in 8.2.

Feedback from people doing the transition is welcome to make sure we document the transition in the best way possible.

New leaf coalesce logic with dynamic limits

I have backported patches from sm's master branch, that implement a new, smarter, logic for leaf coalescing.

Those interested in the patches, see https://github.com/xcp-ng-rpms/sm/commit/ed1a55d727846cf5777c8258e6a8f3b068e8a35b (python code).

Feedback wanted.

Documentation

At this stage you should already be aware that our main documentation is in our wiki and you should also know that you can all take part in completing it. It's improving continuously and still has room for improvement.

How to upgrade from XCP-ng 8.1 beta

Just use yum update or Xen Orchestra as if you were installing updates to a stable release.

How to upgrade from previous releases

Since XCP-ng 8.1.0 is a minor release, both upgrade methods are supported:

• From the installation ISO
• From command line using yum (from XCP-ng 8.0 only!)

Refer to the upgrade howto: https://github.com/xcp-ng/xcp/wiki/Upgrade-Howto

Downloads:

• Standard ISO: http://mirrors.xcp-ng.org/isos/8.1/xcp-ng-8.1.0-RC1.iso
• Net-install ISO: http://mirrors.xcp-ng.org/isos/8.1/xcp-ng-8.1.0-RC1-netinstall.iso
• SHA256SUMS: http://mirrors.xcp-ng.org/isos/8.1/SHA256SUMS
• Signatures of the sums: http://mirrors.xcp-ng.org/isos/8.1/SHA256SUMS.asc

If your browser tells you that the page is redirected in a way that prevents from loading the page correctly, this is a known issue. Long story short, it's caused by a wrong security header sent by the forum software, which impacts all non HTTPS requests to the xcp-ng.org domain. Try in Private Navigation mode or use curl or wget.

It's a good habit to check your downloaded ISO against the SHA256 sum and for better security also check the signature of those sums. Although our mirror redirector does try to detect file changes on mirrors, it's should always be envisioned that a mirror (or in the worst case our source mirror) gets hacked and managed to provide both fake ISOs and SHA256 sums. But they can't fake the signature. See https://github.com/xcp-ng/xcp/wiki/How-to-check-the-authenticity-of-files-downloaded-from-XCP-mirrors.

Stay up to date

Run yum update regularly. We'll fix bugs that are detected regularly until the final release. Subscribe to this thread (and make sure to activate mail notifications in the forum parameters if you need them): we'll announce every important update to the RC here.

What to test

Everything and Anything!

Report or ask anything as replies to this thread.

A community effort to list things to be tested has been started at https://github.com/xcp-ng/xcp/wiki/Test-XCP

Feedback is welcome both about successes and failures.

0 stormi committed to xcp-ng-rpms/sm

New leaf coalesce logic with dynamic limits

Patches backported from master

[Biggen]

So how do you destroy and create a SR with no controlling XO(A) in place?

Do you guys have a written tutorial as there are lots of us running that ext4 driver?

[stevewest15]

Hi,

Is there any harm to leaving the former experimental EXT4 SR driver installed for anyone who has it installed? I understand and agree with sr-create command for ext4 will fail for new SR. I think this might be a better solution for us who used the experimental ext4 and don't want to move, destroy and copy back all VMs, etc.

[stevewest15]

@Biggen You can do this via cli...see if this doc is helpful (just make sure to back/move/copy any data before performing these steps): https://support.citrix.com/article/CTX131328

[Biggen]

@stevewest15 said in XCP-ng 8.1 Release Candidate now available!:

@Biggen You can do this via cli...see if this doc is helpful (just make sure to back/move/copy any data before performing these steps): https://support.citrix.com/article/CTX131328

Thanks for that!

I’ll wait until we official move off the RC and onto a production build before I do this. Hopefully the Vates team puts together a short tutorial that give us a specific upgrade workflow.

[mortenchristensn]

Fresh install of RC1 on a test-machine.
Not part of a pool and no SAN. Just local storage on same nvme-ssd, XCP-ng is installed on.
I choose EXT in the install-process.
After install
file -sL /dev/nvme*
says, that the storage is ext3.

Is EXT4 a possibility on Local Storage, or what is the right choise for Local Storage?

[mortenchristensn]

The homepage welcome-screen on a 8.1.0 server point to download of XCP-ng Center 8.0.1.26.
That xcp-ng-center version cannot connect to a 8.1 server.

[stormi]

@mortenchristensn I'm not sure that your test for the file type is valid. XCP-ng first creates a LV on the device then creates the ext4 filesystem in that. I assume the ext3 filesystem you found is that of the dom0 root, not that of the SR.

Here's how you can check that the local SR is ext4:

[12:36 xcp-ng-fsspniqy ~]# mount | grep XSLocalEXT
/dev/mapper/XSLocalEXT--963b005a--a685--4a5f--31c0--af9a0bd98748-963b005a--a685--4a5f--31c0--af9a0bd98748 on /run/sr-mount/963b005a-a685-4a5f-31c0-af9a0bd98748 type ext4 (rw,relatime)
[12:37 xcp-ng-fsspniqy ~]# file -sL /dev/mapper/XSLocalEXT--963b005a--a685--4a5f--31c0--af9a0bd98748-963b005a--a685--4a5f--31c0--af9a0bd98748
/dev/mapper/XSLocalEXT--963b005a--a685--4a5f--31c0--af9a0bd98748-963b005a--a685--4a5f--31c0--af9a0bd98748: Linux rev 1.0 ext4 filesystem data, UUID=0a1c1c21-afc4-4187-844f-7554f3f95953 (needs journal recovery) (extents) (64bit) (large files) (huge files)

[stormi]

@stevewest15 said in XCP-ng 8.1 Release Candidate now available!:

Hi,

Is there any harm to leaving the former experimental EXT4 SR driver installed for anyone who has it installed? I understand and agree with sr-create command for ext4 will fail for new SR. I think this might be a better solution for us who used the experimental ext4 and don't want to move, destroy and copy back all VMs, etc.

We leave it for 8.1 in the sm-additional-drivers package. But I want to remove it for future releases. It adds maintenance overhead to maintain it. If we had to maintain forever any experiment that we do then we would have to stop experimenting. That storage driver was always advertised as experimental. I wanted to provide an easy transition and I did work on it despite it being experimental so not intended for production, but it's really not simple to do and would be risky for your data.

[stormi]

@mortenchristensn said in XCP-ng 8.1 Release Candidate now available!:

The homepage welcome-screen on a 8.1.0 server point to download of XCP-ng Center 8.0.1.26.
That xcp-ng-center version cannot connect to a 8.1 server.

Thanks. I think that's because there's no officially stable release of XCP-ng Center out yet due do the fact that @borzel is alone to maintain is and doesn't have time for it. Either there will be a stable release of the center before our final release, or we'll update the welcome page to point to the latest dev release (with a warning).

[olivierlambert]

We can also link the GitHub repo directly

[borzel]

@olivierlambert I would prefer that

[dariosplit]

"Existing SRs are untouched (so remain formatted to EXT3)" How to convert existing Local storage? I upgraded to 8.1, but Local storage is Ext3.

[stormi]

@dariosplit said in XCP-ng 8.1 Release Candidate now available!:

"Existing SRs are untouched (so remain formatted to EXT3)" How to convert existing Local storage? I upgraded to 8.1, but Local storage is Ext3.

Quoting further:

There is no easy way to convert an existing SR created with our driver, so those using it will need to move the VDIs out (to another SR or to export them), destroy the SR and create an EXT SR instead. Make sure to do this on XCP-ng 8.1.

So is your question about how to create a SR and how to destroy one? What about @Biggen's link shared above?

[mortenchristensn]

@stormi said in :

I'm not sure that your test for the file type is valid.

You are right. Running your test on my server gives same answer
Linux rev 1.0 ext4 filesystem data

[mortenchristensn]

@stormi said in :

So is your question about how to create a SR and how to destroy one? What about @Biggen's link shared above?

The link is not telling, how to change the Local Storage.
As far as I can see, Local Storage is not a partition like SR's on other physical disks.

[peder]

@mortenchristensn Local storage is an SR.

The commands below will remove the SR named "Local storage" and create a new one with the same name on the same device

If you have a pool with several hosts you need to run these commands on each host and, as stormi points out in the comment below, you need to move any VMs out of the Local SR before removing it.

# Get the IDs and device path to the current Local SR :
# xe host-list | grep -B1 $HOSTNAME | grep ^uuid| cut -d\: -f2 | cut -d" " -f2 > HostUuid.txt
# xe pbd-list sr-name-label=Local\ storage host-uuid=`cat HostUuid.txt` | grep sr-uuid | cut -d\: -f2 | cut -d" " -f2 > SR-Uuid.txt
# xe pbd-list sr-name-label=Local\ storage host-uuid=`cat HostUuid.txt` | grep ^uuid | cut -d\: -f2 | cut -d" " -f2 > Uuid.txt
# xe pbd-list sr-name-label=Local\ storage host-uuid=`cat HostUuid.txt` | grep device-config| cut -d\: -f3 | cut -d" " -f2 > Device.txt
# Unplug and destroy the Local SR : 
# xe pbd-unplug uuid=`cat Uuid.txt | egrep "*-*-*-*-*" `
# xe sr-forget uuid=`cat SR-Uuid.txt | egrep "*-*-*-*-*" `
# Create the new Local SR : 
# xe sr-create content-type=user device-config:device=`cat Device.txt` host-uuid=`cat HostUuid.txt` name-label=Local\ storage shared=false type=ext

[stormi]

Of course, move the VMs out of the SR before destroying it

[dariosplit]

@peder Dosen't work. My version is 8.1 Build date: 2020-02-19 and reciveed the resoult " Name: Local storage Type: Ext3."

[Danp]

@dariosplit What exactly doesn't work? Where did you receive that output?