XCP-ng 8.3 updates announcements and testing

acebmxer

=======================================================================================================================================================
 Package                      Arch                         Version                                          Repository                            Size
=======================================================================================================================================================
Downgrading:
 kernel                       x86_64                       4.19.19-8.0.46.16.xcpng8.3                       xcp-ng-updates                        29 M

Transaction Summary
=======================================================================================================================================================
Downgrade  1 Package

Total download size: 29 M
Is this ok [y/d/N]: y
Downloading packages:
kernel-4.19.19-8.0.46.16.xcpng8.3.x86_64.rpm                                                                                    |  29 MB  00:00:07     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-4.19.19-8.0.46.16.xcpng8.3.x86_64                                                                                            1/2 
depmod: WARNING: /lib/modules/4.19.0+1/kernel/crypto/authencesn.ko needs unknown symbol memcpy_sglist
  Cleanup    : kernel-4.19.19-8.0.49.3.xcpng8.3.x86_64                                                                                             2/2 
  Verifying  : kernel-4.19.19-8.0.46.16.xcpng8.3.x86_64                                                                                            1/2 
  Verifying  : kernel-4.19.19-8.0.49.3.xcpng8.3.x86_64                                                                                             2/2

Removed:
  kernel.x86_64 0:4.19.19-8.0.49.3.xcpng8.3                                                                                                            

Installed:
  kernel.x86_64 0:4.19.19-8.0.46.16.xcpng8.3

Updated:
  forkexecd.x86_64 0:26.1.21-1.4.xcpng8.3             message-switch.x86_64 0:26.1.21-1.4.xcpng8.3   qcow-stream-tool.x86_64 0:26.1.21-1.4.xcpng8.3  
  rrdd-plugins.x86_64 0:26.1.21-1.4.xcpng8.3          sm-cli.x86_64 0:26.1.21-1.4.xcpng8.3           squeezed.x86_64 0:26.1.21-1.4.xcpng8.3          
  varstored-guard.x86_64 0:26.1.21-1.4.xcpng8.3       vhd-tool.x86_64 0:26.1.21-1.4.xcpng8.3         wsproxy.x86_64 0:26.1.21-1.4.xcpng8.3           
  xapi-core.x86_64 0:26.1.21-1.4.xcpng8.3             xapi-nbd.x86_64 0:26.1.21-1.4.xcpng8.3         xapi-rrd2csv.x86_64 0:26.1.21-1.4.xcpng8.3      
  xapi-storage-script.x86_64 0:26.1.21-1.4.xcpng8.3   xapi-tests.x86_64 0:26.1.21-1.4.xcpng8.3       xapi-xe.x86_64 0:26.1.21-1.4.xcpng8.3           
  xcp-networkd.x86_64 0:26.1.21-1.4.xcpng8.3          xcp-rrdd.x86_64 0:26.1.21-1.4.xcpng8.3         xenopsd.x86_64 0:26.1.21-1.4.xcpng8.3           
  xenopsd-cli.x86_64 0:26.1.21-1.4.xcpng8.3           xenopsd-xc.x86_64 0:26.1.21-1.4.xcpng8.3      

Complete!

Andrew

@gduperrey Updates installed. RPU mostly worked... Every host was updated. All VMs moved and returned. Waiting before re-enabling the load balancer task is stuck waiting...

robertblissitt

I did a RPR early this morning just to start fresh and then kicked off an RPU 15 minutes ago. The RPU evacuated all the VMs from the master successfully but the RPU is stuck at 0% (as seen in Pool Tasks). I will allow the process to time out (or restart XO if the wait is too long), promote another node to master, and retry the RPU.

Greg_E

Did my 3 host production pool, only glitches were a few users that were logging in right as vms were migrating. Had a compressed day and had to just Yolo this morning instead of waiting until slower times. Getting paranoid about exploits, and it was Windows patch day too, so just jammed a bunch of stuff together this morning.

vxgmichel

@Andrew said:

Waiting before re-enabling the load balancer task is stuck waiting...

Was it stuck for longer than 30 minutes? If it simply appeared in the task list with a pending status for less than 30 minutes that's actually expected. I ran an RPU on some hosts of my own and made a screenshot for comparison, you can see how the task should look when it's pending and when it's completed.

This task has been introduced in XO 6.7.0 (2026-07-30) by the way, so it's fairly new:

[RPU] Re-enable the load balancer after a configurable safe delay (30 minutes by default) when a rolling pool update ends (PR #10111)

Please let us know if you notice something wrong with it.

vxgmichel

@robertblissitt said:

The RPU evacuated all the VMs from the master successfully but the RPU is stuck at 0% (as seen in Pool Tasks).

Thanks for the report, we have an item in our issue tracker that matches your description. It should be fixed in the upcoming 26.1.23-2.1 xapi version that will be part of the next maintenance update. Please let us know if it keeps happening after the next maintenance update.

julienXOvates

@Andrew could you send us logs over a support ticket so we can understand why the RPR failed ? Thanks

robertblissitt

@vxgmichel I may have found our problem. Looks like an XCP-ng update may have slipped an unwanted site back into our /etc/yum.repos.d/xcp-ng.repo. When I first deployed XCP-ng, I had to remove the "http://mirrors.xcp-ng.org/8/8.3/base/x86_64/" entry for updates to work as it was too troublesome to get it working through our outbound proxy managed by another team. I have re-removed that site and things are working again. It was not fun trying to figure that out in the midst of a vulnerability.

diff /etc/yum.repos.d/xcp-ng.repo /etc/yum.repos.d/xcp-ng.repo.orig
<- baseurl=http://updates.xcp-ng.org/8/8.3/base/x86_64/
-> baseurl=http://mirrors.xcp-ng.org/8/8.3/base/x86_64/ http://updates.xcp-ng.org/8/8.3/base/x86_64/

<- baseurl=http://updates.xcp-ng.org/8/8.3/updates/x86_64/
-> baseurl=http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/ http://updates.xcp-ng.org/8/8.3/updates/x86_64/

stormi

@robertblissitt said:

Looks like an XCP-ng update may have slipped an unwanted site back into our /etc/yum.repos.d/xcp-ng.repo.

Are you sure that this is what happened? XCP-ng updates do not overwrite this file if it has local modifications. It's defined as %config(noreplace) in the RPM spec file for package xcp-ng-release. And no update scriptlets edit it either.

Plus, config files are only updated by rpm when they change, and the file hasn't changed since May 2023.

A few other hypotheses towards the cause:

an upgrade or reinstall via ISO image
a host that never had the edit, added to the pool later
a provisioning tool

robertblissitt

Thank you, @stormi , I'm happy to know that xcp-ng.repo is not touched. These three hosts haven't been rebuilt or restored recently, no hosts have been added, and I did have to make the above modification on each one of them to get them to work. I suppose it's possible I missed the step to remove "mirrors.xcp-ng.org" from xcp-ng.repo during the initial pool build about a year ago and it's just somehow been working with our outbound Internet proxy. In any event, having "mirrors.xcp-ng.org" in our xcp-ng.repo file has definitely caused us problems in the past with our proxy setup.

TGIF/DMCV!