XCP-ng 8.3 updates announcements and testing

Greg_E

Not going to be able to test this, my little mini-lab is having a very hard time with passthrough, to the point where the host won't boot. Tried this on my backup host (for backup DR testing) and finally got it back after the last couple of hours fooling with turning passthrough on and then back off. Sorry.

stormi

New update candidates for you to test!

As we move closer to making XCP-ng 8.3 the new LTS release, taking over from XCP-ng 8.2.1, a new batch of update candidates is now available for user testing ahead of a future collective release. Details are provided below.

biosdevname: Update as a dependency for another component.
blktap: Fix: enable NBD client only after completing handshake
cyrus-sasl: Fix for CVE-2022-24407 (not directly affecting XCP-ng in normal use).
gpumon: Rebuild for XAPI update.
intel-ice: Update ice driver to v1.15.5
kernel: improve timer handling for better compatibility with hardware.
plymouth: Packaging update. No visible changes.
psmisc: Update to version 23.6.
python-urllib3: Update to version 1.26.30.
rsync:
- Update to version 3.4.1
- Fixes for CVE-2024-12084, CVE-2024-12085, CVE-2024-12086, CVE-2024-12087, CVE-2024-12088, CVE-2024-12747
- The rsyncd configuration and systemd unit files now come in a separate package named rsyncd-daemon, not installed by default
smartmontools: Update to version 7.4
xapi:
- Drop FCoE support when fcoe_driver does not exists
  - FCoE support will be removed from next versions
- No more CPU checks for halted VMs in cross-pool migration
- Move CPU check to the target host during cross-pool migration
- Serialize all PCI and VUSB plugs to keep them ordered
- Fixes multiple issues in periodic scheduler
- Fixes multiple issues in the way XAPI handles RRD metrics
- Improve SR.scan by reducing a racing window when updating the XAPI db
- A lot or maintenance-related changes, XAPI being a very active project.
xcp-featured: rebuilt for XAPI
xcp-ng-release: update copyright years and EULA
xen:
- Improve support for Zen 5 and Diamond Rapids CPUs
- IOMMU logic improvements and fixes
- add PCI quirks for problematic hardware (e.g Cisco VIC UCSX-ML-V5D200GV2)
- fix emulation of MOVBE
xenserver-status-report: maintenance update.
xha:
- Support configurable syslog printing
- Fixes issue where sub-threads can't be scheduled enough resources.

Test on XCP-ng 8.3

From an up-to-date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

biosdevname: 0.3.10-5.xcpng8.3
blktap: 3.55.5-2.1.xcpng8.3
cyrus-sasl: 2.1.26-24.el7_9
gpumon: 24.1.0-40.1.xcpng8.3
intel-ice: 1.15.5-2.xcpng8.3
kernel: 4.19.19-8.0.38.1.xcpng8.3
plymouth: 0.8.9-0.31.20140113.3.xcpng8.3
psmisc: 23.6-2.xcpng8.3
python-urllib3: 1.26.20-3.1.xcpng8.3
rsync: 3.4.1-1.1.xcpng8.3
smartmontools: 7.4-2.xcpng8.3
xapi: 25.6.0-1.4.xcpng8.3
xcp-featured: 1.1.8-1.xcpng8.3
xcp-ng-release: 8.3.0-31
xen: 4.17.5-9.1.xcpng8.3
xenserver-status-report: 2.0.11-1.xcpng8.3
xha: 25.0.0-1.1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

We will not be very available on this forum until Monday to help fixing issues if there are any, so don't update too fast if that's a possible problem for you.*

flakpyro

stormi Updated both of my test hosts. Everything rebooted and came up fine.

No VM stats in XO / XOA i see still. I will be curious if this round of updates fixes my EFI / Windows Server reboot hangs.

stormi

flakpyro No VM stats despite a reboot of the hosts?

abudef

stormi No stats until the toolstack restarts, as with previous candidates

stormi

Ok, let's involve Team-XAPI-Network

Greg_E

stormi

My master host hung on reboot, after about 10 minutes I forced the power off, and then back on. Host is HP T740 Thinclient with AMD v1756b processor, 64GB of DDR4 SODIMM, Intel dual x520 PCIe card, and an Intel i226-v in the a+e slot, plus the onboard Realtek NIC, BIOS at 1.20 which I think is still current for this model. All three hosts are identical with possible exception of x520 card revisions, host 3 might have an older revision (donations accepted for x710 based cards )

Otherwise everything went as planned, all three are updated and all three needed to have the toolstack restarted to see the stats. I only have 2 small Linux VMs on this system right now, and both of them started fine.

There were no stats in XO-Lite either, I was doing the second and third hosts from XO-Lite to see if XO was getting mad. The second and third rebooted without issue and something I'll look into with the next update, the delay on master could have been the VMs trying to auto start where I remembered to turn that off for the other two hosts.

bufanda

Installed on my Test pool with 2 HP EliteDesk 800 G3 mini. Except the no stats in XenOrchestra. Did some VM migragration between hosts, reboot testes (only Linux VMs though) and so far no issues.

andriy.sultanov

abudef Greg_E bufanda

Could you please attach /var/log/{xensource.log,daemon.log,xcp-rrdd-plugins.log} from the time after the reboot and before the toolstack restart?

Also: does this reproduce if you reboot again - would you still have no metrics until toolstack restart? Because if not, this is entering paranormal territory

UPD: I've just updated xapi from 24.39 to 25.6 myself - stats are working after the reboot and I can see them in XenOrchestra, no additional toolstack restart required.

abudef

andriy.sultanov said in XCP-ng 8.3 updates announcements and testing:

does this reproduce if you reboot again

Yes, this does. Logs have been provided via pm.

bufanda

andriy.sultanov
Same issue after another reboot. The stats on host just flatline. No CPU usage or anything.
See screenshot

I attached log files
daemon.log.txt xcp-rrdd-plugins.log.txt xensource.log.txt

andriy.sultanov

bufanda Thanks!
Hmm, can't see anything suspicious either in your logs or abudef's

Since you said it reproduces if you reboot the host again, I'd really appreciate if you could send the result of rrd2csv running for 30 seconds or so on a system that doesn't have the stats working - so before any toolstack restarts.

bufanda

andriy.sultanov
I ran it for a minute.

rrd2csv.txt

Greg_E

andriy.sultanov

If still needed, I can probably get you the rrd2csv next week, I need to unrack my system and move it to a new mobile rack. A large expense that I'm not happy to have to do, but now needed going forward (it's a work thing, where I prototype workflows for my production system)

andriy.sultanov

Greg_E Thanks, but that will not be necessary - I think I've figured out where the problem lies now. Good luck with the move

Greg_E

andriy.sultanov

The move is more about spending yet more personal money for something that is primarily used for learning what I need to know for work (a rack, UPS, and some odds and ends at $800). Or half of it is, half of it is a VMware system and that's to let me learn what I need to leave this job and find one that pays more. And then move that work to something easier to use that costs less. Still amazed at how many places just took the price increase and are still not making plans to move to something else, even if they are cutting core counts in half to save half of that new money. But I'm also seeing that the general trend in IT around where I live is to get into a Silo, and never ever take on another task to fill a need. This way you never want to move laterally to other products, just keep doing the same things the same ways until management gets tired of hearing "we can't do that" and fires everyone to replace them with an MSP (or other contractor).

So for the off topic rant, feeling salty again today.