Updates announcements and testing

  • XCP-ng Team

    @maxcuttins Could you produce a system status report (on both hosts), upload them somewhere and send the link to me (in private if you prefer the host data not to be publicly available)?

  • XCP-ng Team

    So after a look at the logs, we are not sure where the issue comes from. Could be caused by faulty DIMM RAM because there are error messages related to RAM errors (but it's ECC RAM and it claims to have corrected them), but we're not sure.

  • XCP-ng Team

    I have withdrawn the update to xenopsd.

    It does fix live migration from older releases when the VM has no platform:device_id defined (see upstream bug), but causes a transient live migration error during the update when the hosts have different patch levels. If you already installed the update and rebooted your hosts no problem, you're already past the issue. If you installed it but haven't moved your VMs around to reboot your hosts, make sure all your hosts have the same patch level and restart their toolstacks before migrating VMs. If you haven't installed the update but you want to install it, it is still available in the xcp-ng-updates_testing repository. Thanks to lavamind for helping me debug this over IRC.

  • @stormi could you provide some more details about the "transient live migration error" with the xenopsd update?

    I have been experiencing issues live migrating larger VM's (20-40GB Ram) especially when the memory is in active use (eg MariaDB database). As far as I can tell my issue is related to the VM dynamic memory being reduced to below the actual memory use of the VM at which point a random processes crash - but not sure why this is happening since both the source and target XCP host are under allocated on memory. We upgraded all hosts in the pool before xenopsd was withdrawn and are currently scheduling in downtime for each VM so we can migrate them in an off state which seems to bypass the issue.

  • XCP-ng Team

    @Digitalllama The issue you had is probably not related to that update. If that was caused by the update, then you'd have seen the VM reboot at the end of the migration process and it would have been duplicated on both hosts.

    A dynamic memory minimum value set too low can cause the kind of process crash you experienced: the available memory for the VM being low, you may reach a point where the VM's kernel needs to fire the Out Of Memory Killer which will select a running process and kill it. If you want to be sure about that, you can try to set the same value for minimum and maximum dynamic memory and see if the migration issue still occurs.

  • XCP-ng Team

    To people not having updated their hosts yet with the latest update: wait a few more days! There's a kernel security update on its way, so you'll probably want to reboot only then.

    Note that the security update will be mostly useful for people who put their hosts on a network that is reachable from a potential attacker.

  • XCP-ng Team

    The new kernel update candidate is available. As usual, I need some feedback before I can push it to everyone.

    Citrix advisory: https://support.citrix.com/article/CTX256725

    • XCP-ng 7.5: install it with yum update kernel --enablerepo='xcp-ng-updates_testing'
    • XCP-ng 7.6: install it with yum update kernel --enablerepo='xcp-ng-updates_testing'
    • XCP-ng 8.0 beta/RC1: simply yum update

    It is a security update. A distant attacker could manage to crash your host or raise its memory usage significantly with specially crafted network requests. Hosts isolated from public networks are safe, unless the attacker managed to get into your private network.

    Reboot required (we do not support live patching at the moment, due to a closed source component in XenServer / Citrix Hypervisor).

  • XCP-ng Team

    Anyone available to test the security update on 7.5 and/or 7.6? It is a security update, so quite urgent.

  • Installed it a view minutes ago. Will report back.

  • Updated 3 hosts and so far no Problems. Transferred some machines etc no bad effects.

Log in to reply