-
@akurzawa I also experienced an issue after the update as well. After the host rebooted and I tried turning on a Windows 10 VM it hung on the Windows boot screen.
-
I eventually force rebooted
-
Force shutdown + Change CPU and RAM (not sure if the resource adjustments helped of not)
-
Start VM and it came up in Windows recovery mode
In the windows recovery mode I just exited and it continued the boot process. This time it booted successfully but I was greeted with the below message upon login.
I had to use XCP-ng center to use keyboard shortcuts to select yes on the dialog box as mouse input was not working. After the restart everything is working as usual again.
I also have the Windows Updates based guest tools enabled on this VM.
@stormi I think this has to do with the new version of guest tools Citrix has pushed from their release of CH8.1. Checking the services they now show major version 9. Not sure if @akurzawa is facing the exact same issue or a variant of it?
-
-
This smells like windows pv drivers upgrades. Had similiar effects on various VMs.
-
FYI: https://support.citrix.com/article/CTX235404
"Important: Updating to this version of the driver removes the quiesced snapshots capability of the VM. If you are using quiesced snapshots and wish to retain this functionality, do not adopt these 9.0.0.x drivers."
They do not tell about crashing VMs though
-
Citrix also has a long history of problematic VM guest-tools. Race conditions with broken drivers, unbootable VMs, broken agents... It's a pain in the a**.
If you like your VMs: Always make a snapshot of every VM before updating the tools.Is known why they remove the VSS-functionality? I don't get why they remove features (again).
-
@stormi said in Updates announcements and testing:
FYI: https://support.citrix.com/article/CTX235404
"Important: Updating to this version of the driver removes the quiesced snapshots capability of the VM. If you are using quiesced snapshots and wish to retain this functionality, do not adopt these 9.0.0.x drivers."
They do not tell about crashing VMs though
I read the support article hoping for more details on how to "...retain this functionality, do not adopt these 9.0.0.x drivers." but the article does not mention that either.
I did not opt to install the new guest tools at any point. I only shutdown the VM to complete the host patches.
Also see the image below, unless it installs guest tools through a side channel that does not show in the Windows updates history.
Would simply disabling the "Windows Update tools" advanced option in XOA/XO stop it from adopting the new guest tools?
-
I have pushed an update to the
netdataandnetdata-uipackages.No need to reboot after installing it even if XO tells you so.
It fixes an upstream issue where in some cases the cache size on disk grows indefinitely without respecting the 256M default limit (reproduced only internally but I don't want to take a risk). Only
netdata-uiis concerned. If you are using netdata through XO, then there's no risk for your hosts. Now netdata will only store 1h of data in RAM, nothing on disk. -
I have pushed an updated
bnxt_endriver asbroadcom-bnxt-en-altfor 8.0 and 8.1. It is inxcp-ng-testingfor 8.0. If fixes a very specific issue from what I can see in the code diff, so no need to install it if everything is already OK for you. -
By the way, is anyone interested (and why) in updated drivers for cisco-enic and cisco-fnic?
-
This post is deleted! -
This post is deleted! -
First update for XCP-ng 8.1:
xcp-ng-depsfor pulling thegpumonpackage. That issue has kept us busy this week when people started reporting installed or upgraded systems that would be unreachable from XO and XCP-ng Center after a reboot. Turns out that XAPI needsgpumon, a package that can only be built using a proprietary (from what I remember. One would need to check the license) nVIDIA developement toolkit. I had removed it from XCP-ng 8.1 because I was convinced that it was only necessary with thevgpufeature with nVIDIA, and we don't have thevgpupackage in XCP-ng because it is closed-source. XAPI's start process stalls without an error message if there's an nVIDIA GPU. Installing gpumon fixes it.- after the update: if you were affected by the issue (you would know), you may need to do an emergency network restart and possibly reconnect the SRs. There may be other consequences for the hosts, such as missing removable media from VMs and possibly others. If you have a way to come back to a backup and reinstall with the fixed ISO (released 2020-04-06), this may be the safest path.
- see also https://github.com/xcp-ng/xcp/wiki/XCP-ng-8.1-Known-Issues#host-unreachable-after-upgrade-or-fresh-installation-on-hosts-having-an-nvidia-gpu
xcp-ng-release*for reducing chrony-wait's timeout from 600s to 120s. So if your host can't connect the ntp server that was configured at installation, you'll only have to wait for 2 minutes, not 10. But your hosts really should be able to connect a ntp server. Accurate time is required to avoid timing and synchronisation issues.
No reboot required if your host is already behaving correctly. If you have a discrete nVIDIA GPU and the host had no issue, then 1. I'm surprised, 2. I advise to reboot.
New installation ISOs including those two updates will be released shortly. 2020-04-06 update: they have been released, named
xcp-ng-8.1.0-2.isoandxcp-ng-8.1.0-2-netinstall.iso.As ususal, see https://github.com/xcp-ng/xcp/wiki/Updates-Howto
-
@stormi Is this related to https://bugs.xenserver.org/browse/XSO-936 (build dependency on gdk-devel) - or a different issue? (I suspect gdk-devel is some part of GNOME, not nVidia proprietary - but could be wrong.)
-
@marekm said in Updates announcements and testing:
@stormi Is this related to https://bugs.xenserver.org/browse/XSO-936 (build dependency on gdk-devel) - or a different issue? (I suspect gdk-devel is some part of GNOME, not nVidia proprietary - but could be wrong.)
Yes it is this issue, and no, it's not GNOME's gdk-devel. Here GDK stands for GPU Deployment Kit. It's unfortunate that someone at some point thought that it would be a good idea to name the dependency gdk-devel, which was already taken by the GNOME project.
-
By the way, if someone wants to have a look at how to build gpumon from its source RPM and prove me wrong about the impossibility to do it only with FOSS tools, I'll gladly revise my judgment
-
@stormi OK, just asked since the issue is about a year old and still unanswered by Citrix, wondering why it didn't show up in earlier xcp-ng releases. Perhaps they have only recently made some change to XAPI so it requires gpumon, and this change could be reverted (or better yet, fixed to handle the missing package gracefully, not hang the whole machine at boot time).
-
@marekm said in Updates announcements and testing:
@stormi OK, just asked since the issue is about a year old and still unanswered by Citrix, wondering why it didn't show up in earlier xcp-ng releases. Perhaps they have only recently made some change to XAPI so it requires gpumon, and this change could be reverted (or better yet, fixed to handle the missing package gracefully, not hang the whole machine at boot time).
It's simpler: it's only with XCP-ng 8.1 that I have removed gpumon. That was a mistake so I had to put it back.
-
@stormi Thanks. Is this https://github.com/xenserver/gpumon - if yes, it doesn't look like something big, perhaps could be patched to remove dependency on proprietary nvidia stuff.
-
Yes, it's this. I've also create an issue for the XAPI project, that got good reception from the devs. All agree that XAPI should be able to start even if gpumon is not there.
-
Note that patching gpumon so that it doesn't depend on nVIDIA stuff probably just means making it a stub service that always answers the same regardless of the actual state of the GPUs.
-
Following yesterday's Xen security updates released by Citrix, here are test packages with security patches. As usual with security matters, they need to be tested quickly so that we can release them to everyone fast.
On XCP-ng 8.1
# on an up to date host yum clean all --enablerepo=xcp-ng-testing yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testingreboot (master first)
On XCP-ng 8.0
This batch of updates contains not only security fixes but also bug fixes that I had queued for the next patch train.
# on an up to date host yum clean all --enablerepo=xcp-ng-testing yum update bugtool-conn-tests qlogic-netxtreme2 qlogic-netxtreme2-4.19.0+1-modules sm sm-rawhba xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenserver-status-report --enablerepo=xcp-ng-testingreboot (master first)
On XCP-ng 7.6
XCP-ng 7.6 is not supported anymore, so there are no updates available for it.
