Windows Server 2016 & 2019 freezing on multiple hosts

borzel

@olivierlambert nope, the lastest tag on https://xenbits.xen.org/gitweb/?p=pvdrivers/win/xenvif.git;a=summary is 8.2.1 (8 months ago)

This is a situation, where Citrix (maybe) publishes it's own drivers... or the repo is hidden elsewhere.

borzel

@olivierlambert I also checked the source disks, they are from Sep 6, 2018 ... no luck for us

olivierlambert

Weird, so they didn't updated the sources? Or maybe the driver from Sep 6 is already more recent? Is there a way to check this? (version of VIF in open source drivers vs Citrix driver?)

borzel

@olivierlambert I assume someone could do binary analysis .. but thats not my field of expertise

olivierlambert

I mean more simply: install latest Xen tools (so ours should be OK I assume), display the VIF driver number, then do the same on Citrix driver and compare the VIF driver number.

borzel

@olivierlambert the first three digits of the version number are the same, the last is usually the buildnumber... I assume they backportet some codechanges from master ... but this is nothing we can easily detect

Edit: maybe somone with IDA can compare the last two version an do a (graphical -> flowchart?) diff?

Edit2: I try to check the git logs in the evening ... maybe some change pop's up

borzel

I asked for help: https://bugs.xenserver.org/browse/XSO-928

michael

@300cpilot

I'm going to try the remote event viewer over the next few days to see if anything of interest comes up. As for remote desktop, and ping I get no response.

@olivierlambert @borzel

I'll pull the latest builds and see if I have any luck. I'm also going to pull the latest Windows Server updates on a few of the VMs (seems they just came out a few hours ago as I checked for updates earlier and had nothing)

borzel

As you can read here: https://bugs.xenserver.org/browse/XSO-928 there is no legal chance to get the changes made by CITRIX, because the originating code is BSD licensed

olivierlambert

Yeah but the Open source drivers must be updated somehow, because there is some people using Xen out there with Windows load (AWS? IBM Rackspace?)

borzel

@olivierlambert

The OpenSource Drivers from the XEN-Project (https://www.xenproject.org/downloads/windows-pv-drivers/winpv-drivers-8/winpv-drivers-821.html) are the same version like ours: https://github.com/xcp-ng/win-pv-drivers/releases/tag/v8.2.1-beta1

Maybe IBM or Rackspace uses own builds?

michael

Quick Update

I haven't had any crashes since posting here. Before posting here it was each of my 2016 & 2019 VMs on XCP-NG that would lock up. Now it's none of them. Some of them I updated, while others I did nothing... Strange.

I'll update this post again in a few days, or sooner if anything changes.

olivierlambert

Thanks a lot for your feedback @michael !

michael

I had one lock up on me. Surprising the one with the shortest uptime.

EDIT: I just updated the xcp-emu-manager-0.0.9-1 that was posted about a bit ago. Not sure if this will help or not.

Have you heard of anyone else having these issues? I'm debating on doing a fresh install of XCP-NG to see if that helps.

EDIT 2: There are no error logs given in Windows. One minute it's working, the next it's not. I'm going through the XCP-NG logs at the moment to see if there is anything there.

EDIT 3: Here is a list of every line with the UUID of the VM associated with it: https://pastebin.com/ip30uyMN

borzel

@michael can you point to a specific time in the log file when your VM was locked up?

michael

@borzel Sometime between the 7:40 and 12:12 on the first line. I'm guessing it went down right as I was posting here.

Like I said whatever it's doing it doesn't seem to log it. Is there any other log files I should check?

EDIT: I have a theory, testing it now.

EDIT 2: I'm wondering if this has something to do with XCP-NG Center. The KMS server (the one that locked up) was the only one I logged into when checking if everything was working. Trying to duplicate this now.

michael

Update: The same VM crashed again, the others are working fine as of this post. I left it up as a remote desktop session last night and I came back to my office today and it was still up as a RDP session, but not responsive. I checked XCP-NG Center and the same blue screen with a cursor was there.

Still trying to find the cause of this.

etlweather

Were you able to resolve this in the end?

apayne

@michael this may sound silly, but perhaps this isn’t a software issue? Maybe you have a faulty stick of RAM hiding in the machine? Bad RAM will make all kinds of strange and flaky things happen. This is just a guess, I’m just putting this idea out there because I’ve seen similar weird behavior in machines where RAM failure isn’t easy to spot (no front panel on the server with a fault light).

nikade

@apayne This is a good suggestion, all tho it is happening on a lot of different physical machines if im not mistaken.

Im running a couple of Windows Server 2016 on xcp-ng 7.4 and 7.5 without any issues, im hoping this isnt something related to the specific version xcp-ng 7.6 since we're planning on migrating from XenServer to this version.