Win Server 2012 R2 VM: repeated BSOD 0xD1 in xenvbd.sys (Citrix 8.1.0.130) - where to get VM Tools 9.2.3?

[ovicz]

Why don't you upgrade the VM to Windows Server 2016 ??. It is possible to do an in-place upgrade from 12 to 16. Of course take a backup first.

[hrvojer]

@ovicz said:

ovicz
17 minutes ago

Why don't you upgrade the VM to Windows Server 2016 ??. It is possible to do an in-place upgrade from 12 to 16. Of course take a backup first.

Thanks @ovicz, good point. An in-place upgrade to 2016 is definitely on the table. The main constraints on our side are a 2016 license and downtime: this is a production SQL server without a regular maintenance window, so we’d first test the whole upgrade on a clone to know the exact downtime and catch surprises. One practical question if you’ve done it: did you remove the old Citrix PV tools before the in-place upgrade (and install the new XCP-ng tools afterwards), or upgrade with the old tools still installed?

[dinhngtu]

@hrvojer You should remove old tools before doing an in-place upgrade because the older ones are almost guaranteed to cause upgrade failures or BSODs if they're still installed when the upgrade takes place. Also note that removal of older tools (and installation in some cases) will wipe out your static IP config if you happen to depend on it.

[dinhngtu]

Also, I'd recommend upgrading to Server 2019 or even Server 2025 if that's feasible, since support for Server 2016 is ending soon anyway.

[ovicz]

Upgrading to another windows version will probably require a new license key. Going straight to 2025 is doable...but i think it's a little too much of a strech. Try to go with 2019-2022. Also you can't jump more than 2 versions from 2012R2 directly to 2022, you can go only to 2019 and then another in-place upgrade to 2022. Only 2025 allows upgrading from 2012R2 directly.

[hrvojer]

Thanks both, very useful. One constraint on our side: this VM runs SQL Server 2014, and per Microsoft it’s supported only up to Windows Server 2019 (not 2022/2025). So our realistic target is 2012 R2 → 2019 in-place (after updating SQL to SP3), with the old Citrix tools removed beforehand and the new XCP-ng tools installed afterwards. We’ll rehearse the whole thing on a clone first, and note down the static IP config before removing the tools.

[john.c]

@hrvojer said:

Hi all,

We have a production Windows Server 2012 R2 VM (SQL Server) that has crashed three times with the same bugcheck in xenvbd.sys. Looking for advice and for a safe download of the last 2012 R2-compatible tools.

Environment

• XCP-ng 7.6.0 (yes, we know it's EOL. Right now we need a stopgap for this VM.)
• Storage: LVM over FC (HPE MSA 1040), multipath
• VM: Windows Server 2012 R2 (build 9600.18505), 8 vCPU, 64 GB RAM (static, dynamic min = max)
• PV drivers: Citrix xenvbd.sys 8.1.0.130 (old Citrix tools)

Crashes (identical each time)

• 2026-06-17, 2026-09-01, 2026-10-08
• BUGCHECK 0xD1 DRIVER_IRQL_NOT_LESS_OR_EQUAL (0x18, 0x2, 0x1, xenvbd+0x8795)
• Write to NULL+0x18: lock xadd dword ptr [rcx+18h],eax with rcx = 0
• Stack (from MEMORY.DMP):
  nt!KiRetireDpcList -> nt!KiExecuteAllDpcs
  -> xenvbd+0xf464 -> xenvbd+0x6f55 -> xenvbd+0xfecb -> xenvbd+0x8795
  i.e. crash in a DPC, apparently in the I/O completion path.

Context

• The VM ran with 16 GB RAM from 2023-09 to 2026-02 with zero crashes (no restarts at all in ~29 months).
• On 2026-02-06 RAM was raised to 64 GB (SQL max server memory 14 GB -> 48 GB).
• First crash 4 months later; intervals between crashes so far: 131, 76, 37 days.
• Host logs at the time of the last crash: no FC/SCSI/multipath errors. The only event on this VM's VDIs was one SMGC coalesce with a clean tapdisk pause/unpause about 12 minutes before the crash.

Questions

1. Is this a known xenvbd bug fixed in later versions (e.g. the 9.2.1 fix "A storage error can cause Windows VMs to crash", or the grant table exhaustion fix in 9.3.1)?
2. A reply in the XSA-468 thread pointed to XenServer VM Tools 9.2.3 as the last version supporting 2012/2012 R2. Where can it be downloaded from an official source? downloads.xenserver.com only offers 9.6.0, and the same URL pattern for 9.2.3 returns 404.
3. Could the RAM increase (16 -> 64 GB) plausibly trigger this with old xenvbd (grant tables, larger I/O)? Is there a workaround that doesn't require a driver update (e.g. a host-side setting)?

We know 2012 R2 is EOL; we are looking for a safe stopgap.

Thanks!

@hrvojer Since a guest upgrade to Windows Server 2022 or 2025 is on the table, it makes the most sense to put a hypervisor upgrade to XCP-ng 8.3 LTS on the table at the exact same time.

Dropping XCP-ng 8.2.1 from your options is the right move anyway, as the 8.2 branch went completely EOL in September 2025. Jumping straight from 7.6 to 8.3 LTS (supported through late 2028) aligns perfectly with your Windows lifecycle project for a few major reasons:

1. Modern Kernel Compatibility: XCP-ng 7.6 does not officially support or recognize modern Windows Server 2022/2025 kernels. Attempting to force modern Windows PV tools onto a 7.6 host will lead to immediate instability.

2. Clean Side-by-Side Migration: XCP-ng 8.3 can safely run your legacy 2012 R2 VM. This means you can upgrade the host infrastructure first to achieve baseline stability. Once the host is on 8.3, you can spin up a fresh WS 2022/2025 VM with modern, stable PV drivers and do a clean SQL database migration, bypassing the need to troubleshoot the legacy xenvbd.sys crash entirely.

3. Storage & Infrastructure Breakthroughs: Moving to 8.3 unlocks SMAPIv3 and QCow2. This eliminates old VHD limits, allowing you to give your new modern Windows Server VMs massive disks up to 16 TB per image alongside the latest advancements in Xen Orchestra.

Upgrading the Windows OS while leaving the hypervisor on EOL 7.6 is essentially building a brand-new house on a crumbling foundation.

Also important to consider upgrading your SQL Server 2014 from that version to 2022, or migrating it to an open source alternative like MariaDB version 12.0-13.0 or PostgreSQL version 17 or 18.

[dinhngtu]

@john.c Indeed I skipped over the part that the host is still on 7.6, that's a rather big upgrade blocker I think.

[john.c]

@dinhngtu said:

@john.c Indeed I skipped over the part that the host is still on 7.6, that's a rather big upgrade blocker I think.

@dinhngtu Yes definitely a major blocker, but so is their MS SQL Server instance being version 2014. Which is also EOL, additionally not supported by newer Windows Server following version 2019 and will need to be upgraded or migrated to an open source alternative, either of my suggestions above.

If they do this MS SQL Server upgrade or migration it will then enable an OS upgrade to 2022 or 2025, that will then bypass EOL upcoming for Windows Server 2016 and 2019. It will also get them out of the current EOL trap of Windows Server 2012 and Windows Server 2012 R2.

As well as upgrading or migrating anything else that depends on this that can’t work with newer components, even with any configuration tweaks.

Staying on what they’re currently got isn’t a recommended or best option as eventually a crash WILL corrupt the database or other data of that VM. Which will propagate into backups of that VM, meaning that you have an unenviable choice of corrupted backup data loss or data loss through too old backups required (missing some data) but not corrupted.

[john.c]

@hrvojer My posts above will help you get out of a technical debt trap that you are currently pitfall-trap trapped in!

If you do this, it will drastically help to reduce your operational costs. Maintaining an old legacy stack like this—where every component from the hypervisor to the database engine is completely End of Life (EOL)—is scandalously expensive. It sucks up valuable engineering hours in firefighting, forces you into hyper-expensive custom support agreements, and leaves you exposed to massive downtime risks—which is a nightmare scenario, especially if your other organizational costs are going up at the same time.

I highly recommend framing this to your leadership using these exact terms:

• The Pitfall Trap: Explain that the infrastructure team is currently stuck at the bottom of a pitfall trap of compounding technical debt due to a historical lack of a system-wide overhaul.

• The Rescue Plan: Make it clear that trying to find deprecated, 2018-era VM tools is just a short-term sticking plaster fix that buries the team deeper. What you need is a definitive rescue plan—a clean overhaul that decouples your data layer, migrates you to a modern OS/database (or an open-source alternative), and permanently lifts you out of the trap. As well as other dependencies - systems that communicate with that one, but won’t be able to without upgrade or migration, even with configuration tweaks.