Hi @olivierlambert, proposing a dual-path approach with an OpenVDDK fallback helps manage edge cases, as well idiosyncratic actions that are specific to VDDK or OpenVDDK when reading VMware specifics on their side.

@olivierlambert said:

Hello @andrija.panic !

I'm aware and I followed your work and announcements It's great!

On our side, we rely on our solution since the end of September: https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md#690-2026-09-30 : "[V2V] Migrations no longer need the VDDK library".

We couldn't wait much more and we had to act very fast to remove this dependency.

Hi @olivierlambert, thanks for the insight on vectura! Completely understand the need to move fast and gain full vertical integration over the V2V pipeline—it’s a great move for standard migrations.

My suggestion to consider a dual-path approach with an OpenVDDK fallback is less about day-to-day operations and more about strategic, ecosystem-level insurance.

If we look at how the Xen Project / Xen Foundation handles governance, it provides a perfect architectural blueprint for this problem:

• Shared Governance vs. Vendor Control: For OpenVDDK to be a reliable fallback against Broadcom's moving goalposts, it ultimately needs a single, unified organizational structure (similar to a Linux Foundation sub-project). If Vates, Red Hat, and the CloudStack/Proxmox communities pool resources into a shared, neutrally governed utility, the burden of reverse-engineering VMware's disk-format shifts is distributed across the entire industry.

• Isolating the Core Stack: Just like XCP-ng handles alternative storage plugins or drivers, OpenVDDK wouldn't need to be baked into your core engine. It could live as a completely decoupled, optional legacy transport layer. If vectura hits an unforeseen edge case, a specific CBT lock, or an old ESXi topology (like the legacy versions dropped in 6.9.0), the system has an automated valve to fall back on.

• Focusing R&D Resources: By helping steer a shared industry standard for the low-level data extraction layer, Vates secures a massive legal and engineering buffer against VMware licensing and structural changes, allowing your internal teams to focus 100% of their energy on making Xen Orchestra the best frontend destination.

It prevents Xen Orchestra from ever being caught by surprise, while still keeping your in-house binary as the primary choice.

@pandusen said:

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

@Team-XAPI-Network @bleader Looping in the toolstack side as well. Even if the underlying Xen hypervisor layers align to handle large BAR configurations, XAPI will need the native capabilities to orchestrate and pass these extended memory mapping boundaries down to the VM configuration templates. Without XAPI awareness, we won't be able to easily toggle or manage virtual ReBAR allocations via the CLI or Xen Orchestra.

@JamesG said:

@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.

@JamesG Absolutely spot on. Media encoding/decoding and VDI density are huge components of this. Being able to slice up a GPU efficiently means VMs need to interact with that VRAM cleanly without hitting performance walls.

Whether it's splitting the card via vGPU profiles or passing it through fully for raw compute, lifting these platform memory limitations is what unlocks that "do more with less" capability for media stacks. It all ties back to how robustly the hypervisor fabric can handle modern GPU memory mapping.

If my feedback on Vates Feedback will help your use case please vote for it.

@pandusen Your Windows test results are the perfect real-world example of this exact issue.

When you noted that the Intel driver panel was explicitly complaining about "PCI compatibility and reBar" and that performance was heavily borked, you hit the exact technical bottleneck. Without ReBAR support at the hypervisor layer, the guest OS is forced to interact with that massive 16 GB VRAM space through a rigid 256 MB window, choking it out with massive memory paging overhead.

Even @teddyastie's Linux testing reflects this—noting that host-device transfer bandwidth is "a bit slower than theoretical."

@olivierlambert From a strategic standpoint, as home labs and enterprise environments aggressively pivot toward localized AI, LLM inferencing, and heavy GPU compute, raw PCIe passthrough is shifting from an edge case to a core infrastructure requirement. Implementing a virtualized form of ReBAR support within Xen/XCP-ng would natively lift this 256 MB restriction. It ensures modern GPU silicon can run at 100% efficiency, keeping XCP-ng highly competitive against monolithic hypervisors that handle large BAR mapping.

Especially with VMware VCF 9.0 and up-to-date Hyper-V on Windows Server 2025 advancing their platforms, Vates needs to have virtualised ReBAR support to maintain its edge against this competition.

@teddyastie I’ve added feedback in regards to the above on Vates Feedback portal, see link below.

https://feedback.vates.tech/posts/123/feature-request-support-for-virtual-resizable-bar-rebar-large-bar-passthrough

@pandusen said:

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

A part of the reason for it being slower without ReBAR (or Resizable BAR) is due to that technology lifting, the 256 MB limit (for per chunk) on CPU accessing GPU VRAM.

To improve performance when passing through modern GPUs (and graphics cards). With lots of VRAM the limit on accessing GPU VRAM by CPU will really crease the performance of the VM at the minimum. So support for ReBAR present in Xen and XCP-ng would help performance, when passing through the GPU. Basically a virtual form of ReBAR support for VMs, when passing through GPUs.

@acebmxer said:

@john.c

Updated to latest commit did not help as expected... Also tried firefox an with out reverse proxy...

If you’re using a script to help with compiling Xen Orchestra From Sources, please check whether there’s been any compilation issues when compiling it (especially ones related to the VM and Host Consoles). If there are any please grab a copy of the logs from the compilation, update the script and try compiling again.

@coolsport00 said:

@julienxovates - playing around some more in XO6. I have quite a list I want to submit for feature enhancements (needs, really). One you all already have is being able to open a VM Console in another Tab. I attempted to but it keeps spinning & Console never displays:

I'm using Brave browser. Sometimes this particular browser can cause issues loading pages because of its security. But, I also tried Chrome and same behavior. Thoughts?

@acebmxer @coolsport00 Just like Microsoft Edge, Brave and Google Chrome both share the same upstream Chromium codebase. If there is an underlying bug in Chromium itself, it can potentially affect both browsers simultaneously.

However, because this involves the XO6 VM Console spinning indefinitely, it is usually caused by environment configurations or active development bugs. Here are the best steps to troubleshoot:

• Update XO From Sources: If you are running XO from sources, try updating by building from the head of the master branch. XO6 is moving fast, and this console bug may already have a fix waiting for you in the latest commit.

• Self-Signed Certificates: Open a new tab and navigate directly to your XO instance's IP/domain to explicitly "Accept the risk and continue". Chromium browsers often silently block background Console WebSocket connections if the root SSL certificate isn't trusted.

• WebSockets & Proxies: Ensure your reverse proxy, firewall, or ad-blocker isn't dropping or blocking wss:// (Secure WebSocket) connections, which the console relies on to stream the visual data.

@hrvojer My posts above will help you get out of a technical debt trap that you are currently pitfall-trap trapped in!

If you do this, it will drastically help to reduce your operational costs. Maintaining an old legacy stack like this—where every component from the hypervisor to the database engine is completely End of Life (EOL)—is scandalously expensive. It sucks up valuable engineering hours in firefighting, forces you into hyper-expensive custom support agreements, and leaves you exposed to massive downtime risks—which is a nightmare scenario, especially if your other organizational costs are going up at the same time.

I highly recommend framing this to your leadership using these exact terms:

• The Pitfall Trap: Explain that the infrastructure team is currently stuck at the bottom of a pitfall trap of compounding technical debt due to a historical lack of a system-wide overhaul.

• The Rescue Plan: Make it clear that trying to find deprecated, 2018-era VM tools is just a short-term sticking plaster fix that buries the team deeper. What you need is a definitive rescue plan—a clean overhaul that decouples your data layer, migrates you to a modern OS/database (or an open-source alternative), and permanently lifts you out of the trap. As well as other dependencies - systems that communicate with that one, but won’t be able to without upgrade or migration, even with configuration tweaks.

• The Linux Performance & Driver Advantage: If you choose to migrate your database server to Linux (running PostgreSQL or MariaDB), you gain massive performance, reliability, and security boosts. Unlike Windows, you will completely bypass the fragile legacy Citrix xenvbd.sys driver ecosystem. On Linux, the paravirtualized storage and network drivers are compiled directly into the Linux kernel. The only thing you need to install separately is the modern, official XCP-ng guest agent—which Vates has beautifully rewritten as an ultra-secure, memory-safe Rust-language agent. This gives you a lean, stable database foundation that can handle heavy memory allocations and high I/O workloads without ever breaking a sweat.

@dinhngtu said:

@john.c Indeed I skipped over the part that the host is still on 7.6, that's a rather big upgrade blocker I think.

@dinhngtu Yes definitely a major blocker, but so is their MS SQL Server instance being version 2014. Which is also EOL, additionally not supported by newer Windows Server following version 2019 and will need to be upgraded or migrated to an open source alternative, either of my suggestions above.

If they do this MS SQL Server upgrade or migration it will then enable an OS upgrade to 2022 or 2025, that will then bypass EOL upcoming for Windows Server 2016 and 2019. It will also get them out of the current EOL trap of Windows Server 2012 and Windows Server 2012 R2.

As well as upgrading or migrating anything else that depends on this that can’t work with newer components, even with any configuration tweaks.

Staying on what they’re currently got isn’t a recommended or best option as eventually a crash WILL corrupt the database or other data of that VM. Which will propagate into backups of that VM, meaning that you have an unenviable choice of corrupted backup data loss or data loss through too old backups required (missing some data) but not corrupted.

@hrvojer said:

Hi all,

We have a production Windows Server 2012 R2 VM (SQL Server) that has crashed three times with the same bugcheck in xenvbd.sys. Looking for advice and for a safe download of the last 2012 R2-compatible tools.

Environment

• XCP-ng 7.6.0 (yes, we know it's EOL. Right now we need a stopgap for this VM.)
• Storage: LVM over FC (HPE MSA 1040), multipath
• VM: Windows Server 2012 R2 (build 9600.18505), 8 vCPU, 64 GB RAM (static, dynamic min = max)
• PV drivers: Citrix xenvbd.sys 8.1.0.130 (old Citrix tools)

Crashes (identical each time)

• 2026-06-17, 2026-09-01, 2026-10-08
• BUGCHECK 0xD1 DRIVER_IRQL_NOT_LESS_OR_EQUAL (0x18, 0x2, 0x1, xenvbd+0x8795)
• Write to NULL+0x18: lock xadd dword ptr [rcx+18h],eax with rcx = 0
• Stack (from MEMORY.DMP):
  nt!KiRetireDpcList -> nt!KiExecuteAllDpcs
  -> xenvbd+0xf464 -> xenvbd+0x6f55 -> xenvbd+0xfecb -> xenvbd+0x8795
  i.e. crash in a DPC, apparently in the I/O completion path.

Context

• The VM ran with 16 GB RAM from 2023-09 to 2026-02 with zero crashes (no restarts at all in ~29 months).
• On 2026-02-06 RAM was raised to 64 GB (SQL max server memory 14 GB -> 48 GB).
• First crash 4 months later; intervals between crashes so far: 131, 76, 37 days.
• Host logs at the time of the last crash: no FC/SCSI/multipath errors. The only event on this VM's VDIs was one SMGC coalesce with a clean tapdisk pause/unpause about 12 minutes before the crash.

Questions

1. Is this a known xenvbd bug fixed in later versions (e.g. the 9.2.1 fix "A storage error can cause Windows VMs to crash", or the grant table exhaustion fix in 9.3.1)?
2. A reply in the XSA-468 thread pointed to XenServer VM Tools 9.2.3 as the last version supporting 2012/2012 R2. Where can it be downloaded from an official source? downloads.xenserver.com only offers 9.6.0, and the same URL pattern for 9.2.3 returns 404.
3. Could the RAM increase (16 -> 64 GB) plausibly trigger this with old xenvbd (grant tables, larger I/O)? Is there a workaround that doesn't require a driver update (e.g. a host-side setting)?

We know 2012 R2 is EOL; we are looking for a safe stopgap.

Thanks!

@hrvojer Since a guest upgrade to Windows Server 2022 or 2025 is on the table, it makes the most sense to put a hypervisor upgrade to XCP-ng 8.3 LTS on the table at the exact same time.

Dropping XCP-ng 8.2.1 from your options is the right move anyway, as the 8.2 branch went completely EOL in September 2025. Jumping straight from 7.6 to 8.3 LTS (supported through late 2028) aligns perfectly with your Windows lifecycle project for a few major reasons:

1. Modern Kernel Compatibility: XCP-ng 7.6 does not officially support or recognize modern Windows Server 2022/2025 kernels. Attempting to force modern Windows PV tools onto a 7.6 host will lead to immediate instability.

2. Clean Side-by-Side Migration: XCP-ng 8.3 can safely run your legacy 2012 R2 VM. This means you can upgrade the host infrastructure first to achieve baseline stability. Once the host is on 8.3, you can spin up a fresh WS 2022/2025 VM with modern, stable PV drivers and do a clean SQL database migration, bypassing the need to troubleshoot the legacy xenvbd.sys crash entirely.

3. Storage & Infrastructure Breakthroughs: Moving to 8.3 unlocks SMAPIv3 and QCow2. This eliminates old VHD limits, allowing you to give your new modern Windows Server VMs massive disks up to 16 TB per image alongside the latest advancements in Xen Orchestra.

Upgrading the Windows OS while leaving the hypervisor on EOL 7.6 is essentially building a brand-new house on a crumbling foundation.

Also important to consider upgrading your SQL Server 2014 from that version to 2022, or migrating it to an open source alternative like MariaDB version 12.0-13.0 or PostgreSQL version 17 or 18.

@poddingue This can be handled quite cleanly by including a simple up to date package inventory (via dpkg -l) for the XVA, saved as a JSON file, along with a "built-on" date within the download portal to accompany the standard checksum.

Can this please be developed and implemented on the download portal where we grab the XVA, and/or where the Security Advisories are held?

Providing this metadata upfront would solve a major visibility gap for two critical reasons:

1. Pre-Boot Compliance & SBOMs: Many enterprise security teams require an asset and package review before an image is ever imported or booted in a production cluster. Having a verifiable JSON manifest alongside the download allows admins to satisfy compliance and Software Bill of Materials (SBOM) requirements without forcing a "blind" deployment.

2. First-Boot Vulnerability Window: Because the XVA acts as a snapshot frozen in time until it hooks into the Vates update servers and Debian repos, it inherently contains a window of unpatched packages upon its very first boot. A build date and package list give engineers immediate clarity on what baseline versions (like OpenSSH or the kernel) are active during that initial setup window.

This wouldn't require changing how the appliance itself updates, but it would provide immense public clarity and enterprise readiness for the deployment process.

@Forza said:

@poddingue said:

For the XVA, I'm not aware of release notes for the appliance image itself: it updates itself after import, and the notes that exist are per XO version, in the monthly blog post and the changelog, both linked from https://docs.xen-orchestra.com/getting-started/releases#release-cadence-and-channels .

The blog doesn't usually mention the base OS image itself, including things like openssh stuff, os version, server software, kernel, etc.

It may not do that but since it’s currently based on Debian Linux it’s either version 12 or 13, having been for a long time version 11. Knowing this you may be able find out a lot of what your listing, unless Vates swaps out the vanilla versions for newer or older ones than the base distribution.

@herhin2017 said:

@john.c
Your absolute wright, i do my best and train about 25 young engineers per year on linux (we use debian). I also see more and more startups or small companies building their infrastructure on linux.
Best greetings from austria

It may be worth getting in line and noting the mention of EFI based servers for official Vates support with XCP-ng version 9.0 and above, in government reports. That way when the school’s hardware is refreshed it can be ensured that your provided with EFI capable servers, in time for XCP-ng version 8.3 EOL.

I personally are already ready for XCP-ng version 9.0 due to my servers being Dell PowerEdge R620 for XCP-ng hosts. Along with Debian version 13.6 on the VMs. While using a Dell Precision 3590 to manage those systems.

The “wright” makes your above now sound like a wheel wright, and its profession instead of “right” in for when correct or agreeing with someone or something.

@herhin2017 said:

Hello John C.
In Austria, the Government pays Microsoft Licenses for everything. So every school can use their Software (you know what i mean). To use Linux or something else is not really well received. I am not able to use corporations with 3rd party comp. and i am not allowed to take money from somewere else!

So that may be about to change, depending on how far an on going transition in Austrian government goes. As with your country’s military now on Linux and LibreOffice your academic system is going to need to be training, Austrian children on those systems to prepare them, for if they join the military.

https://linuxsecurity.com/news/government/linux-security-defense

@herhin2017 said:

Yes, your wright! I know about that. As i mentioned, our machines are really working perfect (they can easily be a productive system) I am asking because our school does not have much money (in fact nothing) and i will go on with my lessons in hypervisors with xcp-ng.
THANKS TO EVERYONE who works on this project - i like it!!
P.S. If you need user tests in grater scale, we can do it (10 machines) and we have time.

Welcome to the forums!

To answer your question while expanding on @Poddingue's excellent point: it all comes down to the AlmaLinux base chosen for the dom0 control domain. While AlmaLinux 10 does offer an alternative x86_64-v2 compilation option for older processors, relying on a sub-baseline build for a modern hypervisor can limit your performance and future third-party repository support.

To truly get the most out of XCP-ng 9.0 when it reaches stable release—and to avoid the hard limitations of the modern EFI and Secure Boot requirements—you will want to target at least generation 9 of HPE servers (Gen9) at a minimum.

Since you are dealing with a tight school budget but need to acquire a large number of machines for your network engineering students, here are the most effective ways to source that volume:

• Corporate Hardware Donations: Depending on your school's official legal status (such as Gemeinnützigkeit), you might be able to get these machines donated for free. Many large companies in Austria cycle out their enterprise hardware every few years and actively look to donate working Gen9 or Gen10 servers to schools for tax benefits or community goodwill. It is highly worth reaching out to local IT departments or tech companies!

• Professionally Refurbished Bulk Batches: Sourcing former enterprise hardware in large, uniform batches is your next best path. Gen9 servers offer excellent performance for a very low price right now. Keeping the hardware identical also makes managing your pool templates significantly easier.

• Academic Programs & Vates Discounts: If your rollout scales up and you eventually need official commercial backing, it is well worth reaching out directly to Vates. Let them know you are an educational institution; they are often very supportive of academia and may offer steep volume or academic purchasing discounts for schools.

Good luck setting up the lab for your hypervisor lessons!

P.S. A quick, friendly tip for your English writing: drop the "w" when you want to agree with someone. If you use "write", it sounds like the action of writing words down on paper (schreiben). To agree with someone, the correct phrases are "You're right" or "You are right" (Du hast recht / Sie haben recht).