XCP-NG 9.0 Support for GRAID Tech GPU-accelerated RAID cards.

hydromike

I have started a conversation with them about getting support for XCP-NG, for the VROC and the Supreme RAID solutions. I am looking at this as a forward looking effort. In these efforts I have found that Alma 10 is roughly what 9.0 is based upon. What do I need to get pulled together to make this happen. I actually am having a call with there DEV team today. To pull what is on there end that needs rot make it work. I will update this with what happens on the call today. I use there GPU-cards in 4 nodes that an are separate, each is part of GPU cluster. I have wanted to build out the next customers clusters with XCP-NG with the accelerators in them.

Mike George
Bitrefinery

poddingue

Thanks for chasing this with them, and for offering to post back after the call.

Your Alma 10 read looks right (the 9.0 demonstrator is Xen 4.19 with a dom0 loosely based on AlmaLinux 10 and a 6.12 kernel), but @yann said in July they're out of prototyping and rebuilding the foundations component by component, so I wouldn't treat that kernel as the one their driver ends up targeting.
On the packaging side, the kernel module policy page is the nearest thing I've found to an answer for what you'd need to pull together: https://docs.xcp-ng.org/project/development-process/kernel-module-policy

For VROC, my understanding is that it sits on VMD, though I could have that backwards. VMD stalled boot on the demonstrator kernel back in January until yann said he'd include a patch for it, so that stretch of the thread is worth a read before your call: https://xcp-ng.org/forum/topic/11698

I don't know enough about driver packaging to say what Vates would actually need from GRAID, so it might be worth pulling in @Team-Hypervisor-Kernel, since drivers are their scope.
If the call gives you a list of what they need from our side, that's probably the most useful thing to bring back here.

hydromike

@poddingue Thanks for the pointers — the kernel module policy page and the VMD stretch of the 9.0 demonstrator thread were both exactly what I needed. Reporting back on the call as promised.

Where Graid landed. They want as much detail as we can give them, and it goes to their R&D team in Taiwan. They flagged that they don't support kernel modules outside their qualified matrix, which is the crux of it. So I went digging into what that matrix actually contains, and there's considerably more overlap than the initial conversation suggested.

Their Linux driver 2.0.1, released 21 August, lists AlmaLinux 10.0–10.2 on kernel 6.12 — the same base and kernel line as the 9.0 demonstrator. They also support Proxmox VE across five separate non-distro kernels (6.5, 6.8, 6.14, 6.17, 7.0), so building against a hypervisor's own kernel is routine for them. This isn't a new platform class; it's one more kernel.

You also had VROC and VMD the right way round, and that turns out to matter more than it first looked.

VROC is the easier half

Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter.

VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0.

Roughly what that path needs:

CONFIG_VMD in the dom0 kernel
The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30)
mdadm with IMSM in dom0
Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help

The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one.

SupremeRAID is harder

I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back.

On packaging I've told them plainly that the .run installer is a non-starter, there's no DKMS in dom0, and every kernel update means a rebuild somebody has to own permanently. @dicode-nl's Ceph work turned out to be the single most useful thing I could send them — a public, current example of third-party kernel modules packaged to policy for dom0, spec file and smoketests included. Between that and the public build container, their team can reach a testable RPM without needing anything from Vates first. Nice timing, and thank you.

Questions for @yann and @Team-Hypervisor-Kernel

These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after:

Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask.
What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere.
What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing.
Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM?
Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in?

If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead.

Storage side

Worth putting in perspective against what @dicode-nl just delivered for Ceph. RBD needed backported kernel modules, a SMAPIv1 driver, a SMAPIv3 volume and datapath plugin, and a change to XAPI's revert logic, because RBD is a network storage protocol whose snapshot semantics fight XAPI's built-in behaviour. That's genuinely hard work.

GRAID needs none of it. A virtual drive is a local block device. No protocol, no snapshot semantics to reconcile, no SMAPI driver to write or maintain — a stock LVM SR goes straight on top. The whole effort sits in getting the driver to load correctly in dom0.

What caught my eye, though, is TwinStor rather than XOSTOR. TwinStor sits on a single local block device and reports it directly (backing: sda: write cache disabled (durable)). A GRAID VD is exactly that shape.

The 2-node case is where GRAID actually earns its keep. With only two copies, one dead drive degrades a whole replica and leaves you one failure from an outage until resilver completes — and that resilver runs over the replication link, in an environment the TwinStor thread shows is already sensitive to network state. GRAID absorbs the drive failure locally on the GPU, no network traffic, both replicas UpToDate throughout. Write amplification is saner too: RAID6 plus 2-way beats RAID6 plus 3-way.

@olivierlambert — one question before I take this further, since it decides whether the idea is viable at all. TwinStor checks that the backing device's volatile write cache is disabled. Does that check assume a plain SCSI/NVMe target, and would it correctly detect a volatile stage inside a compound device like a GRAID virtual drive? I've asked Graid to confirm whether their VDs honour FLUSH/FUA end to end and whether any parity is staged in GPU or host memory before acknowledgement. If there were a volatile stage TwinStor couldn't see, the durability check would report clean while the guarantee was false — which seems worse than an outright incompatibility given what the fencing design exists to prevent. Happy to be told I'm overthinking it.

The obvious counterweight is dom0 pressure: TwinStor already runs DRBD, the LIO target and multipath in there, and the NVIDIA driver plus graid daemon on top is a real squeeze. I've asked Graid for measured memory footprint numbers.

Hardware

I have two bare nodes I can dedicate to 9.0 testing. Usefully, the first gate needs no drives at all — does the card enumerate in dom0, does the NVIDIA driver load in a PV domain, does the graid module load, does it survive with the IOMMU on and other GPUs passed through to guests. That answers the high-risk questions with no hardware commitment from Graid. Drives come at gate two.

Will post again when Taiwan responds.

Mike George
Bitrefinery

TeddyAstie

I can't speak for the whole team, but will still give my view on it, at least on what I'm aware of.

@hydromike a dit:

...

VROC is the easier half

Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter.

VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0.

Roughly what that path needs:

CONFIG_VMD in the dom0 kernel

The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30)

mdadm with IMSM in dom0

Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help

Worth reminding that installing custom software in Dom0, or replacing Dom0 packages is discouraged.

The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one.

SupremeRAID is harder

I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back.

P2P DMA is more of a platform capability (e.g ACS stuff) than a hypervisor issue. At least, as long as there is no DMA to magic regions like MSI-X/PBA as that will potentially fail.

In the past, I had success doing P2P DMA between a PCI card and NVMe drives, so it's definetely doable.

...

Questions for @yann and @Team-Hypervisor-Kernel

These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after:

Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask.

Dom0 will still be running in PV mode by default at least. There are no immediate plan to switch to PVH Dom0.

What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere.

What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing.

Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM?

Features like SecureBoot will add complexity with external out-of-tree modules, and while there can be solutions to that, but I don't expect any of that to land anytime soon.

Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in?

If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead.

...

AIA, I don't expect anything practical with the current state of things unless there is a quite significant effort.
However, if these stack(s) can be moved in a dedicated VM with PCI Passthrough, that can make things more workable with the likes of storage domains.

hydromike

@teddyastie This is exactly the answer I was hoping for, including the parts telling me I was wrong. Thank you.

Context for why I'm chasing this: with GRAID's cards, the value isn't headline throughput — it's that parity doesn't land on the host CPU. On a hypervisor that matters more than on bare metal, since cycles spent on parity are cycles unavailable to guests. I have nodes with lower base clocks where that hurts, and md's RAID5/6 path doesn't parallelise well enough to compensate with core count. Decoupling storage compute from guest compute is the whole objective.

Scope note I should have led with: I don't need to boot from any of this. Dom0 can boot from a plain mirrored pair or a SATA DOM. What I need is performant data storage for guests. That drops the UEFI pre-boot and installer-visibility requirements from the VROC side, and makes storage-domain boot ordering much less severe — nothing critical is waiting on it.

Taking the P2P correction. I had that as make-or-break and framed it as unfixable. If it's an ACS/platform question rather than a hypervisor one, and you've had it working between a PCI card and NVMe drives, that removes what worried me most. I'll ask Graid whether their data path ever targets BAR regions containing MSI-X tables or the PBA.

Noted on custom software in Dom0, which also lands on the VROC track I'd been treating as the cheap half — I'd listed mdadm with IMSM without checking whether that's a Dom0 package change or already present.

The XSM finding

I ran the XSM check before any build work and hit something worth raising. This is 8.3:

(XEN) XSM Framework v1.0.1 initialized
(XEN) Initialising XSM SILO mode
# xl getenforce
Flask XSM Disabled

SILO enforces that DomUs communicate only with Dom0, not with each other — no grants, no event channels between unprivileged domains. A storage driver domain is a DomU, a guest consuming its storage is a DomU, and blkback to blkfront needs grants plus an interdomain event channel with Dom0 at neither end. Unless I'm misreading silo_mode_dom_check, that's exactly what SILO denies.

Which might also explain the hardcoded ssidref you queried on the PR. FLASK ships driver domain labels in its default policy; we're not running FLASK.

The question is whether 9.0 keeps SILO as the default, since that's what I want to build against. I'm installing 9.0 on a bare node tonight and will post the actual output rather than leave it hypothetical. If it's still SILO, the same wall is there. Either way: can a driver domain be privileged enough to pass is_control_domain() without defeating the isolation it exists to provide, and has SILO gained driver-domain handling upstream since it landed? I'd rather be wrong about this than right.

The driver domain changed my plan

Mark Syms' motivation comment on #7169 describes my situation almost word for word — taking something generically available as a Linux appliance and letting it serve storage without rebuilding drivers into XenServer/XCP-ng.

Graid's driver 2.0.1 already lists AlmaLinux 10.0–10.2 on kernel 6.12 as qualified. In a storage driver domain that's their existing build, unmodified: no out-of-tree module in Dom0, so no module policy question, no proprietary-vs-GPL argument, no SecureBoot complication, and none of the rebuild-per-kernel-update maintenance which I suspect was the real objection. It also gets its own vCPU allocation and can be pinned to the socket owning the PCIe root complex where the card and drives live — something Dom0 can't give me.

So rather than backporting to 8.3, I'd rather build against 9.0 and carry the patch there, where xapi should sit much closer to master.

Correction to my own earlier post: I said GRAID needs no SMAPI work because a virtual drive is just a local block device. True for an LVM SR in Dom0, not here — Mark is explicit that the driver domain implementation must satisfy the Storage API, and recommends SMAPIv3. With @dicode-nl having just published a working SMAPIv3 volume and datapath plugin that's tractable, but it isn't free and I shouldn't have implied otherwise.

Where this leads

Your mentioning storage domains sent me back to something I'd half-remembered: https://xcp-ng.org/blog/2021/07/12/dpus-and-the-future-of-virtualization/

Olivier's argument there was that offloading pre-existing software outside Dom0 — storage stack included — also shrinks Dom0's attack surface, ending with storage and networking configured through the DPU from XO while getting bare-metal performance. He signed off promising a follow-up devblog with tests and benchmarks, and I've never found where that went.

It strikes me that #7169 is the missing piece. "Get the storage stack out of Dom0" needs a toolstack mechanism for serving storage from something that isn't Dom0. Once that contract exists, whether the provider is a VM on the same CPU or an Arm SoC on a card is an implementation detail. GRAID in a driver domain is the version buildable this year; a DPU-hosted provider is the same contract on different silicon.

Which also answers @last-genius' concern on the PR that this would sit unused. It's the prerequisite for something Vates was publicly excited about five years ago. XCP-ng on a DPU is genuinely what I want at the end of all this.

Questions

Are 9.0 RPM sources and a build-env target public yet, or internal until an alpha?
Would Vates consider carrying #7169 in the 9.0 line given it's upstream with two approvals, or is that a post-merge conversation?
Has anyone measured blkback driver-domain overhead versus Dom0-native? That number decides whether any of this is worth pursuing. If nobody has, I'd like to produce it — I can test the plumbing with any passed-through NVMe drive, no GRAID hardware needed.

On TwinStor: I suspect this doesn't compose the way I imagined, since TwinStor sits on a local block device in Dom0. Happy to hear it's a dead end now rather than later.

Two bare nodes ready for 9.0 work, and I'm auditing spares for VMD-capable Xeons to see whether I can test the VROC track independently.

Mike George
Bitrefinery