XCP-NG 9.0 Support for GRAID Tech GPU-accelerated RAID cards.

[hydromike]

I have started a conversation with them about getting support for XCP-NG, for the VROC and the Supreme RAID solutions. I am looking at this as a forward looking effort. In these efforts I have found that Alma 10 is roughly what 9.0 is based upon. What do I need to get pulled together to make this happen. I actually am having a call with there DEV team today. To pull what is on there end that needs rot make it work. I will update this with what happens on the call today. I use there GPU-cards in 4 nodes that an are separate, each is part of GPU cluster. I have wanted to build out the next customers clusters with XCP-NG with the accelerators in them.

Mike George
Bitrefinery

[poddingue]

Thanks for chasing this with them, and for offering to post back after the call.

Your Alma 10 read looks right (the 9.0 demonstrator is Xen 4.19 with a dom0 loosely based on AlmaLinux 10 and a 6.12 kernel), but @yann said in July they're out of prototyping and rebuilding the foundations component by component, so I wouldn't treat that kernel as the one their driver ends up targeting.
On the packaging side, the kernel module policy page is the nearest thing I've found to an answer for what you'd need to pull together: https://docs.xcp-ng.org/project/development-process/kernel-module-policy

For VROC, my understanding is that it sits on VMD, though I could have that backwards. VMD stalled boot on the demonstrator kernel back in January until yann said he'd include a patch for it, so that stretch of the thread is worth a read before your call: https://xcp-ng.org/forum/topic/11698

I don't know enough about driver packaging to say what Vates would actually need from GRAID, so it might be worth pulling in @Team-Hypervisor-Kernel, since drivers are their scope.
If the call gives you a list of what they need from our side, that's probably the most useful thing to bring back here.

[hydromike]

@poddingue Thanks for the pointers — the kernel module policy page and the VMD stretch of the 9.0 demonstrator thread were both exactly what I needed. Reporting back on the call as promised.

Where Graid landed. They want as much detail as we can give them, and it goes to their R&D team in Taiwan. They flagged that they don't support kernel modules outside their qualified matrix, which is the crux of it. So I went digging into what that matrix actually contains, and there's considerably more overlap than the initial conversation suggested.

Their Linux driver 2.0.1, released 21 August, lists AlmaLinux 10.0–10.2 on kernel 6.12 — the same base and kernel line as the 9.0 demonstrator. They also support Proxmox VE across five separate non-distro kernels (6.5, 6.8, 6.14, 6.17, 7.0), so building against a hypervisor's own kernel is routine for them. This isn't a new platform class; it's one more kernel.

You also had VROC and VMD the right way round, and that turns out to matter more than it first looked.

VROC is the easier half

Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter.

VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0.

Roughly what that path needs:

• CONFIG_VMD in the dom0 kernel
• The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30)
• mdadm with IMSM in dom0
• Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help

The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one.

SupremeRAID is harder

I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back.

On packaging I've told them plainly that the .run installer is a non-starter, there's no DKMS in dom0, and every kernel update means a rebuild somebody has to own permanently. @dicode-nl's Ceph work turned out to be the single most useful thing I could send them — a public, current example of third-party kernel modules packaged to policy for dom0, spec file and smoketests included. Between that and the public build container, their team can reach a testable RPM without needing anything from Vates first. Nice timing, and thank you.

Questions for @yann and @Team-Hypervisor-Kernel

These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after:

1. Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask.
2. What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere.
3. What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing.
4. Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM?
5. Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in?

If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead.

Storage side

Worth putting in perspective against what @dicode-nl just delivered for Ceph. RBD needed backported kernel modules, a SMAPIv1 driver, a SMAPIv3 volume and datapath plugin, and a change to XAPI's revert logic, because RBD is a network storage protocol whose snapshot semantics fight XAPI's built-in behaviour. That's genuinely hard work.

GRAID needs none of it. A virtual drive is a local block device. No protocol, no snapshot semantics to reconcile, no SMAPI driver to write or maintain — a stock LVM SR goes straight on top. The whole effort sits in getting the driver to load correctly in dom0.

What caught my eye, though, is TwinStor rather than XOSTOR. TwinStor sits on a single local block device and reports it directly (backing: sda: write cache disabled (durable)). A GRAID VD is exactly that shape.

The 2-node case is where GRAID actually earns its keep. With only two copies, one dead drive degrades a whole replica and leaves you one failure from an outage until resilver completes — and that resilver runs over the replication link, in an environment the TwinStor thread shows is already sensitive to network state. GRAID absorbs the drive failure locally on the GPU, no network traffic, both replicas UpToDate throughout. Write amplification is saner too: RAID6 plus 2-way beats RAID6 plus 3-way.

@olivierlambert — one question before I take this further, since it decides whether the idea is viable at all. TwinStor checks that the backing device's volatile write cache is disabled. Does that check assume a plain SCSI/NVMe target, and would it correctly detect a volatile stage inside a compound device like a GRAID virtual drive? I've asked Graid to confirm whether their VDs honour FLUSH/FUA end to end and whether any parity is staged in GPU or host memory before acknowledgement. If there were a volatile stage TwinStor couldn't see, the durability check would report clean while the guarantee was false — which seems worse than an outright incompatibility given what the fencing design exists to prevent. Happy to be told I'm overthinking it.

The obvious counterweight is dom0 pressure: TwinStor already runs DRBD, the LIO target and multipath in there, and the NVIDIA driver plus graid daemon on top is a real squeeze. I've asked Graid for measured memory footprint numbers.

Hardware

I have two bare nodes I can dedicate to 9.0 testing. Usefully, the first gate needs no drives at all — does the card enumerate in dom0, does the NVIDIA driver load in a PV domain, does the graid module load, does it survive with the IOMMU on and other GPUs passed through to guests. That answers the high-risk questions with no hardware commitment from Graid. Drives come at gate two.

Will post again when Taiwan responds.

Mike George
Bitrefinery