@teddyastie This is exactly the answer I was hoping for, including the parts telling me I was wrong. Thank you.
Context for why I'm chasing this: with GRAID's cards, the value isn't headline throughput — it's that parity doesn't land on the host CPU. On a hypervisor that matters more than on bare metal, since cycles spent on parity are cycles unavailable to guests. I have nodes with lower base clocks where that hurts, and md's RAID5/6 path doesn't parallelise well enough to compensate with core count. Decoupling storage compute from guest compute is the whole objective.
Scope note I should have led with: I don't need to boot from any of this. Dom0 can boot from a plain mirrored pair or a SATA DOM. What I need is performant data storage for guests. That drops the UEFI pre-boot and installer-visibility requirements from the VROC side, and makes storage-domain boot ordering much less severe — nothing critical is waiting on it.
Taking the P2P correction. I had that as make-or-break and framed it as unfixable. If it's an ACS/platform question rather than a hypervisor one, and you've had it working between a PCI card and NVMe drives, that removes what worried me most. I'll ask Graid whether their data path ever targets BAR regions containing MSI-X tables or the PBA.
Noted on custom software in Dom0, which also lands on the VROC track I'd been treating as the cheap half — I'd listed mdadm with IMSM without checking whether that's a Dom0 package change or already present.
The XSM finding
I ran the XSM check before any build work and hit something worth raising. This is 8.3:
(XEN) XSM Framework v1.0.1 initialized
(XEN) Initialising XSM SILO mode
# xl getenforce
Flask XSM Disabled
SILO enforces that DomUs communicate only with Dom0, not with each other — no grants, no event channels between unprivileged domains. A storage driver domain is a DomU, a guest consuming its storage is a DomU, and blkback to blkfront needs grants plus an interdomain event channel with Dom0 at neither end. Unless I'm misreading silo_mode_dom_check, that's exactly what SILO denies.
Which might also explain the hardcoded ssidref you queried on the PR. FLASK ships driver domain labels in its default policy; we're not running FLASK.
The question is whether 9.0 keeps SILO as the default, since that's what I want to build against. I'm installing 9.0 on a bare node tonight and will post the actual output rather than leave it hypothetical. If it's still SILO, the same wall is there. Either way: can a driver domain be privileged enough to pass is_control_domain() without defeating the isolation it exists to provide, and has SILO gained driver-domain handling upstream since it landed? I'd rather be wrong about this than right.
The driver domain changed my plan
Mark Syms' motivation comment on #7169 describes my situation almost word for word — taking something generically available as a Linux appliance and letting it serve storage without rebuilding drivers into XenServer/XCP-ng.
Graid's driver 2.0.1 already lists AlmaLinux 10.0–10.2 on kernel 6.12 as qualified. In a storage driver domain that's their existing build, unmodified: no out-of-tree module in Dom0, so no module policy question, no proprietary-vs-GPL argument, no SecureBoot complication, and none of the rebuild-per-kernel-update maintenance which I suspect was the real objection. It also gets its own vCPU allocation and can be pinned to the socket owning the PCIe root complex where the card and drives live — something Dom0 can't give me.
So rather than backporting to 8.3, I'd rather build against 9.0 and carry the patch there, where xapi should sit much closer to master.
Correction to my own earlier post: I said GRAID needs no SMAPI work because a virtual drive is just a local block device. True for an LVM SR in Dom0, not here — Mark is explicit that the driver domain implementation must satisfy the Storage API, and recommends SMAPIv3. With @dicode-nl having just published a working SMAPIv3 volume and datapath plugin that's tractable, but it isn't free and I shouldn't have implied otherwise.
Where this leads
Your mentioning storage domains sent me back to something I'd half-remembered: https://xcp-ng.org/blog/2021/07/12/dpus-and-the-future-of-virtualization/
Olivier's argument there was that offloading pre-existing software outside Dom0 — storage stack included — also shrinks Dom0's attack surface, ending with storage and networking configured through the DPU from XO while getting bare-metal performance. He signed off promising a follow-up devblog with tests and benchmarks, and I've never found where that went.
It strikes me that #7169 is the missing piece. "Get the storage stack out of Dom0" needs a toolstack mechanism for serving storage from something that isn't Dom0. Once that contract exists, whether the provider is a VM on the same CPU or an Arm SoC on a card is an implementation detail. GRAID in a driver domain is the version buildable this year; a DPU-hosted provider is the same contract on different silicon.
Which also answers @last-genius' concern on the PR that this would sit unused. It's the prerequisite for something Vates was publicly excited about five years ago. XCP-ng on a DPU is genuinely what I want at the end of all this.
Questions
Are 9.0 RPM sources and a build-env target public yet, or internal until an alpha?
Would Vates consider carrying #7169 in the 9.0 line given it's upstream with two approvals, or is that a post-merge conversation?
Has anyone measured blkback driver-domain overhead versus Dom0-native? That number decides whether any of this is worth pursuing. If nobody has, I'd like to produce it — I can test the plumbing with any passed-through NVMe drive, no GRAID hardware needed.
On TwinStor: I suspect this doesn't compose the way I imagined, since TwinStor sits on a local block device in Dom0. Happy to hear it's a dead end now rather than later.
Two bare nodes ready for 9.0 work, and I'm auditing spares for VMD-capable Xeons to see whether I can test the VROC track independently.
Mike George
Bitrefinery
