Xen Guru group | XCP-ng and XO forum

TeddyAstie

@dkidd255 @jamesg
I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control).

In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq ?

dinhngtu

@ovicz I've built xapi-26.1.16-1.2.0.pirq.1.xcpng8.3 on xcp-ng-ndinh2. You can update as usual, then install my packages on top.

TeddyAstie

I'm not sure disabling Nagle is a good idea (even though it can improve things here). Fundamentally, we're doing bulk transfer of disk content, which Nagles tries to optimize by coalescing packets, so you're not flooding the network with small TCP packets.

The main problem here is that the progress is gated by NBD replies, which is going to be bad regardless of TCP configuration. TCP_NODELAY will workaround this problem, but with significant tradeoffs (and perhaps will perform worse in some other cases).
What should be done instead is that writes should be streamed (or pipelined) while reading replies in parralel, so that NBD reply delays doesn't bottleneck the whole transfer. But that actually requires a redesign of the whole NBD implementation which is not going to be a easy thing AFAICT.

TeddyAstie

I can't speak for the whole team, but will still give my view on it, at least on what I'm aware of.

@hydromike a dit:

...

VROC is the easier half

Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter.

VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0.

Roughly what that path needs:

CONFIG_VMD in the dom0 kernel

The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30)

mdadm with IMSM in dom0

Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help

Worth reminding that installing custom software in Dom0, or replacing Dom0 packages is discouraged.

The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one.

SupremeRAID is harder

I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back.

P2P DMA is more of a platform capability (e.g ACS stuff) than a hypervisor issue. At least, as long as there is no DMA to magic regions like MSI-X/PBA as that will potentially fail.

In the past, I had success doing P2P DMA between a PCI card and NVMe drives, so it's definetely doable.

...

Questions for @yann and @Team-Hypervisor-Kernel

These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after:

Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask.

Dom0 will still be running in PV mode by default at least. There are no immediate plan to switch to PVH Dom0.

What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere.

What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing.

Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM?

Features like SecureBoot will add complexity with external out-of-tree modules, and while there can be solutions to that, but I don't expect any of that to land anytime soon.

Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in?

If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead.

...

AIA, I don't expect anything practical with the current state of things unless there is a quite significant effort.
However, if these stack(s) can be moved in a dedicated VM with PCI Passthrough, that can make things more workable with the likes of storage domains.

dinhngtu

@ovicz xapi-26.1.16-1.1.0.pirq.2.xcpng8.3 is now available on xcp-ng-ndinh2, which disables HVM PIRQ for all existing VMs by default on reboot. Please try it out.

I've prepared an upstream PR for this feature. If you'd like to be listed as a tester, please respond with your name and email address (or your fake github email if you prefer) and I'll update my commit with your credit.

dinhngtu

@archw Which AV is causing the warning, can you report it as a false positive?

dinhngtu

@clip This often happens with Driver Verifier enabled, what do verifier /querysettings and verifier /query say?

dinhngtu

@ovicz Thanks for testing. Is everything working without the pci=nomsi or GRUB command line tweaks? We'll work to integrate this fix into a future update.

Xen Guru

Posts

VROC is the easier half

SupremeRAID is harder

Questions for @yann and @Team-Hypervisor-Kernel

Member List