@olivierlambert said: Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward. My earlier ratios were too low: the rig was the bottleneck Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all. Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved. arm build MiB/s range vs stock stalls B/seg A stock control 71.3 71.2-71.4 1.00x 78.3% 1441.6 B TCP_NODELAY only +socket 303.8 303.1-305.0 4.26x 1.5% 1426.6 C NODELAY + pipelining both 526.6 522.8-528.5 7.39x 1.1% 1425.7 D pipelining only +pipeline 528.3 523.0-531.9 7.41x 16.1% 1442.0 [image: 1788093035722-af369410-36dc-4a28-b822-35a00f7098f3-image.jpeg] Per-arm spread is 0.3% to 1.7%. The full stack is worth 7.4x, not 5.51x, and TCP_NODELAY alone is worth 4.26x, not 2.80x. The two changes are substitutes, not complements: TCP_NODELAY alone: 4.26x pipelining alone: 7.41x both together: 7.39x So: Pipelining alone captures the whole win. Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing. The reverse is not true: pipelining on top of NODELAY is still worth 1.73x. NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput. [image: 1788093043264-2ab335d6-fe4b-4019-a8f9-c32d12e971bd-image.jpeg] What the socket option still does after pipelining is remove the stall, 16.1% of samples down to 1.1%, for 1.14% more segments. On this rig that buys no throughput. See the RTT caveat before writing it off. Correction 1: I said Chunked might regress. It does. xe vdi-copy between two host-local SRs, 100 GiB, interleaved A/B/A/B. @TeddyAstie was right. arm transfer data_segs_out B/seg stalls A control 442.1 s 74,746,948 1438.5 0.0% A2 control 439.4 s 74,677,058 1439.9 0.0% B NODELAY 449.5 s 75,205,400 1430.1 0.0% B2 NODELAY 455.5 s 75,260,858 1428.7 0.0% [image: 1788093061682-872deeb0-4514-4f82-96c0-ca3f96564f07-image.jpeg] Both controls beat both patched runs with no overlap, so this is not drift. TCP_NODELAY costs +2.7% wall clock and +0.70% segments here, and buys nothing. Nothing, because the stall it fixes does not occur on this path: Sub-MSS stalls are 0.0% in every arm, including the unpatched control. Chunked writes a 12 byte header then data, with no per-request reply. So the send queue stays backed up (median notsent about 3.3 MB, rwnd_limited 99.7%). Nagle never faces a small-segment decision. Packing is already 0.993x MSS before the patch. Where the extra packets come from, and it is not mainly the headers: The disk is fully allocated, so expand_copy (f.ml:2745) splits at its 2 MiB cap: 51,200 chunks. That is only 9.8% of the 521,126 extra segments. The rest is TLS record boundaries. The channel is unbuffered (channels.ml:129), so each 16 KiB TLS record is its own 16,413 byte write, 11.33x MSS, leaving a sub-MSS remainder. With Nagle those coalesce with the next record. With NODELAY they go out uncoalesced whenever the send queue drains. strace confirms both shapes: 41 byte writes are the headers (12 plus 29 of AES-GCM), 16,413 byte writes are the bulk records. Data is correct either way: the patched copy md5s identical to the source, ce647d9436b48401cd4b489c955ef0f7. Protocol confirmed two ways, for anyone reproducing: An https destination not advertising transfer-encoding: nbd yields [Chunked; NoProtocol] (impl.ml:1084), and the code takes List.hd. At runtime the receiver runs vhd-tool serve --source-format raw --source-protocol chunked --source-fd 8. Correction 2: my reasoning for "NODELAY is a no-op after pipelining" was wrong I said that with 8 requests in flight there is almost always an MSS queued, so the sub-MSS condition rarely arises. Not what happens. Arm D is pipelining with the socket patch removed, and it still stalls on 16.1% of samples against 1.1% with it. Depth 8 does not keep the queue above MSS. The conclusion survives, the reasoning does not: NODELAY after pipelining gives no throughput gain because the stall stops being the limiting factor, not because the stall goes away. Correction 3: the export path needs nothing Export runs stream_raw (impl.ml:335) : export_raw_vdi.ml:56 passes "none" as the destination protocol. No headers, no replies, so no write-write-read pattern. The patch could not cover it anyway: --destination-fd becomes fd://N, then File_descr, then Channels.of_raw_fd (impl.ml:991), never touching the socket helper. It does not need to. http_svr.ml:624 already sets TCP_NODELAY on the listening socket and Linux passes it to accepted sockets. Verified on the 4.19 kernel these hosts run, with a listener without the option as a control returning 0. stunnel sets it independently too (xapi_stunnel_server.ml:66-68). The cost of TCP_NODELAY, every test we ran path Nagle on TCP_NODELAY cost NBD 107 GiB, NVMe 1441.7 (0.996x MSS) 1425.2 (0.984x) +1.16% segments NBD pipelined, RAM 1442.0 (0.996x) 1425.7 (0.984x) +1.14% segments NBD 6-pair repeat, RAM 1441.9 (0.996x) 1424.6 (0.984x) +1.20% segments Chunked 100 GiB 1438.5-1439.9 (0.993x) 1428.7-1430.1 (0.988x) +0.70% segments [image: 1788093077310-865bccc1-8e7e-4948-b329-50ad84a56e25-image.jpeg] About 1.2% more segments, reproducible to three digits across four independent experiments. Small, and it is the real permanent cost of the option. Where the ceiling is now At 7.4x we are at about 525 MiB/s, and it is not the obvious things: Not the network: 10G link, roughly 46% utilised. Not the sender: sparse_dd at mean 57%, peak 64% of one core. On the receiving host, tapdisk is roughly 72% of all busy CPU (python3 9%, xapi 6%, stunnel 6%). That sample spans more than one leg and sums several tapdisk processes, so read it as dominance rather than a precise figure. It is where we would look next. Our reading Both changes are worth having, they are independent, and they are not additive, so the order is a real decision. TCP_NODELAY Strongest argument is precedent, not the benchmark: vhd-tool is the only NBD client in the toolstack that leaves Nagle on. QEMU sets it unconditionally (nbd/client-connection.c:143) and forwards it through TLS. blktap treats failing to set it as fatal (drivers/block-nbd.c:793). xapi and stunnel both set it. 12 lines, 4.26x on code that ships today. Costs, now measured rather than assumed: about 1.2% more segments everywhere, and 2.7% wall clock on vdi-copy between two host-local SRs where it gains nothing. Pipelining 7.41x on its own, and it makes the socket option irrelevant to throughput. Less work than it looks: the multiplexer already exists and is unused (nbd/lib/client.ml:78), so this is not the NBD redesign it first appears to be. The real obstacle is buffer ownership. expand_copy hands out slices of one shared 2 MiB buffer, so a naive pipeline silently corrupts data while reporting success. Our prototype works around it with a local buffer pool and a memcpy per block. The proper fix belongs inside expand_copy in f.ml, a shared library with other consumers. The prototype is not mergeable as it stands: progress counts issued rather than completed work, and a failed write leaves its siblings unawaited. We will follow whichever route the XAPI team prefers and are happy to do the work either way. Our own order would be the socket option first, then pipelining done properly rather than our prototype. That is a sequencing preference, not a claim that the gains compose. If you would rather go straight to pipelining and skip the socket change, our data supports that: it reaches 7.41x on its own. One caveat that cuts in favour of the socket option, and our rig cannot measure it: Every number here is from a 0.2 ms RTT link. The Nagle stall costs a delayed-ACK round trip, so its cost scales with RTT. 0.2 ms is therefore the regime least favourable to fixing it. On a higher-RTT link (cross-rack, cross-site, DR replication) the same 16.1% stall rate that costs nothing here should cost real throughput. So read 4.26x as a floor for TCP_NODELAY, and "pipelining makes it redundant" as a statement about short links specifically. We have not measured a long-RTT link and would like to hear from anyone who has. Open questions Is the 2.7% on vdi-copy acceptable, or should the option be scoped to NBD? Unconditional is simpler and matches every other component. Scoping avoids a measured regression on a path that cannot benefit. Is the Chunked header worth batching regardless? It is a separate unbuffered 12 byte write before every chunk, which under TLS becomes its own 41 byte record. Writing header and payload together removes that independently of any socket option, and helps the Nagle-on case too. Should expand_copy own a buffer pool? Any pipelining implementation needs it. Fixing it in f.ml fixes it for every consumer, but that is a wider blast radius than we wanted to take unilaterally. Is the receiving tapdisk the next real ceiling? At 7.4x the sender and the network both still have headroom and tapdisk dominates destination CPU. We have not dug into why. Does the picture change on a high-RTT or faster link? Both conclusions, that NODELAY is worth 4.26x and that pipelining makes it redundant, are properties of a short fast link that never saturated. Rig Two host XCP-ng 8.3 pool, xapi 26.1.16, 10G, 0.2 ms RTT. Arms differ by exactly one setsockopt where relevant, same tree and toolchain. Every run verified by binary sha256 before it started, transfers checksum verified. RAM SRs are ext4 on a brd ramdisk. tmpfs does not work: no O_DIRECT, so VDIs can be written onto such an SR but never migrated back off. Excellent work. IMHO, disabling Nagle's algorithm is the best and least risky option. Application code paths and protocol behaviour stays exactly the same, except for the socket option. For a very risk free change we gain a lot or performance in a known bottleneck. Pipelining exposes a new code path which needs more debugging and verification before it should be deployed, especially in a LTS situation. Speaking as a paying enterprise customer, stability and predictably is important to us. Therefore I propose that the short-term solution is to disable Nagle's algorithm and ship it in the current LTS release as a normal bug fix. Then, in a longer term, develop the Pipelining option and ensure it get proper testing and verification before release, particularly if backporting it to current LTS.

@koushik you have to use the Helm controller addons for RKE2 to deploy autmatically the Helm chart for the Xen Orchestra CCM with the field spec.bootstrap set to true: Field | Default | Description -------------------------------------------------- `spec.bootstrap` | `false` | Set to True if this chart is needed to bootstrap the cluster (Cloud Controller Manager, etc) EDIT: I wrote a wiki page with what I've tested https://github.com/vatesfr/xenorchestra-cloud-controller-manager/wiki/Rancher-RKE2

Did a lot of work this weekend and added more to the volume and datapath. in native mode, everything is rbd native including CBT! still working on SXM in tapdisk mode, everything is working with optimized rbd calls except SXM in qemu mode everything works by using qemu-dp and also SXM in MSAPIv3 Huge improvements made, but qemu is the slowest mode due to extra nbt hop. I only provided tapdisk and qemu modes to get more information on how CBT works and how SXM works. Right now I'm using that knowledge to provide SXM over native rbt. I have some good ideas but need to figure out how to make them work. I've also made a backport to ceph v15.2 which runs on dom0 to also include the newer cephx authentication protocol. This is because the ceph dashboard cannot provide a diff between snaps (which is needed for CBT) but a pull request is pending to have this working in the future without the need for rbd userspace. I'll update my git and packages later this week.

@olivierlambert said: I have no issue using Claude, but I would prefer your own conclusions/recommendations in the end and less text Sory its early morning. I wanted to show the statement what it said about what was missing on Vates side along what was missing from my script. Also to verify if Claude was correct or not. I have been battle with that alot lately.

@teddyastie This is exactly the answer I was hoping for, including the parts telling me I was wrong. Thank you. Context for why I'm chasing this: with GRAID's cards, the value isn't headline throughput — it's that parity doesn't land on the host CPU. On a hypervisor that matters more than on bare metal, since cycles spent on parity are cycles unavailable to guests. I have nodes with lower base clocks where that hurts, and md's RAID5/6 path doesn't parallelise well enough to compensate with core count. Decoupling storage compute from guest compute is the whole objective. Scope note I should have led with: I don't need to boot from any of this. Dom0 can boot from a plain mirrored pair or a SATA DOM. What I need is performant data storage for guests. That drops the UEFI pre-boot and installer-visibility requirements from the VROC side, and makes storage-domain boot ordering much less severe — nothing critical is waiting on it. Taking the P2P correction. I had that as make-or-break and framed it as unfixable. If it's an ACS/platform question rather than a hypervisor one, and you've had it working between a PCI card and NVMe drives, that removes what worried me most. I'll ask Graid whether their data path ever targets BAR regions containing MSI-X tables or the PBA. Noted on custom software in Dom0, which also lands on the VROC track I'd been treating as the cheap half — I'd listed mdadm with IMSM without checking whether that's a Dom0 package change or already present. The XSM finding I ran the XSM check before any build work and hit something worth raising. This is 8.3: (XEN) XSM Framework v1.0.1 initialized (XEN) Initialising XSM SILO mode # xl getenforce Flask XSM Disabled SILO enforces that DomUs communicate only with Dom0, not with each other — no grants, no event channels between unprivileged domains. A storage driver domain is a DomU, a guest consuming its storage is a DomU, and blkback to blkfront needs grants plus an interdomain event channel with Dom0 at neither end. Unless I'm misreading silo_mode_dom_check, that's exactly what SILO denies. Which might also explain the hardcoded ssidref you queried on the PR. FLASK ships driver domain labels in its default policy; we're not running FLASK. The question is whether 9.0 keeps SILO as the default, since that's what I want to build against. I'm installing 9.0 on a bare node tonight and will post the actual output rather than leave it hypothetical. If it's still SILO, the same wall is there. Either way: can a driver domain be privileged enough to pass is_control_domain() without defeating the isolation it exists to provide, and has SILO gained driver-domain handling upstream since it landed? I'd rather be wrong about this than right. The driver domain changed my plan Mark Syms' motivation comment on #7169 describes my situation almost word for word — taking something generically available as a Linux appliance and letting it serve storage without rebuilding drivers into XenServer/XCP-ng. Graid's driver 2.0.1 already lists AlmaLinux 10.0–10.2 on kernel 6.12 as qualified. In a storage driver domain that's their existing build, unmodified: no out-of-tree module in Dom0, so no module policy question, no proprietary-vs-GPL argument, no SecureBoot complication, and none of the rebuild-per-kernel-update maintenance which I suspect was the real objection. It also gets its own vCPU allocation and can be pinned to the socket owning the PCIe root complex where the card and drives live — something Dom0 can't give me. So rather than backporting to 8.3, I'd rather build against 9.0 and carry the patch there, where xapi should sit much closer to master. Correction to my own earlier post: I said GRAID needs no SMAPI work because a virtual drive is just a local block device. True for an LVM SR in Dom0, not here — Mark is explicit that the driver domain implementation must satisfy the Storage API, and recommends SMAPIv3. With @dicode-nl having just published a working SMAPIv3 volume and datapath plugin that's tractable, but it isn't free and I shouldn't have implied otherwise. Where this leads Your mentioning storage domains sent me back to something I'd half-remembered: https://xcp-ng.org/blog/2021/07/12/dpus-and-the-future-of-virtualization/ Olivier's argument there was that offloading pre-existing software outside Dom0 — storage stack included — also shrinks Dom0's attack surface, ending with storage and networking configured through the DPU from XO while getting bare-metal performance. He signed off promising a follow-up devblog with tests and benchmarks, and I've never found where that went. It strikes me that #7169 is the missing piece. "Get the storage stack out of Dom0" needs a toolstack mechanism for serving storage from something that isn't Dom0. Once that contract exists, whether the provider is a VM on the same CPU or an Arm SoC on a card is an implementation detail. GRAID in a driver domain is the version buildable this year; a DPU-hosted provider is the same contract on different silicon. Which also answers @last-genius' concern on the PR that this would sit unused. It's the prerequisite for something Vates was publicly excited about five years ago. XCP-ng on a DPU is genuinely what I want at the end of all this. Questions Are 9.0 RPM sources and a build-env target public yet, or internal until an alpha? Would Vates consider carrying #7169 in the 9.0 line given it's upstream with two approvals, or is that a post-merge conversation? Has anyone measured blkback driver-domain overhead versus Dom0-native? That number decides whether any of this is worth pursuing. If nobody has, I'd like to produce it — I can test the plumbing with any passed-through NVMe drive, no GRAID hardware needed. On TwinStor: I suspect this doesn't compose the way I imagined, since TwinStor sits on a local block device in Dom0. Happy to hear it's a dead end now rather than later. Two bare nodes ready for 9.0 work, and I'm auditing spares for VMD-capable Xeons to see whether I can test the VROC track independently. Mike George Bitrefinery

@christopher-petzel Ok! Thanks!

@ovicz I've built xapi-26.1.16-1.2.0.pirq.1.xcpng8.3 on xcp-ng-ndinh2. You can update as usual, then install my packages on top.

@poddingue said: I haven't used that tab myself, and since it's built around VMs that do have backups, I don't know if it gives you the negative view you want. If it doesn't, https://feedback.vates.tech is where that gets counted. Maybe you can build some sort of filter but I don't think so. I'll use https://feedback.vates.tech for future suggestions. Tks

@poddingue You're right. The RPU task shown in the list was a previous one. Even that one finished fine. So both RPU tasks didn't show at all today! My bad. The evacuation and remirroring tasks were all shown. Latest XOA 6.7.1

It was too tempting not to test, so I went and tested the other half. On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it. The clone came up at 800x600. A control clone of the same original, with nothing set, came up at 1024x768. So your workaround holds, the resolution really does ride along into VMs built from the template. For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record. One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024. So you may not need to raise videoram for the common ones. Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it. At least, that's my understanding.

Same only remote pools failed. Main XOA pool completed with no errors. { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "compactMjml", "hideSuccessfulItems": true }, "id": "1788224400022", "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d", "jobName": "Salem - Delta Backups", "message": "backup", "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426", "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4", "start": 1788224400022, "status": "failure", "end": 1788224460395, "result": { "message": "fetch failed", "name": "TypeError", "stack": "TypeError: fetch failed\n at node:internal/deps/undici/undici:14976:13\n at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n at processImmediate (node:internal/timers:484:21)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)" } }

Nice, thanks for the feeder entry and the explanation, @CAPS!

@mike.potapov Can you upgrade to the latest blktap-3.55.5-9.3.xcpng8.3 to check is the issue is still there?

@olivierlambert - Just updating. The issue is still present in the version: 6.7.1. I doubt any fix was made. May be not many people are deploying windows 11 with vTPM through templates. Otherwise this issue should have been reported by others.

Just wanting to express my thanks for the script which seemed to do the trick in resolving this issue for us, once we were able to get the HVM migrations completed to enable the move to 8.3. We understand it is not a long term fix for the underlying issue, but appreciate all we have been given. Thanks.

@Danp said: I haven't seen a way to do this from within XO. However, you can gather the details using xe on the command line -- for i in `xe snapshot-list | grep uuid | awk '{print$5}'`; do export j=$(xe snapshot-param-get uuid=$i param-name=children); k=$(xe snapshot-param-get uuid=$i param-name=snapshot-time); echo "VM UUID: $j - Snapshot UUID: $i - Creation Time: $k"; done This doesn't sort the results, so you will need to do that in Excel or your favorite text editor. @Danp adding | sort -t: -k4 to your command will sort it by date. -t: set fields separator to : -k4 sort by 4th field

I am not sure if it is the same issue or not. I doing daily replication from: 4 VMs nvme-1 SR vhd --> nvme-2 SR qcow2 2 VMs nvme-2 SR qcow2 --> nvme-1 SR vhd That is homelab for testing only. Xen Orchestra, commit 6a441 [image: 1788116186019-fd87b54f-fe95-449b-8f7e-17c1da61a4cd-image.jpeg] Standalone "Host up-to-date!" with latest august updates. Replication qcow2 --> vhd works as expected. Replication of vhd --> qcow2 creates daily new replica VM and reports this: [image: 1788116670461-b6ae1b45-0735-44b8-b490-c4f89ba3f953-image.jpeg] Log: 2026-08-30T18_42_49.945Z - backup NG.json.txt