@olivierlambert said: Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward. My earlier ratios were too low: the rig was the bottleneck Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all. Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved. arm build MiB/s range vs stock stalls B/seg A stock control 71.3 71.2-71.4 1.00x 78.3% 1441.6 B TCP_NODELAY only +socket 303.8 303.1-305.0 4.26x 1.5% 1426.6 C NODELAY + pipelining both 526.6 522.8-528.5 7.39x 1.1% 1425.7 D pipelining only +pipeline 528.3 523.0-531.9 7.41x 16.1% 1442.0 [image: 1788093035722-af369410-36dc-4a28-b822-35a00f7098f3-image.jpeg] Per-arm spread is 0.3% to 1.7%. The full stack is worth 7.4x, not 5.51x, and TCP_NODELAY alone is worth 4.26x, not 2.80x. The two changes are substitutes, not complements: TCP_NODELAY alone: 4.26x pipelining alone: 7.41x both together: 7.39x So: Pipelining alone captures the whole win. Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing. The reverse is not true: pipelining on top of NODELAY is still worth 1.73x. NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput. [image: 1788093043264-2ab335d6-fe4b-4019-a8f9-c32d12e971bd-image.jpeg] What the socket option still does after pipelining is remove the stall, 16.1% of samples down to 1.1%, for 1.14% more segments. On this rig that buys no throughput. See the RTT caveat before writing it off. Correction 1: I said Chunked might regress. It does. xe vdi-copy between two host-local SRs, 100 GiB, interleaved A/B/A/B. @TeddyAstie was right. arm transfer data_segs_out B/seg stalls A control 442.1 s 74,746,948 1438.5 0.0% A2 control 439.4 s 74,677,058 1439.9 0.0% B NODELAY 449.5 s 75,205,400 1430.1 0.0% B2 NODELAY 455.5 s 75,260,858 1428.7 0.0% [image: 1788093061682-872deeb0-4514-4f82-96c0-ca3f96564f07-image.jpeg] Both controls beat both patched runs with no overlap, so this is not drift. TCP_NODELAY costs +2.7% wall clock and +0.70% segments here, and buys nothing. Nothing, because the stall it fixes does not occur on this path: Sub-MSS stalls are 0.0% in every arm, including the unpatched control. Chunked writes a 12 byte header then data, with no per-request reply. So the send queue stays backed up (median notsent about 3.3 MB, rwnd_limited 99.7%). Nagle never faces a small-segment decision. Packing is already 0.993x MSS before the patch. Where the extra packets come from, and it is not mainly the headers: The disk is fully allocated, so expand_copy (f.ml:2745) splits at its 2 MiB cap: 51,200 chunks. That is only 9.8% of the 521,126 extra segments. The rest is TLS record boundaries. The channel is unbuffered (channels.ml:129), so each 16 KiB TLS record is its own 16,413 byte write, 11.33x MSS, leaving a sub-MSS remainder. With Nagle those coalesce with the next record. With NODELAY they go out uncoalesced whenever the send queue drains. strace confirms both shapes: 41 byte writes are the headers (12 plus 29 of AES-GCM), 16,413 byte writes are the bulk records. Data is correct either way: the patched copy md5s identical to the source, ce647d9436b48401cd4b489c955ef0f7. Protocol confirmed two ways, for anyone reproducing: An https destination not advertising transfer-encoding: nbd yields [Chunked; NoProtocol] (impl.ml:1084), and the code takes List.hd. At runtime the receiver runs vhd-tool serve --source-format raw --source-protocol chunked --source-fd 8. Correction 2: my reasoning for "NODELAY is a no-op after pipelining" was wrong I said that with 8 requests in flight there is almost always an MSS queued, so the sub-MSS condition rarely arises. Not what happens. Arm D is pipelining with the socket patch removed, and it still stalls on 16.1% of samples against 1.1% with it. Depth 8 does not keep the queue above MSS. The conclusion survives, the reasoning does not: NODELAY after pipelining gives no throughput gain because the stall stops being the limiting factor, not because the stall goes away. Correction 3: the export path needs nothing Export runs stream_raw (impl.ml:335) : export_raw_vdi.ml:56 passes "none" as the destination protocol. No headers, no replies, so no write-write-read pattern. The patch could not cover it anyway: --destination-fd becomes fd://N, then File_descr, then Channels.of_raw_fd (impl.ml:991), never touching the socket helper. It does not need to. http_svr.ml:624 already sets TCP_NODELAY on the listening socket and Linux passes it to accepted sockets. Verified on the 4.19 kernel these hosts run, with a listener without the option as a control returning 0. stunnel sets it independently too (xapi_stunnel_server.ml:66-68). The cost of TCP_NODELAY, every test we ran path Nagle on TCP_NODELAY cost NBD 107 GiB, NVMe 1441.7 (0.996x MSS) 1425.2 (0.984x) +1.16% segments NBD pipelined, RAM 1442.0 (0.996x) 1425.7 (0.984x) +1.14% segments NBD 6-pair repeat, RAM 1441.9 (0.996x) 1424.6 (0.984x) +1.20% segments Chunked 100 GiB 1438.5-1439.9 (0.993x) 1428.7-1430.1 (0.988x) +0.70% segments [image: 1788093077310-865bccc1-8e7e-4948-b329-50ad84a56e25-image.jpeg] About 1.2% more segments, reproducible to three digits across four independent experiments. Small, and it is the real permanent cost of the option. Where the ceiling is now At 7.4x we are at about 525 MiB/s, and it is not the obvious things: Not the network: 10G link, roughly 46% utilised. Not the sender: sparse_dd at mean 57%, peak 64% of one core. On the receiving host, tapdisk is roughly 72% of all busy CPU (python3 9%, xapi 6%, stunnel 6%). That sample spans more than one leg and sums several tapdisk processes, so read it as dominance rather than a precise figure. It is where we would look next. Our reading Both changes are worth having, they are independent, and they are not additive, so the order is a real decision. TCP_NODELAY Strongest argument is precedent, not the benchmark: vhd-tool is the only NBD client in the toolstack that leaves Nagle on. QEMU sets it unconditionally (nbd/client-connection.c:143) and forwards it through TLS. blktap treats failing to set it as fatal (drivers/block-nbd.c:793). xapi and stunnel both set it. 12 lines, 4.26x on code that ships today. Costs, now measured rather than assumed: about 1.2% more segments everywhere, and 2.7% wall clock on vdi-copy between two host-local SRs where it gains nothing. Pipelining 7.41x on its own, and it makes the socket option irrelevant to throughput. Less work than it looks: the multiplexer already exists and is unused (nbd/lib/client.ml:78), so this is not the NBD redesign it first appears to be. The real obstacle is buffer ownership. expand_copy hands out slices of one shared 2 MiB buffer, so a naive pipeline silently corrupts data while reporting success. Our prototype works around it with a local buffer pool and a memcpy per block. The proper fix belongs inside expand_copy in f.ml, a shared library with other consumers. The prototype is not mergeable as it stands: progress counts issued rather than completed work, and a failed write leaves its siblings unawaited. We will follow whichever route the XAPI team prefers and are happy to do the work either way. Our own order would be the socket option first, then pipelining done properly rather than our prototype. That is a sequencing preference, not a claim that the gains compose. If you would rather go straight to pipelining and skip the socket change, our data supports that: it reaches 7.41x on its own. One caveat that cuts in favour of the socket option, and our rig cannot measure it: Every number here is from a 0.2 ms RTT link. The Nagle stall costs a delayed-ACK round trip, so its cost scales with RTT. 0.2 ms is therefore the regime least favourable to fixing it. On a higher-RTT link (cross-rack, cross-site, DR replication) the same 16.1% stall rate that costs nothing here should cost real throughput. So read 4.26x as a floor for TCP_NODELAY, and "pipelining makes it redundant" as a statement about short links specifically. We have not measured a long-RTT link and would like to hear from anyone who has. Open questions Is the 2.7% on vdi-copy acceptable, or should the option be scoped to NBD? Unconditional is simpler and matches every other component. Scoping avoids a measured regression on a path that cannot benefit. Is the Chunked header worth batching regardless? It is a separate unbuffered 12 byte write before every chunk, which under TLS becomes its own 41 byte record. Writing header and payload together removes that independently of any socket option, and helps the Nagle-on case too. Should expand_copy own a buffer pool? Any pipelining implementation needs it. Fixing it in f.ml fixes it for every consumer, but that is a wider blast radius than we wanted to take unilaterally. Is the receiving tapdisk the next real ceiling? At 7.4x the sender and the network both still have headroom and tapdisk dominates destination CPU. We have not dug into why. Does the picture change on a high-RTT or faster link? Both conclusions, that NODELAY is worth 4.26x and that pipelining makes it redundant, are properties of a short fast link that never saturated. Rig Two host XCP-ng 8.3 pool, xapi 26.1.16, 10G, 0.2 ms RTT. Arms differ by exactly one setsockopt where relevant, same tree and toolchain. Every run verified by binary sha256 before it started, transfers checksum verified. RAM SRs are ext4 on a brd ramdisk. tmpfs does not work: no O_DIRECT, so VDIs can be written onto such an SR but never migrated back off. Excellent work. IMHO, disabling Nagle's algorithm is the best and least risky option. Application code paths and protocol behaviour stays exactly the same, except for the socket option. For a very risk free change we gain a lot or performance in a known bottleneck. Pipelining exposes a new code path which needs more debugging and verification before it should be deployed, especially in a LTS situation. Speaking as a paying enterprise customer, stability and predictably is important to us. Therefore I propose that the short-term solution is to disable Nagle's algorithm and ship it in the current LTS release as a normal bug fix. Then, in a longer term, develop the Pipelining option and ensure it get proper testing and verification before release, particularly if backporting it to current LTS.

@Danp said: I haven't seen a way to do this from within XO. However, you can gather the details using xe on the command line -- for i in `xe snapshot-list | grep uuid | awk '{print$5}'`; do export j=$(xe snapshot-param-get uuid=$i param-name=children); k=$(xe snapshot-param-get uuid=$i param-name=snapshot-time); echo "VM UUID: $j - Snapshot UUID: $i - Creation Time: $k"; done This doesn't sort the results, so you will need to do that in Excel or your favorite text editor. @Danp adding | sort -t: -k4 to your command will sort it by date. -t: set fields separator to : -k4 sort by 4th field

@ovicz I've built xapi-26.1.16-1.2.0.pirq.1.xcpng8.3 on xcp-ng-ndinh2. You can update as usual, then install my packages on top.

Nice, thanks for the feeder entry and the explanation, @CAPS!

@poddingue You're right. The RPU task shown in the list was a previous one. Even that one finished fine. So both RPU tasks didn't show at all today! My bad. The evacuation and remirroring tasks were all shown. Latest XOA 6.7.1

@acomav We will actually propose to change the date and time format in XO6 settings, so you would be able to choose between : YYYY-MM-DD MM/DD/YYYY DD/MM/YYYY and 12h or 24h time format. Hopefully in one of the next 3 months ! I hope that will answer your need, otherwise let me know !

@christopher-petzel Ok! Thanks!

@poddingue said: I haven't used that tab myself, and since it's built around VMs that do have backups, I don't know if it gives you the negative view you want. If it doesn't, https://feedback.vates.tech is where that gets counted. Maybe you can build some sort of filter but I don't think so. I'll use https://feedback.vates.tech for future suggestions. Tks

It was too tempting not to test, so I went and tested the other half. On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it. The clone came up at 800x600. A control clone of the same original, with nothing set, came up at 1024x768. So your workaround holds, the resolution really does ride along into VMs built from the template. For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record. One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024. So you may not need to raise videoram for the common ones. Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it. At least, that's my understanding.

@mike.potapov Can you upgrade to the latest blktap-3.55.5-9.3.xcpng8.3 to check is the issue is still there?

@dinhngtu Something called "Elpha Secure" ...none of our other antivirus shows it being bad but I wanted to ask around before I unflagged it.

Thanks a lot for this feedback, @pedro!

@mpiton Thanks for looking into this, I guess it wasn't apparent that I needed to click the Save Configuration button for that plugin. I did that and confirmed that the secret now survives an xo-server restart. Thanks again!

@AtaxyaNetwork Merci pour tes recherches ! Oui "cd_label" serait cool comme ajout au plugin ce qui permet sur les distro type Fedora/Redhat de ne pas avoir de boot_command à gérer

Thanks, keep us posted!

I measured this on a two-NIC 8.3 host. As far as I understand, there's no way to do it from XO: the call that edits a network doesn't take a bond mode, every bondMode in the codebase sits on a create path, and the REST API on a running XOA offers create_bonded_network plus get and delete on a network, with nothing that edits one. The CLI route looks cheap though. xe bond-set-mode on a live bond cost zero dropped packets in both directions I tried, pinging every 200ms across the change, and the command returned in about 1.4 seconds. Two caveats, and the second is about your case specifically: creating and destroying the bond did interrupt the host (roughly 8 and 16 seconds), and I only went active-backup to balance-slb and back, never touching lacp, because my switch port isn't configured as a LAG and I'd have dropped the box. So I can't tell you your lacp to active-backup move is free, only that mode changes in general didn't cost me anything. Worth putting on https://feedback.vates.tech either way so the votes have somewhere to gather.

I went digging and three threads turned up people who already have this hardware. @yeopil21 in https://xcp-ng.org/forum/topic/11799 had an LPe37102-D on a Dell R770 that never showed up at all, @jinju0504 in https://xcp-ng.org/forum/topic/11946 ended up on the -alt package on an R670, and @lukas in https://xcp-ng.org/forum/topic/8847 was running an LPe31002-M6-D with NVMe-oF back on 8.3 beta2. That last one is the older generation, so I'd guess it's the side where you most want to hear nothing regressed. Tagging them in case the cards are still on their benches.

I'm late to this, but I've been building lately a JetBrains plugin against the REST API and ACL v2 turned out to decide its whole design, so here's some feedback. Everything below is just one appliance, one pool (my small homelab), on a plan 4 trial, with an admin control call taken in the same breath as every scoped one. What I experienced is that selectors narrow reads (tags: and id: both, 1 VM against an admin control of 11), they're re-evaluated per request rather than fixed when the privilege is created, deny composes the way your Carol example says (allow-all plus deny on a tag gave 9, which is 11 minus the 2 tagged), and they scope the power verbs too, not just reads. The event stream is, to me, scoped as well, which was the thing I most wanted to check, because a scoped read next to an unscoped feed would have been a nasty trap. It's not. With two streams open at once, a change to an out-of-scope VM produced an update on the admin stream and nothing at all on the scoped one, so it's genuinely filtered rather than merely quiet. And the bit I'd underline: the same single change is update to the admin and add or remove to the scoped user. The verb is computed per subscriber, not per object. Across that run the admin only ever saw update, and I originally wrote that an admin never sees add or remove at all. That was wrong and I've since measured it: an admin does get add and remove when an object is genuinely created or deleted. So the rule is that a scope change reads as update and an existence change as add or remove, which supports your design better than what I first wrote did. "From the user's perspective, not XOA's" turns out to be literal rather than a figure of speech, and that's a nicer piece of design than the sentence let me imagine. Now the three things that I didn't see in the post above. vm-snapshot is a separate privilege resource and vm doesn't imply it. Maybe that's obvious, but it was not obvious to me. With all six VM privileges granted, GET /vm-snapshots came back empty while admin saw 6, including a snapshot the scoped user had just taken. Anything with a restore or revert screen gets an empty list and no error. Inherited snapshot tags look like a snapshot-time copy rather than a link. Tag a VM and it's in scope immediately, but its existing snapshots keep tags: [] and stay invisible. So someone onboarded into a tag scope after their snapshots exist sees the VM and not its history. Subscribing delivers no initial dump. Both streams sat on init and keepalives until something changed, so it's a delta feed and a client has to fetch the collection over REST and maintain it from events. Worth a line, since the natural assumption (at least to me, don't make that the rule for everyone) is the other one. While I'm here: two smaller ones. Privilege action names aren't REST action names: shutdown:clean grants clean_shutdown, revert-snapshot grants revert_snapshot, and a mistyped action quietly gives you a privilege that grants nothing. And the event: init frame's field is id, not connectionId (why did I think it was connectionId, no idea, I thought it was "natural"), which cost me a while of thinking the stream was dead when I was posting to /events//subscriptions. Yes, I know, I should have read the documentation instead of experimenting in the dark, sending made-up field names in the wild. One last thing: selector is optional, so a privilege created without one reads back as {id, resource, action, effect, roleId} with no hint the field exists. I granted allow read on vm, saw all 11 VMs, read the object back, and (falsely) concluded the REST API had no object dimension at all. It's all in the previous post and it's in the swagger, and of course, in the official documentation. Once again, I'm an innocent victim because I didn't RTFM. I just never saw a privilege that had one. If a privilege echoed selector: null, or if the first example anyone met were a scoped one, I don't think I would have spent much time on that. Once again, my bad, didn't RTFM. This was a small, targeted test, at best. I didn't have the intent to test what was brought up in this very thread, I just happened to tinkle with the REST API and ACL V2 for my PoC, so lots of things got untested. Please, don't take it from me as settled: only tags: and id: selector forms, nothing on a second pool or a real multi-user deployment, and I have not checked what happens to a live subscription when the privilege itself changes rather than the VM's tags. If you read me until there, you're brave, or have too much time on your hands.