@acebmxer we are far more explicit on the real usage of NBD, paing the way to mark a failed NBD connexion as an error in a few month also working on a diagnostic tool to be more efficient on the checks thank you all for your time

@olivierlambert said: Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward. My earlier ratios were too low: the rig was the bottleneck Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all. Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved. arm build MiB/s range vs stock stalls B/seg A stock control 71.3 71.2-71.4 1.00x 78.3% 1441.6 B TCP_NODELAY only +socket 303.8 303.1-305.0 4.26x 1.5% 1426.6 C NODELAY + pipelining both 526.6 522.8-528.5 7.39x 1.1% 1425.7 D pipelining only +pipeline 528.3 523.0-531.9 7.41x 16.1% 1442.0 [image: 1788093035722-af369410-36dc-4a28-b822-35a00f7098f3-image.jpeg] Per-arm spread is 0.3% to 1.7%. The full stack is worth 7.4x, not 5.51x, and TCP_NODELAY alone is worth 4.26x, not 2.80x. The two changes are substitutes, not complements: TCP_NODELAY alone: 4.26x pipelining alone: 7.41x both together: 7.39x So: Pipelining alone captures the whole win. Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing. The reverse is not true: pipelining on top of NODELAY is still worth 1.73x. NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput. [image: 1788093043264-2ab335d6-fe4b-4019-a8f9-c32d12e971bd-image.jpeg] What the socket option still does after pipelining is remove the stall, 16.1% of samples down to 1.1%, for 1.14% more segments. On this rig that buys no throughput. See the RTT caveat before writing it off. Correction 1: I said Chunked might regress. It does. xe vdi-copy between two host-local SRs, 100 GiB, interleaved A/B/A/B. @TeddyAstie was right. arm transfer data_segs_out B/seg stalls A control 442.1 s 74,746,948 1438.5 0.0% A2 control 439.4 s 74,677,058 1439.9 0.0% B NODELAY 449.5 s 75,205,400 1430.1 0.0% B2 NODELAY 455.5 s 75,260,858 1428.7 0.0% [image: 1788093061682-872deeb0-4514-4f82-96c0-ca3f96564f07-image.jpeg] Both controls beat both patched runs with no overlap, so this is not drift. TCP_NODELAY costs +2.7% wall clock and +0.70% segments here, and buys nothing. Nothing, because the stall it fixes does not occur on this path: Sub-MSS stalls are 0.0% in every arm, including the unpatched control. Chunked writes a 12 byte header then data, with no per-request reply. So the send queue stays backed up (median notsent about 3.3 MB, rwnd_limited 99.7%). Nagle never faces a small-segment decision. Packing is already 0.993x MSS before the patch. Where the extra packets come from, and it is not mainly the headers: The disk is fully allocated, so expand_copy (f.ml:2745) splits at its 2 MiB cap: 51,200 chunks. That is only 9.8% of the 521,126 extra segments. The rest is TLS record boundaries. The channel is unbuffered (channels.ml:129), so each 16 KiB TLS record is its own 16,413 byte write, 11.33x MSS, leaving a sub-MSS remainder. With Nagle those coalesce with the next record. With NODELAY they go out uncoalesced whenever the send queue drains. strace confirms both shapes: 41 byte writes are the headers (12 plus 29 of AES-GCM), 16,413 byte writes are the bulk records. Data is correct either way: the patched copy md5s identical to the source, ce647d9436b48401cd4b489c955ef0f7. Protocol confirmed two ways, for anyone reproducing: An https destination not advertising transfer-encoding: nbd yields [Chunked; NoProtocol] (impl.ml:1084), and the code takes List.hd. At runtime the receiver runs vhd-tool serve --source-format raw --source-protocol chunked --source-fd 8. Correction 2: my reasoning for "NODELAY is a no-op after pipelining" was wrong I said that with 8 requests in flight there is almost always an MSS queued, so the sub-MSS condition rarely arises. Not what happens. Arm D is pipelining with the socket patch removed, and it still stalls on 16.1% of samples against 1.1% with it. Depth 8 does not keep the queue above MSS. The conclusion survives, the reasoning does not: NODELAY after pipelining gives no throughput gain because the stall stops being the limiting factor, not because the stall goes away. Correction 3: the export path needs nothing Export runs stream_raw (impl.ml:335) : export_raw_vdi.ml:56 passes "none" as the destination protocol. No headers, no replies, so no write-write-read pattern. The patch could not cover it anyway: --destination-fd becomes fd://N, then File_descr, then Channels.of_raw_fd (impl.ml:991), never touching the socket helper. It does not need to. http_svr.ml:624 already sets TCP_NODELAY on the listening socket and Linux passes it to accepted sockets. Verified on the 4.19 kernel these hosts run, with a listener without the option as a control returning 0. stunnel sets it independently too (xapi_stunnel_server.ml:66-68). The cost of TCP_NODELAY, every test we ran path Nagle on TCP_NODELAY cost NBD 107 GiB, NVMe 1441.7 (0.996x MSS) 1425.2 (0.984x) +1.16% segments NBD pipelined, RAM 1442.0 (0.996x) 1425.7 (0.984x) +1.14% segments NBD 6-pair repeat, RAM 1441.9 (0.996x) 1424.6 (0.984x) +1.20% segments Chunked 100 GiB 1438.5-1439.9 (0.993x) 1428.7-1430.1 (0.988x) +0.70% segments [image: 1788093077310-865bccc1-8e7e-4948-b329-50ad84a56e25-image.jpeg] About 1.2% more segments, reproducible to three digits across four independent experiments. Small, and it is the real permanent cost of the option. Where the ceiling is now At 7.4x we are at about 525 MiB/s, and it is not the obvious things: Not the network: 10G link, roughly 46% utilised. Not the sender: sparse_dd at mean 57%, peak 64% of one core. On the receiving host, tapdisk is roughly 72% of all busy CPU (python3 9%, xapi 6%, stunnel 6%). That sample spans more than one leg and sums several tapdisk processes, so read it as dominance rather than a precise figure. It is where we would look next. Our reading Both changes are worth having, they are independent, and they are not additive, so the order is a real decision. TCP_NODELAY Strongest argument is precedent, not the benchmark: vhd-tool is the only NBD client in the toolstack that leaves Nagle on. QEMU sets it unconditionally (nbd/client-connection.c:143) and forwards it through TLS. blktap treats failing to set it as fatal (drivers/block-nbd.c:793). xapi and stunnel both set it. 12 lines, 4.26x on code that ships today. Costs, now measured rather than assumed: about 1.2% more segments everywhere, and 2.7% wall clock on vdi-copy between two host-local SRs where it gains nothing. Pipelining 7.41x on its own, and it makes the socket option irrelevant to throughput. Less work than it looks: the multiplexer already exists and is unused (nbd/lib/client.ml:78), so this is not the NBD redesign it first appears to be. The real obstacle is buffer ownership. expand_copy hands out slices of one shared 2 MiB buffer, so a naive pipeline silently corrupts data while reporting success. Our prototype works around it with a local buffer pool and a memcpy per block. The proper fix belongs inside expand_copy in f.ml, a shared library with other consumers. The prototype is not mergeable as it stands: progress counts issued rather than completed work, and a failed write leaves its siblings unawaited. We will follow whichever route the XAPI team prefers and are happy to do the work either way. Our own order would be the socket option first, then pipelining done properly rather than our prototype. That is a sequencing preference, not a claim that the gains compose. If you would rather go straight to pipelining and skip the socket change, our data supports that: it reaches 7.41x on its own. One caveat that cuts in favour of the socket option, and our rig cannot measure it: Every number here is from a 0.2 ms RTT link. The Nagle stall costs a delayed-ACK round trip, so its cost scales with RTT. 0.2 ms is therefore the regime least favourable to fixing it. On a higher-RTT link (cross-rack, cross-site, DR replication) the same 16.1% stall rate that costs nothing here should cost real throughput. So read 4.26x as a floor for TCP_NODELAY, and "pipelining makes it redundant" as a statement about short links specifically. We have not measured a long-RTT link and would like to hear from anyone who has. Open questions Is the 2.7% on vdi-copy acceptable, or should the option be scoped to NBD? Unconditional is simpler and matches every other component. Scoping avoids a measured regression on a path that cannot benefit. Is the Chunked header worth batching regardless? It is a separate unbuffered 12 byte write before every chunk, which under TLS becomes its own 41 byte record. Writing header and payload together removes that independently of any socket option, and helps the Nagle-on case too. Should expand_copy own a buffer pool? Any pipelining implementation needs it. Fixing it in f.ml fixes it for every consumer, but that is a wider blast radius than we wanted to take unilaterally. Is the receiving tapdisk the next real ceiling? At 7.4x the sender and the network both still have headroom and tapdisk dominates destination CPU. We have not dug into why. Does the picture change on a high-RTT or faster link? Both conclusions, that NODELAY is worth 4.26x and that pipelining makes it redundant, are properties of a short fast link that never saturated. Rig Two host XCP-ng 8.3 pool, xapi 26.1.16, 10G, 0.2 ms RTT. Arms differ by exactly one setsockopt where relevant, same tree and toolchain. Every run verified by binary sha256 before it started, transfers checksum verified. RAM SRs are ext4 on a brd ramdisk. tmpfs does not work: no O_DIRECT, so VDIs can be written onto such an SR but never migrated back off. Excellent work. IMHO, disabling Nagle's algorithm is the best and least risky option. Application code paths and protocol behaviour stays exactly the same, except for the socket option. For a very risk free change we gain a lot or performance in a known bottleneck. Pipelining exposes a new code path which needs more debugging and verification before it should be deployed, especially in a LTS situation. Speaking as a paying enterprise customer, stability and predictably is important to us. Therefore I propose that the short-term solution is to disable Nagle's algorithm and ship it in the current LTS release as a normal bug fix. Then, in a longer term, develop the Pipelining option and ensure it get proper testing and verification before release, particularly if backporting it to current LTS.

@florent No explicit assigned migration or backup network set. Backups generate errors somewhat randomly. As one VM won't get the "error" one time. But on the next backup run it will. And the third time it might not. This even happens on VMs that are turned off, and therefore shouldn't have any changed bytes between backups. Host is reachable from XO Network has NBD enabled All VMs use VHD IPv4 Backup type is Delta

Hi, Thanks for your answers For the second problem, after deletion, it is impossible to have the connection to the xo-server despite the refresh of the XOCE page. In a new firefox tab = same problem I closed and then opened Firefox and xo-server connection = OK Best regards

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

@Forza Comparing against the weaker XOCE Strong: net.ipv4.tcp_mem = 90147 120196 180294 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 Weak: net.ipv4.tcp_mem = 41766 55688 83532 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304

@poddingue sm-3.2.12-23.5.xcpng8.3.x86_64 across all three hosts. Just tried running through the chain of repair and restart, and I'm getting this on the latest as well: [12:49 host1 ~]# xe vm-start uuid=c0faf508-9f14-e091-43b1-d6c5ef7e07d4 Error code: SR_BACKEND_FAILURE_1200 Error parameters: , [Errno 30] Read-only file system: '/dev/drbd/by-res/xcp-volume-693829e3-0b60-4b0c-bed2-611995ec83f1/0',

New version: https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases/tag/v20260903 This one includes native Ceph rbd SXM over SMAPIv3! GitHub updated with the latest commits and changes. As always, use with caution. I did run a lot of test scenario's but please do test yourself and let me know your findings!

@dkidd255 @jamesg I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control). In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

install_xen_orchestra v0.5.0 — build your own cloud-init VM templates As I am sure most of you know that XOA Hub is not accessible from Xen Orchestra from sources. So v0.5.0 of my install script adds a VM Template Library — a new --build-templates flag (and menu entry) that builds the equivalent on your own pool, from each distribution's own published cloud image. ./install-xen-orchestra.sh --build-templates It runs from your workstation over SSH against the pool master, like --deploy does. Nothing is installed locally and your XO install isn't touched. What you get A sealed template that shows up next other vm templates you have, with: cloud-init — supply an SSH key or a full cloud-config at VM creation. XCP-ng guest tools, installed from guest-tools.iso on the pool. Without these XO never reports the VM's IP. On Debian 13 there's no xe-guest-utilities package, and a failed package install doesn't stop cloud-init, so an apt-first approach silently gives you a template that looks fine and is useless. The ISO is what the XCP-ng docs prescribe anyway. A scrubbed identity — machine-id, SSH host keys, cloud-init state and logs are all wiped before sealing, so clones don't collide on DHCP leases or share an SSH fingerprint. growroot, so the disk size you ask for at creation is actually filled. One template for both BIOS and UEFI — the Debian generic images carry an ESP and a BIOS boot partition, so you just flip "Boot firmware" in XO's advanced settings. The build itself isn't prompted for storage or network — it uses your pool's default SR (or the emptiest one, if you haven't set a default) and the management network. Both are just build-time choices; you pick whatever you want when you actually create a VM from the template. Images are pulled from the distribution's own mirror (nothing redistributed), downloaded onto the pool master so the ~3 GB never crosses your workstation's link, and verified against the origin's published SHA512SUMS at build time —so new upstream releases get picked up without me editing anything. Allow roughly five to ten minutes per template; it has to boot the VM once to install the guest agent and run the scrub. Distros Debian 13 (Trixie) is the only one for now. That's a starting point, not the limit — the catalogue is a table with one row per distro, and adding another is a row in that table, not a change to the build. So: comment here if there's a specific distro you'd want to see sooner rather than later and I'll prioritise accordingly. Repo: https://github.com/acebmxer/install_xen_orchestra Docs for this feature: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

If you are running XOA, not XO from sources, then we could take a look remotely using the support tunnel. Otherwise, make sure that you have patched and rebooted each pool member.

This was sent to me yesterday!! See our Engineering Team lead's response: We evaluated XCP-ng 8.3 some time ago, but its dom0 kernel was too old for our driver to work properly. Running SupremeRAID inside a DomU should be feasible and is similar to the approach we previously proposed for VMware. However, with XCP-ng 8.3, we did not find a practical native path to export the SupremeRAID VD back to dom0 and use it as an XCP-ng SR. Using iSCSI for this purpose would add significant protocol and networking overhead, which is not ideal for high-performance NVMe storage. I revisited XCP-ng 9.0 and the current Xen/XAPI development. A more promising approach is to run SupremeRAID in a dedicated AlmaLinux VM with the GPU and NVMe drives passed through, then use Xen's native xen-blkback interface to export the SupremeRAID block device back to dom0. Dom0 would see the exported VD as a normal Xen block device, which could then potentially be used to create a standard XCP-ng LVM SR. This approach is much more attractive than iSCSI or NVMe/TCP because the data path uses Xen's blkif shared-memory interface rather than a network protocol. xen-blkback itself is an established Xen mechanism, and Xen supports using a separate domain as a block backend. However, XCP-ng does not currently provide complete first-class lifecycle management for this configuration, so we still need to validate the exact behavior on XCP-ng 9.0, particularly persistent attachment, storage VM startup ordering, and recovery after a host or storage VM reboot. If they can help confirm that the GPU and NVMe drives can be passed through to the storage VM, SupremeRAID can run normally there, and the resulting VD can be exported through xen-blkback to dom0 and used as an XCP-ng SR, I think this could be a very solid architecture for SupremeRAID on XCP-ng. As for write durability, SupremeRAID always operates in write-through mode. An I/O is acknowledged only after all associated data, including parity, has been committed to the drives. Therefore, acknowledged writes do not depend on data or parity remaining only in volatile GPU or host memory Can you do the initial testing with SupremeRAID PRO within your environment? I will begin the initial testing and builds with 4 nodes: 1 & 2 are HP DL380 Gen10 - 2x Xeon Gold 6151 36 Cores 384GB RAM 4 3.84Tb PCI4.0 NVMe per node. Twinstore Testing - Run the Build environment on these 3 is a HP Dl360 Gen10 2x Xeon Gold 6151 36 Cores 512Gb RAM 4x1.92TB NVMe Drives SuperServer SYS-122H-TN- X14 2x Xeon 6740 96 cores, 512GB RAM, GRAID Card - 4x Pci5.0 7.68TB drives SupremeRAID. All the nodes have 2x100Gbps ports, 2x25Gbps ports. Arista Backed network.

@koushik you have to use the Helm controller addons for RKE2 to deploy autmatically the Helm chart for the Xen Orchestra CCM with the field spec.bootstrap set to true: Field | Default | Description -------------------------------------------------- `spec.bootstrap` | `false` | Set to True if this chart is needed to bootstrap the cluster (Cloud Controller Manager, etc) EDIT: I wrote a wiki page with what I've tested https://github.com/vatesfr/xenorchestra-cloud-controller-manager/wiki/Rancher-RKE2

@ovicz I've built xapi-26.1.16-1.2.0.pirq.1.xcpng8.3 on xcp-ng-ndinh2. You can update as usual, then install my packages on top.

Hello everyone ! I rebuild a new OME appliance with the latest version (4.7) What i did on the rebuild: Via live CD add xen-blkfront and xen-netfront driver, and rebuild the initrd change root password activate SSH. Via TUI Change admin password Set interface to DHCP Via SSH sed -i -e 's/sda/xvda/g' /opt/dell/omc/utilities/tui/bin/ome_disk_config.sh sed -i -e 's/sda/xvda/g' /opt/dell/mcsi/appliance/scripts/appliance_resources.sh bash /mnt/Linux/install.sh # To install XCP-ng tools As usual, please send a DM if you want the link to the VHD Also, if you want to build it yourself, the driver are here https://cecilemorange.fr/drivers/xen-netfront.ko https://cecilemorange.fr/drivers/xen-blkfront.ko Feedback is welcome ! (Disclamer: This build is something I create on my free time and to help people who have difficulty running Dell appliances on XCP-ng. I do not guarantee a fully functional appliance, but I do my best to make it work. If you have a support contract with Dell and require official support for OME, please contact Dell and request compatibility with XCP-ng.)

@andriy.sultanov I finally got around to fixing my capture card and I gotta say, it's a real pain to be using both methods and doesn't feel "correct" especially when there are warnings about possibly losing dom0 configurations after updates and the knowledge base saying that the old hide PCI from dom0 method is now considered obsolete. Perhaps the best solution would just be to provide users with a CLI command that would add that particular device to the xl pci-assignable-list if it doesn't fall into classes 01XX, 02XX or 03XX. Because I had added my GPU to my VM using the webUI, but then I also wanted to add the capture card through the console, but it appears that in the end you basically need to undo what's been done in the webUI and do it all through the console, otherwise you'd add the capture card manually with command line and then that would remove the GPUs... So I think some sort of manual override is necessary, even if it's a CLI option and then once that's done we could go into the webUI to enable it once the device has been marked as safe to passthrough.