Data point from a 2-host XCP-ng 8.3 pool that supports @tosh's diagnosis, plus a no-patch workaround that gave us ~2.7x until the TCP_NODELAY fix lands.
Setup
XCP-ng 8.3, xapi-core / vhd-tool 26.1.16-1.2.xcpng8.3, dom0 kernel 4.19.0+1
Two hosts, local ext SRs on NVMe (Samsung 970 EVO Plus / WD Red SN700)
Dedicated migration network: 2.5 GbE (Intel i226-V, igc), MTU 9000, no errors
Offline storage migration (halted VM, VM.migrate_send) of 2 VDIs (disk + snapshot), 16 GiB virtual / 7773 MiB allocated each; sparse_dd ... -prezeroed ... -dest-proto nbd over TLS
Baseline (stock)
132 s per VDI, i.e. ~59 MiB/s on the wire, flat line, same in both directions
Link at ~20 % of line rate, source SR read latency 0.3 ms, dom0 total <= 0.5 vCPU, no single dom0 vCPU above 0.08 (RRD, 60 s averages)
Sending host with turbo or capped at 2.5 GHz gave the same 132 s, so not CPU-bound here
Workaround: disable delayed ACKs on the receiver, only for the migration-network route
# on the destination host; adapt prefix/dev/src to the output of: ip route show <migration-net>
ip route change 10.10.10.0/24 dev xenbr1 proto kernel scope link src 10.10.10.1 quickack 1
With the receiver ACKing immediately, Nagle on the sender only waits one RTT instead of the ~40 ms delayed-ACK timer.
Result (same VM, same direction, same data)
stock
quickack 1 on receiver
per VDI
132 s / 132 s
50 s / 46 s
throughput
~59 MiB/s
~155-169 MiB/s
whole migration
4:50
2:00
dom0 on the receiver went up to ~1.1 vCPU total, still nothing saturated.
Caveats
One run per arm, one pool. I did not capture ss -ti during the runs, so the notsent:512 signature is inferred, not observed here.
It does not address the reply gating @TeddyAstie described, it only removes the delayed-ACK half. Consistent with that, we land at ~2.7x, below the ~4.3x reported above for TCP_NODELAY (different rig, so not directly comparable).
Not persistent: lost on reboot and when xapi re-plugs the PIF. We re-apply it from a small systemd unit at boot as a stopgap and will remove it once vhd-tool ships TCP_NODELAY.
Scope is only connections routed via the migration network (storage and live migration). More ACK packets on that link, nothing else changes.
Is there a PR or issue for the TCP_NODELAY patch in xapi-project/xen-api that we can follow? I could not find one.