Migrating an offline VM disk between two local SRs is slow

olivierlambert

Can you describe exactly the steps that were done so we can double check/compare and understand the why?

edit: also, are you comparing a live migration vs an offline copy? It's very different, since in live you have to replicate the blocks while the VM on top is running.

pkgw

@olivierlambert This is all offline. Unfortunately I can't describe exactly what was done, since someone else was doing the work and they were trying a bunch of different things all in a row. I suspect that the apparently fast migration is a red herring (maybe a previous attempt left a copy of the disk on the destination SR, and the system noticed that and avoided the actual I/O?) but if there turned out to be a magical fast path, I wouldn't complain!

olivierlambert

You can also try warm migration, which can go a lot faster.

Forza

Using XOA "Disaster Recovery" backup method can be a lot faster than normal offline migration.

One time I did it, it took approx 10 minutes instead of 2 hours...

magicker

I think I am seeing a similar issue. Raid1 NVME copy to raid 10 4x2tb HDD on same host

a 300gb transfer is estimated at 7 hours. (11% done in 50 mins)

the vm is live.

according to the stats almost nothing is happening on this server or the 2 storage

Davidj 0

@olivierlambert
Is the CPU on the sending host or the receiving host the limiting factor for single disk migrations?

olivierlambert

I can't really tell, gut feeling is the sending host, but I have no numbers to confirm.

magicker

@Davidj-0 in my case there CPU activity is minimal. I think something is wrong with the software raid 10 setup. On an identical setup warm migration between to the raid 10 array between hosts is showing horrible iowait similar to the sr to sr transfer on the other host

olivierlambert

Maybe the IO scheduler is not the right one?

tosh

@olivierlambert
I think I found the root cause of the slow Storage Migration performance I reported above.

After tracing sparse_dd, the issue appears to be an interaction between Nagle's algorithm and TCP delayed ACKs on the NBD connection.

With strace, I found that sparse_dd sends the NBD request header and payload using separate write() calls:

write(fd, <NBD header>, 28) = 28
write(fd, <data>, 2097152) = 2097152
read(fd, <NBD reply>, 16) = 16

Normally this is fast, but there are also periodic 512-byte requests:

write(fd, <NBD header>, 28) = 28
write(fd, <data>, 512) = 512

~40 ms delay

read(fd, <NBD reply>, 16) = 16

During these stalls, ss -tinp showed:

ato:40
unacked:1
notsent:512

This seems to produce the following sequence:

The 28-byte NBD header is sent.
It remains unacknowledged.
The following 512-byte payload is queued.
Nagle's algorithm prevents that small payload from being sent while the previous data is unacknowledged.
The peer's delayed ACK timer expires after approximately 40 ms.
The ACK arrives and the 512-byte payload is finally transmitted.

I also checked a packet capture. The destination iSCSI write only occurred after this delay, so the storage itself was not causing the 40 ms stall.

To verify this before modifying the package, I attached GDB to the running sparse_dd process and enabled TCP_NODELAY with setsockopt() on its existing TCP socket.

The effect was immediate: the notsent:512 stalls disappeared and migration throughput increased substantially.

I then patched ocaml/vhd-tool/src/impl.ml.

The current code is:

let socket sockaddr =
  let family =
    match sockaddr with
    | Lwt_unix.ADDR_INET (addr, port) ->
        Unix.domain_of_sockaddr (Lwt_unix.ADDR_INET (addr, port))
    | Lwt_unix.ADDR_UNIX _ ->
        Unix.PF_UNIX
  in
  Lwt_unix.socket family Unix.SOCK_STREAM 0

I changed it to enable TCP_NODELAY for TCP sockets:

--- a/ocaml/vhd-tool/src/impl.ml
+++ b/ocaml/vhd-tool/src/impl.ml
@@
-  Lwt_unix.socket family Unix.SOCK_STREAM 0
+  let sock = Lwt_unix.socket family Unix.SOCK_STREAM 0 in
+  ( match sockaddr with
+  | Lwt_unix.ADDR_INET _ ->
+      Lwt_unix.setsockopt sock Unix.TCP_NODELAY true
+  | Lwt_unix.ADDR_UNIX _ ->
+      ()
+  ) ;
+  sock

I rebuilt vhd-tool for XCP-ng 8.3 and tested Storage Migration again.

Before the patch, I was consistently seeing only around:

30-40 MB/s

After enabling TCP_NODELAY, I am seeing roughly:

150-300 MB/s

depending on storage activity.

For example, during one test:

eth2: ~174 MB/s
eth3: ~174 MB/s
lo:   ~320 MB/s

and there were also physical-interface peaks around 300 MB/s.

After the patch, ss still shows ato:40, which is expected because delayed ACK is still enabled on the peer:

rtt:0.059/0.017 ato:40 ... unacked:1

but the important difference is that the persistent:

notsent:512

is gone, so the delayed ACK timer no longer stalls the NBD payload.

I also checked the upstream xen-api v26.1.16 source, and the socket creation code still does not enable TCP_NODELAY.

So I believe this explains the ~30-40 MB/s limitation I was seeing with sparse_dd NBD Storage Migration.

Would it make sense to enable TCP_NODELAY for the ADDR_INET socket in vhd-tool upstream?

olivierlambert

Worth mentioning @Team-Storage

TeddyAstie

I'm not sure disabling Nagle is a good idea (even though it can improve things here). Fundamentally, we're doing bulk transfer of disk content, which Nagles tries to optimize by coalescing packets, so you're not flooding the network with small TCP packets.

The main problem here is that the progress is gated by NBD replies, which is going to be bad regardless of TCP configuration. TCP_NODELAY will workaround this problem, but with significant tradeoffs (and perhaps will perform worse in some other cases).
What should be done instead is that writes should be streamed (or pipelined) while reading replies in parralel, so that NBD reply delays doesn't bottleneck the whole transfer. But that actually requires a redesign of the whole NBD implementation which is not going to be a easy thing AFAICT.

olivierlambert

@TeddyAstie You're right on the important part, and I went and measured the rest.

First, the boring argument: TCP_NODELAY is already the default everywhere else in this stack.

Where	Call
QEMU, used for the qcow2 path	`nbd/client-connection.c:143`, `qio_channel_set_delay(..., false)`, unconditional on every connection, and deliberately forwarded through TLS in `io/channel-tls.c`
blktap's own NBD client	`drivers/block-nbd.c:793`, where failing to set it is treated as fatal
xapi generally	`Unixext.set_tcp_nodelay`, and stunnel with `r:/a:/l:TCP_NODELAY=1`

vhd-tool is the only NBD client in the toolstack that leaves Nagle on. If it were a bad default for bulk NBD, QEMU wouldn't do it unconditionally. So this patch is less "new tuning" and more "stop being the exception".

On "bulk transfers benefit from Nagle's coalescing": correct, and it costs 1.16%.

Two-host 8.3 pool, 10G, live SXM, two RPMs from the same tree differing only by that one setsockopt (control build, so the A/B isn't a build-environment artifact).

	bytes / data segment	vs MSS (1448)	segments / GiB
Nagle on	1441.7	0.996x	744,763
TCP_NODELAY	1425.2	0.984x	753,391

Nagle really does pack 99.6% of MSS. But NODELAY still packs 98.4%, because vhd-tool's payload writes are 2 MiB and were never sub-MSS in the first place. The only thing being coalesced is the 28-byte NBD header, and it's paid for with a delayed-ACK round trip. No small-packet flood.

What Nagle is actually holding (107 GiB migration, ss -tinp every 200 ms):

	Nagle on	TCP_NODELAY
Samples with a sub-MSS payload stuck in the send queue	62.2%	0.6%
Median bytes held	541 B	49,239 B
Segments in flight while held	4.1	284.9

62% of the run sitting on a 541-byte header with 4 segments in flight, on a 0.2 ms RTT link. The 0.6% left in the patched arm have 285 segments in flight, so that's a full pipe, not a stall.

Result: transfer phase 2016.9s to 679.9s (~3x), end-to-end 2080.8s to 743.8s. Same stall signature on iSCSI and on local NVMe, so it's the protocol pattern rather than the storage.

On the redesign: I agree, and I don't think it's either/or.

The reply-gating is the real ceiling. Even patched we only reach ~210 MiB/s on a 10G link, because it's still one request at a time. Pipelining is the bigger win and it's the correct fix.

But once writes are pipelined there's almost always at least an MSS queued, so the sub-MSS condition rarely arises and TCP_NODELAY becomes close to a no-op. It isn't something anyone would have to unwind afterwards. 12 lines now, redesign later, no conflict.

Where you might still be right: the regression case for NODELAY is many small writes with no application-level batching. vhd-tool's NBD path is header + 2 MiB so there's nothing to merge, but the Chunked path (12-byte header, no per-request reply) is a different shape, and that's what vdi-copy negotiates between two host-local SRs. I haven't tested that one. If anyone expects a regression, that's where I'd look.

olivierlambert

Here is the exact patch used for the benchmarks above, so the XAPI team can pick it up directly.

Target: xapi-project/xen-api, file ocaml/vhd-tool/src/impl.ml

Note the standalone xapi-project/vhd-tool repo is not the right target. It has been dead since 2021-05-21 (vendored into xen-api on 2021-09-20) and has diverged: socket sits at line 778 there versus 795 in what actually ships.

The patch

diff --git a/ocaml/vhd-tool/src/impl.ml b/ocaml/vhd-tool/src/impl.ml
--- a/ocaml/vhd-tool/src/impl.ml
+++ b/ocaml/vhd-tool/src/impl.ml
@@ -800,7 +800,20 @@ let socket sockaddr =
     | Lwt_unix.ADDR_UNIX _ ->
         Unix.PF_UNIX
   in
-  Lwt_unix.socket family Unix.SOCK_STREAM 0
+  let sock = Lwt_unix.socket family Unix.SOCK_STREAM 0 in
+  (* Disable Nagle's algorithm on TCP sockets. The stream protocols used here
+     (NBD and Chunked) write a small header and its payload with separate
+     write(2) calls, and NBD then waits for a per-request reply. Combined with
+     the peer's delayed ACKs this is the classic write-write-read stall: the
+     header sits in the send queue for up to 40ms waiting for an ACK that the
+     peer is itself delaying. Not applicable to Unix domain sockets. *)
+  ( match sockaddr with
+  | Lwt_unix.ADDR_INET _ ->
+      Lwt_unix.setsockopt sock Unix.TCP_NODELAY true
+  | Lwt_unix.ADDR_UNIX _ ->
+      ()
+  ) ;
+  sock
 
 let split ~limit ~sep str =
   Xapi_stdext_std.Xstringext.String.split ~limit sep str

Why this spot

socket is the only Lwt_unix.socket call site in the whole of vhd-tool, and both Lwt_unix.connect call sites use it:

impl.ml:1030, the tcp: endpoint
impl.ml:1057, the http/https destination that sparse_dd uses for storage motion

So one change covers all of vhd-tool's outbound TCP.

Two things a reviewer will probably ask

Does it survive TLS? Yes. The option is set on the fd before Channels.of_ssl_fd wraps it, so https:// destinations are covered. QEMU does the same thing deliberately in io/channel-tls.c, forwarding set_delay down to the underlying socket.

Why the ADDR_UNIX guard? unix: is a real endpoint scheme (impl.ml:764), and setsockopt(TCP_NODELAY) on an AF_UNIX socket fails with EOPNOTSUPP. Verified on the built binary: the AF_UNIX path issues no setsockopt at all and produces no EOPNOTSUPP.

The same helper is also used for the listening socket in serve (impl.ml:1301). That is harmless, since Linux inherits TCP_NODELAY onto accepted sockets, so it is a small bonus rather than a bug.

Build and verification

Built against XCP-ng 8.3 from xcp-ng-rpms/xapi branch 8.3 (1502e68), source xen-api-26.1.16.tar.gz, in ghcr.io/xcp-ng/xcp-ng-build-env:8.3. Applies cleanly on top of the 20 existing XCP-ng patches, including the qcow2 hybridqcow one, which touches a different region of the same file.

vhd-tool ships as a subpackage of the xapi SRPM. It has no package dependencies beyond shared libraries, and xapi-core's Requires: vhd-tool is unversioned, so it can be installed standalone for testing:

rpm -Uvh vhd-tool-<version>.x86_64.rpm     # add --oldpackage to downgrade
yum downgrade vhd-tool                     # to roll back

No daemon restart needed, since sparse_dd is forked per migration.

Confirmed on the resulting binary:

socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 6
setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0
connect(6, {AF_INET, ...})

The control build (same tree, patch removed) shows zero TCP_NODELAY calls, which is what makes the A/B in the previous post attributable to this change alone.

I have deliberately left the DCO Signed-off-by line off so whoever opens the PR can add their own. Original report and diagnosis credit goes to @tosh.

olivierlambert

Also adding @Team-XAPI-Network