Migrating an offline VM disk between two local SRs is slow

Forza

@olivierlambert said in Migrating an offline VM disk between two local SRs is slow:

80/100MiB/s for one storage migration is already pretty decent. You might go faster by migrating more disks at once.

I'm not sure to understand what difference are you referring too? It's always has been in that ballpark, per disk.

This is not over a Network, only between local ext4 SRs on the same server. I tried the same migration using XCP-ng center and it is at the moment double as fast:

Can't really see any difference though. It is the same sparse_dd and nbd connection Perhaps it's a fragmentation issue. Though, doing a copy of the same VHD file gives close to 500MB/s.

pkgw

What generally limits the migration speed? My installation is seeing a phenomenon that might be similar — when migrating between SRs we're only getting about 20 MB/s, while everything is wired together with 10 Gbps links.

olivierlambert

20 MiB/s is rather slow, are you migrating one disk per one disk?

pkgw

@olivierlambert No, this is an array of SSDs on a PowerVault, and we have evidence that the raw thoughput that we can get out of the system is much, much higher. I'm not 100% sure, but it seems that some software aspect of the migration framework is really bottlenecking things, although I've poked around and don't see anything that appears to be CPU-bound either.

olivierlambert

I meant one virtual disk per one virtual disk. If you migrate more VMs/disks at once, you should see it scaling.

pkgw

@olivierlambert Ah, yes, that is true. We are migrating one extremely large VM, so there isn't much to parallize for us, unfortunately. But it is true that someone tried something that caused the system to migrate two disks at once, and the total throughput did double.

(Specifics: the machine is 4 TiB spread across 2 big disks and 1 small OS disk, and the way that we're attempting to migrate right now it all goes serially, so the task is taking multiple days and hitting various timeouts. We think we can solve the timeouts, and we don't expect to need to do this kind of migration at all frequently, but I'd still like to understand why the single-disk throughput is so much lower than what we believe the hardware is capable of.)

olivierlambert

Mostly CPU bound (single disk migration isn't multithreaded). Higher your CPU frequency, faster the migration.

pkgw

@olivierlambert Thanks, that's good to know. I appreciate your taking the time to discuss. I don't suppose there are any settings we can fiddle with that would speed up the single-disk scenario? Or some workaround approach that might get closer to the hardware's native speed? (In one experiment, someone did something that caused the system to transfer the OS disk with the log message "Cloning VDI" rather than "Creating a blank remote VDI", and the effective throughput was higher by a factor of 20 ...)

olivierlambert

Can you describe exactly the steps that were done so we can double check/compare and understand the why?

edit: also, are you comparing a live migration vs an offline copy? It's very different, since in live you have to replicate the blocks while the VM on top is running.

pkgw

@olivierlambert This is all offline. Unfortunately I can't describe exactly what was done, since someone else was doing the work and they were trying a bunch of different things all in a row. I suspect that the apparently fast migration is a red herring (maybe a previous attempt left a copy of the disk on the destination SR, and the system noticed that and avoided the actual I/O?) but if there turned out to be a magical fast path, I wouldn't complain!

olivierlambert

You can also try warm migration, which can go a lot faster.

Forza

Using XOA "Disaster Recovery" backup method can be a lot faster than normal offline migration.

One time I did it, it took approx 10 minutes instead of 2 hours...

magicker

I think I am seeing a similar issue. Raid1 NVME copy to raid 10 4x2tb HDD on same host

a 300gb transfer is estimated at 7 hours. (11% done in 50 mins)

the vm is live.

according to the stats almost nothing is happening on this server or the 2 storage

Davidj 0

@olivierlambert
Is the CPU on the sending host or the receiving host the limiting factor for single disk migrations?

olivierlambert

I can't really tell, gut feeling is the sending host, but I have no numbers to confirm.

magicker

@Davidj-0 in my case there CPU activity is minimal. I think something is wrong with the software raid 10 setup. On an identical setup warm migration between to the raid 10 array between hosts is showing horrible iowait similar to the sr to sr transfer on the other host

olivierlambert

Maybe the IO scheduler is not the right one?

tosh

@olivierlambert
I think I found the root cause of the slow Storage Migration performance I reported above.

After tracing sparse_dd, the issue appears to be an interaction between Nagle's algorithm and TCP delayed ACKs on the NBD connection.

With strace, I found that sparse_dd sends the NBD request header and payload using separate write() calls:

write(fd, <NBD header>, 28) = 28
write(fd, <data>, 2097152) = 2097152
read(fd, <NBD reply>, 16) = 16

Normally this is fast, but there are also periodic 512-byte requests:

write(fd, <NBD header>, 28) = 28
write(fd, <data>, 512) = 512

~40 ms delay

read(fd, <NBD reply>, 16) = 16

During these stalls, ss -tinp showed:

ato:40
unacked:1
notsent:512

This seems to produce the following sequence:

The 28-byte NBD header is sent.
It remains unacknowledged.
The following 512-byte payload is queued.
Nagle's algorithm prevents that small payload from being sent while the previous data is unacknowledged.
The peer's delayed ACK timer expires after approximately 40 ms.
The ACK arrives and the 512-byte payload is finally transmitted.

I also checked a packet capture. The destination iSCSI write only occurred after this delay, so the storage itself was not causing the 40 ms stall.

To verify this before modifying the package, I attached GDB to the running sparse_dd process and enabled TCP_NODELAY with setsockopt() on its existing TCP socket.

The effect was immediate: the notsent:512 stalls disappeared and migration throughput increased substantially.

I then patched ocaml/vhd-tool/src/impl.ml.

The current code is:

let socket sockaddr =
  let family =
    match sockaddr with
    | Lwt_unix.ADDR_INET (addr, port) ->
        Unix.domain_of_sockaddr (Lwt_unix.ADDR_INET (addr, port))
    | Lwt_unix.ADDR_UNIX _ ->
        Unix.PF_UNIX
  in
  Lwt_unix.socket family Unix.SOCK_STREAM 0

I changed it to enable TCP_NODELAY for TCP sockets:

--- a/ocaml/vhd-tool/src/impl.ml
+++ b/ocaml/vhd-tool/src/impl.ml
@@
-  Lwt_unix.socket family Unix.SOCK_STREAM 0
+  let sock = Lwt_unix.socket family Unix.SOCK_STREAM 0 in
+  ( match sockaddr with
+  | Lwt_unix.ADDR_INET _ ->
+      Lwt_unix.setsockopt sock Unix.TCP_NODELAY true
+  | Lwt_unix.ADDR_UNIX _ ->
+      ()
+  ) ;
+  sock

I rebuilt vhd-tool for XCP-ng 8.3 and tested Storage Migration again.

Before the patch, I was consistently seeing only around:

30-40 MB/s

After enabling TCP_NODELAY, I am seeing roughly:

150-300 MB/s

depending on storage activity.

For example, during one test:

eth2: ~174 MB/s
eth3: ~174 MB/s
lo:   ~320 MB/s

and there were also physical-interface peaks around 300 MB/s.

After the patch, ss still shows ato:40, which is expected because delayed ACK is still enabled on the peer:

rtt:0.059/0.017 ato:40 ... unacked:1

but the important difference is that the persistent:

notsent:512

is gone, so the delayed ACK timer no longer stalls the NBD payload.

I also checked the upstream xen-api v26.1.16 source, and the socket creation code still does not enable TCP_NODELAY.

So I believe this explains the ~30-40 MB/s limitation I was seeing with sparse_dd NBD Storage Migration.

Would it make sense to enable TCP_NODELAY for the ADDR_INET socket in vhd-tool upstream?

olivierlambert

Worth mentioning @Team-Storage

TeddyAstie

I'm not sure disabling Nagle is a good idea (even though it can improve things here). Fundamentally, we're doing bulk transfer of disk content, which Nagles tries to optimize by coalescing packets, so you're not flooding the network with small TCP packets.

The main problem here is that the progress is gated by NBD replies, which is going to be bad regardless of TCP configuration. TCP_NODELAY will workaround this problem, but with significant tradeoffs (and perhaps will perform worse in some other cases).
What should be done instead is that writes should be streamed (or pipelined) while reading replies in parralel, so that NBD reply delays doesn't bottleneck the whole transfer. But that actually requires a redesign of the whole NBD implementation which is not going to be a easy thing AFAICT.