Performance enancement for backups possibe?

Forza

Hi,

As far as I can tell, XO uses synchronous transfers when doing backups. It looks like it reads a chunk and then sends it off to the remote target, then reads another chunk, sends it off and so on.

I wonder if this can be improved by using asynchronous transfers. This would mean using a separate reader and writer with a buffer in between. This could improve latencies and overall throughput quite a bit.

One example of this, unrelated to XO, is to use mbuffer instead of pv when using zfs send|receive or btrfs send|receive. With pv, the sender has to wait while the receiver is processing each block of data, which makes the whole chain slow. With mbuffer, the sender and receiver is handling data asynchronously and they continue at their full speeds as long as there is data in the buffer. Small interruptions in read/write speeds doesn't affect as much, which is really good when sending things over a network connection.

Here is an example running on a local box from a nvme drive to a spinning hdd:

Using pv:

# btrfs fi sync /mnt/nasvol-hdd/
# echo 3 > /proc/sys/vm/drop_caches
# time btrfs send /mnt/rootvol/snapshots/root.20210908T0701/ |pv -pr | btrfs receive /mnt/nasvol-hdd/test/
At subvol /mnt/rootvol/snapshots/root.20210908T0701/
At subvol root.20210908T0701
[ 175MiB/s] [                                                                                                                             <=>           ]

real    0m52.807s
user    0m4.102s
sys     0m41.291s

Now using mbuffer:

# btrfs fi sync /mnt/nasvol-hdd/
# echo 3 > /proc/sys/vm/drop_caches
# time btrfs send /mnt/rootvol/snapshots/root.20210908T0701/ |mbuffer -m1g | btrfs receive /mnt/nasvol-hdd/test/
At subvol /mnt/rootvol/snapshots/root.20210908T0701/
At subvol root.20210908T0701
in @ 36.0 MiB/s, out @ 34.0 MiB/s, 8420 MiB total, buffer   0% full
summary: 8489 MiByte in 42.5sec - average of  200 MiB/s

real    0m45.079s
user    0m3.082s
sys     0m46.534s

(I did run each test several times and the timing is consistent between runs).

There is an interesting thread about this over at https://www.cnx-software.com/2016/12/16/compress-decompress-files-faster-with-lbzip2-multi-threaded-version-of-bzip2/#comment-537262

olivierlambert

I have no idea if it's something we can do in our context. But pinging @julien-f and @florent about it anyway

julien-f

Hey there,

XO simply downloads a VHD file from XCP-ng/XenServer, and writes it on the remote as the data is coming.

Also, XO is written using Node.js which is a high level JavaScript platform, and we have no control on how it writes a data stream on a file.

Forza

@julien-f said in Performance enancement for backups possibe?:

Hey there,

XO simply downloads a VHD file from XCP-ng/XenServer, and writes it on the remote as the data is coming.

Also, XO is written using Node.js which is a high level JavaScript platform, and we have no control on how it writes a data stream on a file.

Hi. This is not the observed behaviour I see. It's definitely a read some, send some, read some step wise behaviour that I saw. But I'll look at it again. Maybe some other factors were playing in here.

julien-f

Ok, let me know