S3 Backup "Please Match The Requested Format"

olivierlambert

80MiB/s between LA and Sydney is already pretty impressive, knowing the latency between those. When you write a lot of small blocks, each block have to way for a round trip before being ACK. This takes a lot of time.

Higher the latency, longer the backup, except if we choose bigger blocks, which isn't trivial.

When using NBD, it should be a lot better however since we can have more blocks worked in parallel. I achieved a huge bump with more blocks at the same time.

Also, you can also try XO from the source on a physical machine to check the difference (vs XO in a VM)

There's many many many ways to get faster, what's important is to measure each modification boost, because this might help to identify bottlenecks

vincentp

@olivierlambert said in S3 Backup "Please Match The Requested Format":

80MiB/s between LA and Sydney is already pretty impressive

No, that's local to sydney - no backups occuring between sites.

olivierlambert

Do you have NBD enabled on the network used by XO to backup?

vincentp

@olivierlambert said in S3 Backup "Please Match The Requested Format":

Do you have NBD enabled on the network used by XO to backup?

yes.

I will try the insecure nbd - ok without encryption since its a direct connection between the machines

olivierlambert

@florent is there a way to easily raise the blocks in parallel to see the impact for @vincentp ?

vincentp

@florent said in S3 Backup "Please Match The Requested Format":

@vincentp If you backup multiple VMs in parallel, does the total speed stays at 80MB/s or does it scales with the number of VM ?

Yes it does.

NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode

I ran 2 full backups together - netdata and truenas are showed 1.45Gb/s so that's definitely an improvement - although it's using a lot of cpu

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23882 root      20   0 1022156 101484  20560 S 105.0  1.3  18:08.94 xapi
25349 root      20   0   31884   3872   2660 R  66.0  0.0   0:25.54 tapdisk
10163 root      20   0   32696   4896   2900 R  28.7  0.1   2:25.91 tapdisk

VM1
Duration: 10 minutes
Size: 41.51 GiB
Speed: 67.6 MiB/s

VM2
Duration: 10 minutes
Size: 23.25 GiB
Speed: 39.13 MiB/s

Some other backups are running at the moment, when they are done I will measure VM1 by itself to compare to the earlier one I posted.

vincentp

@florent said in S3 Backup "Please Match The Requested Format":

NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode

Duration: 9 minutes
Size: 41.52 GiB
Speed: 75.26 MiB/s

So didn't really make any meaningful difference (was 80MiB/s previously).

florent

@vincentp
if you are on master, can you apply the setting from here : https://xcp-ng.org/forum/topic/7209/slow-backups-updated-xo-source-issue/8 ?

vincentp

@florent I already had that setting applied, at least I think so
Is this the correct config file?

/opt/xo/xo-server/config.toml

vincentp

Assuming I have the setting in the correct place, I'm not seeing any significant difference whether the setting is there or not.

without setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.5 MiB/s

with setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.87 MiB/s

CPU usage appears to be about the same.