S3 Backup "Please Match The Requested Format"
-
@vincentp If you backup multiple VMs in parallel, does the total speed stays at 80MB/s or does it scales with the number of VM ?
NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode
-
80MiB/s between LA and Sydney is already pretty impressive, knowing the latency between those. When you write a lot of small blocks, each block have to way for a round trip before being ACK. This takes a lot of time.
Higher the latency, longer the backup, except if we choose bigger blocks, which isn't trivial.
When using NBD, it should be a lot better however since we can have more blocks worked in parallel. I achieved a huge bump with more blocks at the same time.
Also, you can also try XO from the source on a physical machine to check the difference
(vs XO in a VM)There's many many many ways to get faster, what's important is to measure each modification boost, because this might help to identify bottlenecks
-
@olivierlambert said in S3 Backup "Please Match The Requested Format":
80MiB/s between LA and Sydney is already pretty impressive
No, that's local to sydney - no backups occuring between sites.
-
Do you have NBD enabled on the network used by XO to backup?
-
@olivierlambert said in S3 Backup "Please Match The Requested Format":
Do you have NBD enabled on the network used by XO to backup?
yes.
I will try the insecure nbd - ok without encryption since its a direct connection between the machines
-
-
@florent said in S3 Backup "Please Match The Requested Format":
@vincentp If you backup multiple VMs in parallel, does the total speed stays at 80MB/s or does it scales with the number of VM ?
Yes it does.
NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode
I ran 2 full backups together - netdata and truenas are showed 1.45Gb/s so that's definitely an improvement - although it's using a lot of cpu
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23882 root 20 0 1022156 101484 20560 S 105.0 1.3 18:08.94 xapi 25349 root 20 0 31884 3872 2660 R 66.0 0.0 0:25.54 tapdisk 10163 root 20 0 32696 4896 2900 R 28.7 0.1 2:25.91 tapdiskVM1
Duration: 10 minutes
Size: 41.51 GiB
Speed: 67.6 MiB/sVM2
Duration: 10 minutes
Size: 23.25 GiB
Speed: 39.13 MiB/sSome other backups are running at the moment, when they are done I will measure VM1 by itself to compare to the earlier one I posted.
-
@florent said in S3 Backup "Please Match The Requested Format":
NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode
Duration: 9 minutes
Size: 41.52 GiB
Speed: 75.26 MiB/sSo didn't really make any meaningful difference (was 80MiB/s previously).
-
@vincentp
if you are on master, can you apply the setting from here : https://xcp-ng.org/forum/topic/7209/slow-backups-updated-xo-source-issue/8 ? -
@florent I already had that setting applied, at least I think so
Is this the correct config file?/opt/xo/xo-server/config.toml -
Assuming I have the setting in the correct place, I'm not seeing any significant difference whether the setting is there or not.
without setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.5 MiB/swith setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.87 MiB/sCPU usage appears to be about the same.
