Assuming I have the setting in the correct place, I'm not seeing any significant difference whether the setting is there or not.
without setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.5 MiB/s
with setting
Duration: 10 minutes
Size: 41.71 GiB
Speed: 72.87 MiB/s
CPU usage appears to be about the same.
@florent I already had that setting applied, at least I think so
Is this the correct config file?
/opt/xo/xo-server/config.toml
@florent said in S3 Backup "Please Match The Requested Format":
NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode
Duration: 9 minutes
Size: 41.52 GiB
Speed: 75.26 MiB/s
So didn't really make any meaningful difference (was 80MiB/s previously).
@florent said in S3 Backup "Please Match The Requested Format":
@vincentp If you backup multiple VMs in parallel, does the total speed stays at 80MB/s or does it scales with the number of VM ?
Yes it does.
NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode
I ran 2 full backups together - netdata and truenas are showed 1.45Gb/s so that's definitely an improvement - although it's using a lot of cpu
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23882 root 20 0 1022156 101484 20560 S 105.0 1.3 18:08.94 xapi
25349 root 20 0 31884 3872 2660 R 66.0 0.0 0:25.54 tapdisk
10163 root 20 0 32696 4896 2900 R 28.7 0.1 2:25.91 tapdisk
VM1
Duration: 10 minutes
Size: 41.51 GiB
Speed: 67.6 MiB/s
VM2
Duration: 10 minutes
Size: 23.25 GiB
Speed: 39.13 MiB/s
Some other backups are running at the moment, when they are done I will measure VM1 by itself to compare to the earlier one I posted.
@olivierlambert said in S3 Backup "Please Match The Requested Format":
Do you have NBD enabled on the network used by XO to backup?
yes.
I will try the insecure nbd - ok without encryption since its a direct connection between the machines
@olivierlambert said in S3 Backup "Please Match The Requested Format":
80MiB/s between LA and Sydney is already pretty impressive
No, that's local to sydney - no backups occuring between sites.
@planedrop I haven't used smb for backups but will test that - currently testing with nfs which I imagine would be faster than smb though.
I have 3 sites, each with a separate XO instance managing them
LA DC
I tried CR between the hosts, and it works well.. once you get past the full backup - but those take 24hrs (800GB vms),
S3 Backup - same experience - backups seem to work well once you get past the full backup - but that sometimes takes longer than 24hrs and runs into the next backup schedule. XO has access to the 10G interface and I confirmed it's using that.
Sydney DC
Testing full backups to nfs on truenas over 25G network(NDB Enabled) with the shared option to use multiple files enabled. Also XO is connecting via http to avoid stunnel which was using a huge amount of cpu when I first tried it, and it has access to a 25G interface and I confirmed via netdata that the correct interface is being used for the backups
Small vm test.
Duration: 9 minutes
Size: 41.48 GiB
Speed: 81.03 MiB/s
More testing required but its frustrating that we cannot get better speeds than this - could have saved some money and just used 1G cards in the machines!
HomeLab
just doing disaster recovery backups to 2nd local storage on the machine for now (and just as slow) - will use the truenas once I get it in the rack this week.
The bottleneck with the backups is definitely not the network, even on the homelab machine that backs up to local storage I'm only seeing around 60MiB/s
@planedrop TBH I didn't monitor for the full 24hrs - but the transfer speeds were just too slow. The same happens with CR between hosts - way too slow to be usable.
Backup performance on xcp/xo is a major problem - currently battling with this at another site where we just installed a truenas server with dual 25G direct connections between the xcp host and the truenas server (with NDB - and we see < 100MiB/s transfers (nfs share on truenas scale latest) - this time the xcp host is a dual eypc 7543 with all nvme and tons of ram.
This thread did help a bit - top on the host was showing stunnel using a lot of cpu - switching the connection from xo to the server tp be http removed stunnel from the equation but it's still too slow - and I did read somewhere that in the next xcp/xen version http is going away and everything will be https only.
I'm currently researching backups on proxmox to see if they have the same sort of performance issues. I really like xcp and xo but when we end up with backups taking so long that they overlap with the next backup schedule that is not usable.
@florent said in S3 Backup "Please Match The Requested Format":
@vincentp Did you try activating the NBD option ?
Can you back one VM on a NFS on a separate job to check if it's really reading speed that is slow, or if it's the S3 and CR writer ?
No, I don't have any shared storage at that site so cannot test NFS.
I do have NDB enabled on the direct 10G connection between the two hosts.
@planedrop I'm about to give up on using backblaze - the peformance is really variable, but I am seeing full backups take > 24hrs at the moment. I'm not sure where the bottleneck is, servers are co-located in LA - backblaze region set to us.west - 1Gbps connection but I just cannot get the backups done in a reasonable time. I'm having similar issues with continous replication between the 2 servers (which have a 10G direct connection) - the incremental backups are fine, but the full backups just take too long.
For now I'm just going to focus on backing up the data inside the vm's (db, websites etc) as that will be quicker.
@olivierlambert said in Continous Replication Network:
XCP-ng can't push anything, the traffic is always fetch by XO (so the flow is host 1 -> XO -> host2)
Perhaps I need a 10G switch then. Is the replication speed the same as the migration speed (which only just got up to 1Gbps with multiple migrations running)?
@olivierlambert said in Continous Replication Network:
how could XO transfer the CR if XO doesn't have access to this network?
I guess I thought that XO was directing xcp to do the replication - as a newbie to the platform it's hard to know what is provided by XO and what is provided by the platform.
@olivierlambert I have that option set, and it's working for vm migration.
However this is a direct connection between the two hosts, so I cannot connect XO to it.
@Darkbeldin worked perfectly thanks.
Testing out Continous Replication - but it seems the transfer happens over the management network interface (so through our 1G switch to the other host) rather than over the Migration interface (10G direct connection to other host).
Our plan is to have CR both ways, replicating 2 vm on each host.
Given how slow storage migration is, I am concerned about how long a full replication would take and that we would saturate the management interface with replication tasks, and we also have daily backups to Backblaze using the same interface.
Is there a way to change this?
@Darkbeldin Thanks, I will give that a try, when the current running tasks are done.
I had some backups that ran while the vms were live migrating (including vdi's) to another host - the backup shows as started but there are no tasks running for those backups and I cannot manually start the backups now.
How do I reset the backup status?
If I delete the backup jobs will it delete the backup data on S3/Backblaze (that's ok, just need to know)?
@planedrop said in S3 Backup "Please Match The Requested Format":
I am also curious if multiple backups at the same time results in faster overall speeds? Not quite sure how parallel a single VM backup is, but maybe each can run at that 15 or so MiB/s so multiple backups can use more bandwidth, I'll have to test this to find out.
So I did get to test this scenario out last night - finally got my production vms migrated over - that's another story - and did run 2 full backups at the same time.
Duration: 9 hours
Size: 512.03 GiB
Speed: 16.98 MiB/s
Duration: 7 hours
Size: 511.97 GiB
Speed: 20.25 MiB/s
I did monitor for a short while (it was 2am) with netdata and observed the outbound connection running between 600Mbs and 900Mbs - so I guess that's about as good as it's going to get (1Gbs connection). In the long term I really need to get a truenas server at this site and run the backups to backblaze from there.
Single host - no shared storage.
When I reboot the host, I have to use the Force Restart Host feature due to the "no hosts available" error that happens when just using the Restart Host function.
So I used the Force Restart, and what I observed is :
Linux vms all shutdown cleanly (status turns orange)
Windows vm's appear to have been forced shutdown (no orange status).
XOA vm shutdown (clean)
Host restarted.
This host took probably < than a minute to shutdown (amd gen 3 all nvme, it's fast).
All the vms have the xcp-ng client tools installed. All patches installed on host. If I manually shutdown a windows vm from XO then it does shutdown cleanly.
Using XO from source (2dbe35a31c4742d0ee79e40eef43e53bcef4615f) built 12 march. Looking at the commit history I don't see any changes around this area.
I have rebuilt XO from the latest commit today, but cannot restart the host again at the moment - and I have concerns about screwing up the windows vms doing this too often (I do have backups but still) - windows is fragile!
Before I create an issue on github, is this as designed or a bug? My expectation would be that all vms with the client tools installed would be shutdown cleanly.
Is there something I can look for in the log files?
@planedrop I plan on spinning up a few more vms this week to test the backups speed, I will post my results when done.
We are migrating from Hyper-V - where we had an adhoc backup scheme at best. We have 2 identical servers, and each host backed up to an smb share on the other host and then we periodially pulled those down locally (the hosts are in a LA, I'm in Canberra, Australia).
One of the machines hung on reboot due to a failed raid array (hyper-v makes it hard to monitor things so probably failed weeks ago) - so I decided to migrate to xcp-ng as we did that recently with our dev servers in Sydney (we do have a truenas server and synology for shared storage/backups there) and it was a major upgrade on hyper-v!
In the LA DC we just have those two servers and no shared storage - which makes life difficult since you can't use the hosts for backup like we did with hyper-v. So I do need to use a cloud service for backups - it will be a while before I can afford another storage server.