Extrlemy slow backup speec = about few MB/s
-
Question: how the backup differs from DR ? And why I cannot "Backup" to the SR?
-
Because… a backup is targeting a remote, and DR a SR. Have you even watched this video? It's a good recap on all backup options
Also doing another DR test on another SR than source, it's a bit faster because the disk don't have to read and write at the same time:
-
Those values are SICK. I know that you are not responsible for this situation becouse the Citrix is the author of the xenserver and this issue is deep in code. But I didn't expected this. Currenly all my exported VM's occupies about 3TB of space. And it will grown. Divide 3 or 4 or 7 TB of data by 4, 6 or even 21MB/s and will will know that this is a problem. XS/XCP is great but of nothing will change in this matter i think that we will be forces to switch to VmWare in next year.
@olivierlambert do You have xcp-ng 8 at LAB and can test the DR backup speed and give some info if something changed in this matter?
-
Those value are low because of small VM size. There's is a non negligible time here to create the VM objects etc. that you can't reduce.
So on a larger transfer, speed will be more about 30 or 40MiB/s.
CR is a bit faster because it's VHD export, which is faster than XVA creation.
Also ZSTD compression is a good way to increase speed.
-
@akurzawa This test was done on XCP-ng 8.0
That's what I say since the start: I can't reproduce your issue
-
@olivierlambert said in Extrlemy slow backup speec = about few MB/s:
So on a larger transfer, speed will be more about 30 or 40MiB/s.
I haven't niticed that - moreover I've noticed that the general import/export speed generally decrases over time. Also I relay on the export speed reported after backup finish.
CR is a bit faster because it's VHD export, which is faster than XVA creation.
Also ZSTD compression is a good way to increase speed.I'll test CR more deeply, but I don't like the idea when production VMs will have 2-month old shapshot... but I think what will simply not work due the fact that I use rolling snapshot in many different time configuration and CR backup only will make a mess in this area.
Any way thank You for answers and suggestions.
-
You can use both at the same time CR and rolling snapshots (so they'll work "together"). Also, if you CR every day or week, your oldest snapshot will be only 1 day or 1 week old. Not 1 month.
Regarding speed, that's not the case here, longer the backup, smaller the "static time" to export (attach VBD/VDI to the control domain and pass the content to vhd-util) on the total time.
-
Can You explain me more how the 'base backup' of CR works?
-
Have you watched the video I posted earlier?
-
We have been using CR at work for over a year now and it is working great. There is no point of using the DR-feature since it is way slower and doesnt support incremental backups.
-
@akurzawa, slow speeds like that have been inexplicable to me as well. With the Spectre patches and security recommendations to turn off HyperThreading, we need to squeeze performance from other parts of the infrastructure. Storage has got to be better.
By default, Dom0 is not allocated very much RAM (maybe 4GB?) which must limit how much shared memory and ramdisks can offer to increase performance. How much RAM do you allocate to Dom0 on your hosts? @Balardo posted SOLVED: Slow disk write speed inside VM, increasing the speed of VM performance by tweaking xen_blkfront parameters and he did it on the same eleventh generation Dells as yours.
I know our numbers do not come close to saturating 1Gbps ethernet, but regardless the best Citrix and Unitrends Enterprise Backup support would do is suggest moving to a hyperconverged infrastructure and away from shared storage. Have you heard of RemoteDirectMemoryAccess? RoCE and iWARP are the two major modern versions according to the SNIA.org and would be used by all hypervisors. The SNIA RoCE vs iWARP storage debate PDF is talked about in this webinar on youtube.
If i had the time and hardware, I would like to compare the opensource ProxMox Hypervisor performance with the same tests using CEPH because CEPH also uses RDMA technology. If it is faster, what is different under their /proc/config.gz? For that matter, vmware would also have a /proc/config.gz.
Lastly, I have always heard that Oracle contributed packages or tweaks to drastically speed up storage, but not sure these are applied by default in other kernels.
-
Backup speed has almost nothing to do with RDMA, hyperconvergence vs shared storage or tweaking VM perfs. It's mainly a dom0 userland process/tools that must be improved.
-
@rjt said in Extrlemy slow backup speec = about few MB/s:
How much RAM do you allocate to Dom0 on your hosts
16GB to dom0, but the backup speed is still dramatic - in Disaster Recovery its a few mB/s. Nothing has changed from since then. I think this is dead end, and will have to make some business desicions regarding the hypervizor when jumping next year.
-
And what about CR?
Obviously, you have a problem on DR/VM export mechanism (XVA). Also 8.1 will bring an improved XVA generation system, that will solve your issue.
-
@olivierlambert Is there a github issue for this? Does the xcp-ng or xen project have something in mind?
-
For what issue?
-
@olivierlambert Would the ”dom0 userland process/tools that must be improved” actually mean functionality that does not actually require Dom0? Wondering if dom0less Xen would speed things up.
-
And how/where would you attach the disk to be exported? You must have a dom0 for some operation (or at least an extra domain for export).
I was more talking about improving tools that are exposing the disk itself.
-
@olivierlambert said in Extrlemy slow backup speec = about few MB/s:
And what about CR?
Obviously, you have a problem on DR/VM export mechanism (XVA). Also 8.1 will bring an improved XVA generation system, that will solve your issue.
I'm planning the upgrade from 7.6 to 8.x in about 3-4 months, also I'll build the test environement. I'll see how it goes.
-
Hi all,
i'm faced with the same issue:
TestVM (redfour) Snapshot Start: Jan 13, 2020, 10:13:51 PM End: Jan 13, 2020, 10:13:56 PM Local iSCSI transfer Start: Jan 13, 2020, 10:13:56 PM End: Jan 13, 2020, 10:17:18 PM Duration: 3 minutes Size: 813.17 MiB Speed: 4.03 MiB/s Start: Jan 13, 2020, 10:13:56 PM End: Jan 13, 2020, 10:17:18 PM Duration: 3 minutes Start: Jan 13, 2020, 10:13:51 PM End: Jan 13, 2020, 10:17:19 PM Duration: 3 minuteswhere the backup volume is served on iscsi by a machine having raid5 over 1 Gbps back to back connection to the hypervisor.
Mouting that volume in the hypervisor and doing some transfers, traffic easily goes up and beyond 800 Mbps whereas the backups takes forever to execute.
I am uzing zstd as compression and xcp-ng 8.0 on the hypervisor
One thing i noticed is that xo-server takes 100-130% cpu when the backup is executing.
The vm above is about 2gb uncompressed, however larger VMs take over 8 hours to complete a normal backup, nothing fancy.Thanks,
Viorel
