Slow backups (updated XO source issue)
-
I had the same problem. I have a CR that until applying the setting took up to 5 days (I interrupted the backup to see where the problem is) to make a backup, now with the setting it takes 12 hours.
But the stability of the transfer is better by at least 15% compared to commit afadc8f95adf741611d1f298dfe77cbf1f895231.@Andrew said in Slow backups (updated XO source issue):
@julien-f The time it takes for CR to run seems to be back to normal with the added config
[backups.vm.defaultSettings] validateVhdStreams = false
I have Concurrency set to 4.
There's more than enough local ethernet bandwidth between machines and XO is running on the pool master. There's lots of cores and memory for XO. But with
validateVhdStreams = true
there seems to be some single threaded issues with node because it uses 90%-120% cpu.With
validateVhdStreams = false
the CPU usage is about 110%-150%, so still some single threaded issues but a lot less as node is able to use more than one core (but not all of them).Also... I see the warning that appeared from an earlier commit but does not cause problems:
transfer Suspend VDI not available for this suspended VM
-
-
@Andrew https://github.com/vatesfr/xen-orchestra/commit/136718df7e28d63f340c700564e5c1a90d1bf05e
Regarding the perf issue, I will work on it as soon as I'm able to reproduce on my side
If the impact is too big, we'll disable it by defaultβ¦
-
From what I've seen:
- when nothing is set, XOCE makes the transfer using a single vCPU to the maximum and the rest up to 5% max . The maximum transfer being up to 23Mb/s
- when the setting exists, XOCE uses all 12 vCPUs between 30% and 45%. The transfer is between 145Mb/s and 350Mb/s.
In both cases, the transfer is done on a 10Gib LAN and I use http mod for connecting to Servers
-
Hmm this points to a concurrency issue with the backup workers
-
This is on testing environment.
Version of XOCE
Transfer without setting
Transfer with setting
-
That's indeed very different. Out of curiosity, what's the CPU model?
-
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
-
I too noticed by backup times went sky high recently. I just made the suggested switch in the config and that is the difference between before/after. My box has 2x E5-2697a v4 processors.
-
I've disabled the VHD stream validation by default for now: https://github.com/vatesfr/xen-orchestra/commit/0444cf0b3bc6902fc1a995da5cf29474b7f4f167
We'll re-enable it when we'll find a better solution.
Thank you all for your reports
-
@julien-f Thanks. I updated to 0444cf0 and CR times are normal again (as expected).