Painfully slow backup with Xen Orchestra from sources in freeBSD jail

zizzithefox

Hello guys, this is just a report about a silly problem that I "solved", which may be of interest to someone. I have to point out the scenario I am describing is not supported at all. So.

Short story: do not use sync=always on a ZFS remote of any kind with Xen Orchestra.

Long story: I have switched from VMware a year ago and my setup is:

a FreeNAS server for my backups,
two XCP-ng 8.1 hosts, with local nvme or hdd storages.

Basically, all on ZFS.

I have been messing around with XCP-ng and Xen Orchestra for quite some time now, with a decent level of satisfaction I must say, except backup speeds on my home 1Gbit network has never been quite as I expected.

First I tried XO on a bhyve ubuntu VM, reaching somewhere around 20-30 MB/s. Then I moved it on a linux vm on one of the hosts, still around 30+ MB/s. Not painful for my needs, but annoying.

Finally recent versions of XO updated node.js to version 12 (as opposed to version 8 which in FreeBSD has been EOL for quite some time now) and I figure: since I am using XO from sources, why not run it directly from a FreeBSD (FreeNAS) jail on the backup target? No VM, less network overhead and more direct access to the backup storage (I mean: nullfs, but who cares).

Given I don't know s**t about FreeBSD, nodejs & C., so, so many thanks to this guy:

https://sysadm.russerver.org/wiki/Install_Xen_Orchestra_on_FreeBSD

Pretty straightforward, it worked like a charm.

Problem was: backup speed of 5 MB/s. Uh? I was getting so pissed.

Then I remembered this is around the expected performance with sync=always on the same pool (raid-z2 6x4TB HDD) without the SLOG nvme device. Weird coincidence, right?

So, zfs set sync=standard on a dedicated dataset and voilà, 90-100 MB/s. Now, that is more like it and I am content with this setup, but I still wonder: what is going on here?

If I run backups via an agent like veeam linux/windows free agent, acronis, etc, let's say on SMB shares, backup speed is stable at least at 50+ MB/s with sync=always. Locally, if I move data from the second pool I have, it's minimum 100 MB/s, even on a pretty full pool with less than 20% free space.

I am sure there is some explanation for the abysmal performance I am getting with synchronous writes for XO backups.

So that's the story.

To sum it up, the only thing I don't like at the moment with XCP-ng and XO is the lack of support for quiescent snapshots.

I mean, that's very, VERY bad. Memory snapshot? Mmm... Maybe. But no, sorry. If this was a production environment, I would probably end up with agent-based backup.
Of course, this is no topic for this forum because we all know whose "fault" that is.

olivierlambert

Hi,

FreeBSD install was already documented officially: https://xen-orchestra.com/docs/installation.html#freebsd
You could have test to use HTTP instead of HTTPS to see if perfs are improved
New XO backup code will come in few releases that will speed up things
Quiesce snapshot: it's not needed for Linux VMs because of the disk driver automatically handling this.