XEN Orchestra Snapshot Space for Backups
-
Hallo,
I have a question about the space I need for backup VM´s to an external storage.
Situation:
We want to setup a new server system using XCP-NG and XO. The SR should be on a freenas ZFS machine connected over iSCSI. So thin provisioning is sadly not possible.The storage should contains two DB systems (Windows Server 2016 or 2019 with SQL Server Express, Ubuntu 18.04 with mysql). To get a good performance we want to use SSDs in the freenas system.
The Backup Storage should be a freenas system too, but with some large HDDs.
At this moment the space used by both VMs is approximately 430GB (Windows 400GB Linux 30GB). We believe that 700GB space should be enough for the next years (Windows 600GB Linux 100GB).
Because good SSDs are not realy cheap, we don´t want to waste space.
Questions:
Could you please tell me, how many snapshots are required on the SSD SR to make external backups using the different backup methods of XO?
Full Backups
Rolling Snapshots
Delta Backup
Disaster Recovery
Metadata Backups
Continuous ReplicationIf I need more than one snapshot e.g. for Delta Backups, does the second snapshot also need the full space of the VM?
By the way:
Do you know if Quiesced Snaphots are supported from SQL Server Express?
As far as I read they should, but it was not a reliable source.
I think the ubuntu mysql VM sould not support them - right?Thanks in advance
Buschi
-
Have you seen https://www.youtube.com/watch?v=FfUqIwT8KzI ? It's a good intro to understand how it works.
- Can you switch to NFS in your FreeNAS? You'll reduce a LOT your required space to make backup/snapshots
- All delta mechanism will require to keep one snapshot. On a thick pro storage, you'll need to roughly double space needed for your VMs. On a thin pro storage, it's merely few extra %.
- Only the first snap will require doubling space, not extra snaps.
- VSS: it's removed by Citrix in 8.1, so it will be also removed in XCP-ng 8.1. However, we are working on another way to make consistent backups on Windows. Until then, you can use "Offline snapshot" option in Advanced view of your backup job.
- Linux VM: they are aware of a snapshot because of the disk driver they have by default. No problem at all without VSS, it's somehow "bundled".
-
Hello Olivier,
thanks for the quick response.
Have you seen https://www.youtube.com/watch?v=FfUqIwT8KzI ?
I did not know the video yet, but now I watched it. Thank you.
- Can you switch to NFS in your FreeNAS? You'll reduce a LOT your required space to make backup/snapshots
We make some tests about 3 years ago. At this time NFS was slower than iSCSI. Overall, we felt better with iSCSI. Are there any other experiences in this regard?
- Only the first snap will require doubling space, not extra snaps.
Also on thick provisioning?
- Linux VM: they are aware of a snapshot because of the disk driver they have by default. No problem at all without VSS, it's somehow "bundled".
Did I understand you right, that I could backup a linux VM with mysql database every time without any risk of losing data?
Backup Flow:
- Backup start
- XO takes Snapshot1
- XO exports Snapshot1 to another System
- IF Full-Backup: XO deletes Snapshot1
- IF Delta OR Contiuous Backup: XO archives the snapshot until the next backup run
- Next Delta Backup Start:
- XO takes Snapshot2
- XO "exports" the Delta between Snapshot1 and Snapshot2 to another System
- XO deletes Snapshot1 and keeps Snapshot2 for the next delta backup run
- and so on
So while the backup runs I need space for 1 snapshot when I do a full backup, and 2 snapshots if I do a delta backup.
Is this right?
Thank you very much
Buschi
-
NFS is SO much simpler than iSCSI and less prone to issues. I would say around 30% of our pro user trouble tickets involve fixing people's iSCSI issues, and almost zero NFS related tickets. I would retest your performance comparisons, the performance gap between the two has shrunk considerably in the past. Not to mention you will also have thin provisioning
-
@fohdeesha My experience is the exact opposite - both regarding issues and performance. That said, part of your observation of support tickets differences could be that those Pro users submitting a lot of iSCSI-related tickets are users who don't have the knowledge or experience to be deploying iSCSI in the first place, and should stick to NFS anyway. Those users submitting NFS tickets are likely the above users, and also us standard iSCSI guys who are trying to accommodate a business need with NFS and find it to be very finicky and frustrating to setup in various types of environments.
I'm not sure what hardware you guys run on all the time that we get a lot of "NFS is amazing, get rid of your iSCSI!"; because a lot of us aren't running the latest and greatest, and, iSCSI is a considerably more efficient type of storage setup.
I digress.
On topic, my experience is that backups take somewhere between 2x and 3x the storage space initially consumed by the VM (most of this being the snapshot itself); and, I'm curious about the VSS stuff you guys are working through. Have you guys reached out to someone like Quadric to query how their doing their VSS? (officially supported inside both Xen, XCP-ng, and Hyper-V environments)
-
We did benchmarks, and we had the same perf level between both NFS and iSCSI.
-
@jtbw911 This is on mostly gen 12 dell hardware (R720, R620, etc) so coming up on 8 years old now, not exactly what I would call the latest and greatest. If you peruse the freenas forums, you will also find a lot of users there (mostly esxi users in fact) have also been migrating from iSCSI to NFS as the performance delta is all but gone these days (aided by running a recent freeNAS release). Some tickets are certainly related to configuration mistakes, but they are a small minority. Even perfectly setup, iscsi is much more prone to race conditions, leaving a giant mess of VDI chains when there's an issue, etc. If you'd like to test for yourself, just interrupt the connection to an iSCSI SR during a chain operation, it will take hours to clean up the aftermath. With NFS, there's no issue at all, it will gracefully remount and continue working when the remote returns. This is one example of the tickets we have to deal with.
If you are set on iSCSI that's of course an option, (it's still supported for a reason), but as you note you will be wasting a lot of storage space to snapshots
-
@fohdeesha I'm going to be honest with you. I've had terrible luck with FreeNAS more times than I can count. It's always been a mix of hardware compatibility issues, appearance of reliability, a not-so-great interface, and a slew of others. I've found unRAID to suffer from similar things. I know a lot of people who use it; and, online, it's obviously widely used by a very large audience - it just doesn't seem to work well for me. I just recently spent at least a couple days trying to use it in front of a fiber-channel array to no avail.
I am about to drain and rebuild an x86-based storage appliance (currently running XPenology) in the next few weeks, so, I may go ahead and try it on this piece of hardware and see where it gets me. I loved OpenFiler back in the day, and, had pretty good luck with Nexenta for a while; but, one is just asking to be compromised using it in 2020, and the other doesn't scale well with modern storage without spending a fortune. XPenology works great, and gets you all the ease of Synology; but, it is very finicky about hardware. I ultimately had to settle with a solution that I really don't like; but, it works for now (Windows Server on vmWare, with the FC array passed through as a RAW disk).
-
@jtbw911 I won't argue with you there, freenas has definitely had some issues. I've been lucky enough to make it work for my own needs, but it certainly isn't plug and play, especially when it comes to backing virtualization platforms. They've at least acknowledged the awful old interface and have been working on replacing it but of course that's brought more issues with the new UI code, I've just been toughing it out until it approaches stable. I will say the bug(s) I've filed on their bug tracker got very quick attention and patches, at least. Just don't even bother reporting things on their forum, I won't mention any usernames but I'm sure you can guess
-
You can also use a "generic" distro to handle your NFS SR (Debian or CentOS).
There's not a lot of work involved after NFS is ready. You can even use ZoL for good disk management.
-
@Buschi said in XEN Orchestra Snapshot Space for Backups:
Quiesced Snaphots are supported from SQL Server [Express]
Quiesced Snaphots will not be available in next releases of xenserver / xcp-ng probably too, so be aware of that.
Snapshot and restoring can do mess, you can't relay on snapshots in regards of SQL dabase or Active Directory backups, but You probably know that
Olivier how do You plan make the snapshots consistent? Are You planning to develop the Quiesced Snaphots feature or You are working on something different?
-
We have a coming feature to get consistent backups on Windows VMs (or any kind of VM in fact).
-
@olivierlambert I'll give you that one if all I need is a simple NFS share for something like ISOs. I've got a virtualization stack that's got a demand of upwards of 10,000 IOPs, and, every time I've tried to push to NFS, performance drops like a rock. My XPenology setup (with SSD cache) worked quite well. We'll see how things develop here in a couple\few weeks as I dive back in with this appliance rebuild.
-
@akurzawa said in XEN Orchestra Snapshot Space for Backups:
Snapshot and restoring can do mess, you can't relay on snapshots in regards of SQL dabase
Isn't any SQL server crash safe? I.e. they should handle a system power-loss and gracefully recover (journals). A filesystem snapshot is equivalent of a power loss. You need to configure your databases accordingly.
-
@S-Pam
Of course that's true.My car has airbags and is well insured, but I would not voluntarily drive into a wall just to see if it worked.
-
@S-Pam said
Isn't any SQL server crash safe? I.e. they should handle a system power-loss and gracefully recover (journals). A filesystem snapshot is equivalent of a power loss. You need to configure your databases accordingly.
Most of maria/mysql have enabled innodb doublewrite enabled by default:
https://dev.mysql.com/doc/refman/5.7/en/innodb-doublewrite-buffer.htmlBut just in case of Active Directory... I really really want to have consistent snapshot/backups
