Benchmarks between XCP & TrueNAS
-
@john-c good info, I didnt know this.
We're using TrueNAS for our backup NFS which is the target for our XOA backups. -
@john-c thank you for this info, I will investigate immediately tonight.
(not wanting to misuse the XO forum for TrueNAS questions), but we're moving into production 4x TrueNAS servers next week. We have a couple of TrueNAS core servers already and have been very happy with the reliability it presents, and we opted to install TrueNAS core on the new servers simply on the fact of "what you know is what you know". It however has also come to my attention that TrueNAS Core is effectively "end of life"
Based on what you are sharing now, it would be a stupid idea for us to continue using Core if the forecast of better performance for our usecase would be present. We don't use the TrueNAS servers for anything else than NFS shares (could care less about the rest of the apps / plugins etc).
Are you perhaps using Scale for NFS VM sharing? And if so, would you recommend we onboard ourselves with Scale rather than Core?
-
@mauzilla I would definately check on TrueNAS Scale - Its their "new" thing and its been stable for a while now.
Big difference is that its based on Linux on and not FreeBSD which the old FreeNAS/TrueNAS was. -
@nikade we'll be doing it right away, and given that I now know that there is potential challenges I can address with performance I'd rather give it a go. Thank you all for your feedback so far, a huge relief!
-
@mauzilla Happy to help, good luck with your project!
-
Considering the feedback we got yesterday (and not previously knowing TrueNAS support ISCSI), we took a chance this morning and setup a ISCSI connection:
- Created a "file" ISCSI connection on the same pool with the SLOG
- Setup in XCP / XO
- Migrated VM to the ISCSI SR
And wow, what a difference, jumping from approximately 44.6MB to 76MB/s and IOPS jumping from 10.9 to 17.8k
This leads me then to believe if we designate a sub section of the pool in TrueNAS for ISCSI, we will be able to reap the benefits of both by leaving "data" disks on the NFS (so that we can benefit from thin) and put disks within the ISCSI SR (understanding that it will be full VDI)
Questions:
- With regards to backups / Delta replication, can I mix / match backups where some disks are VHD (thin) and some are full LVM? I assume that the first backup will always be the longest (as it will need to backup the full say 50GB of the disk isntead of the thin file) - Will this impact delta replication / backups?
- Any risks with the intended setup? I can only assume that other people might be doing something similar?
I was really amazed with the simplicity of the ISCSI setup between TrueNAS and XCP - We have an equallogic and I attempted to setup ISCSI but it was extremely difficult if you dont understand the semantics involved
-
@mauzilla sounds like we are doing similar things! I gather you're now running TrueNAS Scale as we are? We decided to go with RC2 of Electric Eel since there's been some big architecture changes between 24.04 and 24.10.
We are just running it through testing and have run into performance issues for backups to an NFS target and I was interested to read the huge gain in speed you got from moving to iSCSI.
I was thinking of seeing if we can backup to an iSCSI target while keeping many of the VMs on NFS. Have you tried this and had any success?
-
@andrewperry - we've started our migration from local storage to TrueNAS based NAS (NFS) the last couple of weeks, and ironically ran into our first "confirmed" issue on Friday. For most of the transferred VM's we had little "direct" issues, but we can definately see a major issue with VM's that are reliant on heavy databases (and call recordings).
We will be investigating options this week, will keep everyone posted. Right now we dont know here the issue is (or what a reasonable benchmark is) but with 4 x TrueNAS, 2 pools per TrueNAS with enterprise SSD, SLOG's for each pool and a dual 10GB Fibre connection to Arista having only 2-3 VM's per pool is giving mixed results.
We will first try to rule out other hardware but will update the thread here as we think it will be of value for others and maybe we find a solution. Right now I am of the opinion that it's NFS, hoping we can do some tweaks as the prospects of just running iSCSI is concerning as thin provisioning is then not possible anymore.
-
@mauzilla we've been running iscsi based backups with our previous 'bare metal' Debian Xen system, thanks to rdiff-backup. It worked really well for our purposes and I am still a bit uncomfortable with XO / XCP-ng backing up over a 1Gbps link. Hopefully a 10Gbps link will resolve that.
Have you considered using ZFS replication with TrueNAS as part of your backup strategy?
-
@andrewperry yes, we replicate TrueNAS to a "standby" TrueNAS using zpool / truenas replication. Our current policy is hourly. We then (plan) to do Incremental replication of all VM's to a standby TrueNAS over weekends giving a 3 possible methods for recovery.
Currently we're running into a major drawback with VM's on TrueNAS over NFS (specifically VM's that rely on fast storage such as databases and recordings). We did not anticipate such a huge drop in performance having VHD files over NFS. We're reaching out to XCP to ask them to give us some advice as it could likely be in part due to customization we can make.
