Switching to XCP-NG, want to hear your problems
-
This is a duplicate of my reddit post so I apologize. The XCP-NG subreddit doesn't seem to have a ton of traffic so I wanted to spread this post a bit wider.
https://www.reddit.com/r/xcpng/comments/1czm1oi/switching_to_xcpng_want_to_hear_your_problems/
My org is strongly considering switching to XCP-NG as our hypervisor ASAP. We need to make a decision on this in the next couple weeks. Our environment is probably medium-sized, with over 1000 VMs. I've done a lot of testing and it seems like a good product so far. We're worried about what we might be missing, though.
What issues have you encountered? What has thrown a serious wrench in your plans or been hard to deal with?
A couple things I've run into that I'm curious to hear about more:
-
Seems there are issues with Secure Boot on Windows: https://docs.xcp-ng.org/guides/guest-UEFI-Secure-Boot/#:~:text=Enabling Secure Boot for the,Citrix work with Secure Boot. It sounds like we need to have Citrix guest tools installed? I've done that but Secure Boot still fails on Windows VMs. Or are the "Citrix VP drivers" a different thing that I need to look into?
-
We use iSCSI with SANs for storage. This doesn't support this provision in XCP-NG, which we absolutely need. Our SANs do support thin provision. In my limited testing the SAN thin provision seems to work, but I'm wondering if anybody has used this long term. If we start running out of storage in 1 year because relying on Dell SAN thin provision isn't good, that's not going to be pretty.
-
RAM access performance of VMs seems to be much lower on XCP-NG compared to ESXi and Hyper-V (over 10x slower when testing via sysbench. Windows doesn't show this speed decrease). Overall, XCP-NG performance seems mostly on par with ESXi (save a couple test cases), so I'm not really worried about that, but wondering if anybody knows why RAM access is so much slower.
-
Importing VMs from ESXi via Xen Orchestra works, but it is very slow and the live import is broken. StarWind has a converter that works with some other hypervisors, and it allows you to import a VM while it's running through snapshots. This feature seems to be broken in Xen Orchestra, and would be very nice to have.
-
-
Welcome to the forum!
We are running both XCP-NG and VMWare and there are a ton of things that we've been annoyed with in XCP-NG, all tho it does work and it does the job if you have basic needs.
As long as you're not doing anything "special" it will work and it will be reliable and stable.Somethings we have learned and are living with:
- Disk I/O is bad compared to VMWare, but it works and its not "too bad".
- Live storage migration is a bit slow, but since you're using shared storage this isnt an issue.
- Always use NFS if you can, we're also using Dell storage (Powerstore series) and we're using NFS and it is working really well. Dedup ratio is at 6.7:1 and performance is pretty good.
- There are issue with Secure boot, so we're not using it.
- VM's may "lagg" if they have a lot of RAM when you migrate between hosts, in our case we're running a lot of WSFC SQL Server failover-clusters and it might cause a failover. To resolve that we've been tweaing the registry in our Windows installations running WSFC.
- Backup performance compared to veeam is very slow, in VMware we're doing 9Gbit/s and in XCP-NG it's about 50-80MB/s.
-
@nikade
For the NFS thing, how's your performance compared to just iSCSI or whatever else you have underneath NFS. We thought it sounded a bit overcomplicated to put NFS servers in front of storage for that many VMs and that critical an infrastructure piece, and were concerned about performance. -
@crazyadm1n Its pretty good, im not able to max it out with XCP on either iSCSI or NFS so it doesnt really matter.
The Powerstore supports native NFS server, so it is really easy to setup a new filesystem with a NFS server so it is actually easier to setup than iSCSI with multipathing. -
I struggled a bit with network setup. Its super easy to setup a BOND but you have to remove it again to join hosts into a pool. This was a bit challenging for us as the juniper switches in our co-location dont support LACP fallback and we dont have direct switch access. There was also an issue where LACP was working in 8.2 but not in 8.3. It was fixed with a firmware.
We come from an setup where we had multiple 2-node vsan cluster in 3 sites with vReplication for DR. Running more windows VMs than anything else. XCP-NG with XOSTOR and build in backup was an 1:1 replacement for us. We are very happy how everything turned out. vmware migration was working fine as well. Support is great too and much more responsive compared to what we had at vmware.
-
Yeah some setups are easy to migrate, some are not. But I agree with the support, Vates is top class and it seems like they really understand that response time and quality is key.
VMware support is slow and sometimes I wonder if they even know english or if they are using some google translate... -
@crazyadm1n Over the past few months, we transitioned from VMware to XCP. It was a significant project with its challenges, but overall, I'm very satisfied with the move. We didn't utilize many complex VMware features, mainly using iSCSI storage. Within XCP, we tested both NFS and iSCSI, ultimately experiencing much better performance with iSCSI. There are still some improvements to be made in backup, but Vates is working hard on this. I agree with others that Vates' support makes a big difference—they respond quickly and provide the necessary partner support. So far, we are very pleased with the transition. If you have specific questions or issues regarding Windows VMs, let me know; we have gained considerable experience recently.
-
I moved four sites from Esxi to XCP-NG (there are about twenty ish virtual machines).
Once I figured out how to make XCP-NG work, it was relatively easy. I began by installing it on an unused old Dell.
My comments are from that standpoint of a general contractor (construction) that also does IT work so take some of my terminology with a grain (boulder) of salt.
Things that gave me some pause:
-
Figuring out how XOA works vs XO was somewhat confusing. I ended up watching two of Tom's videos at Lawrence Systems (shown below) to get me started.
https://lawrence.technology/xcp-ng-and-xen-orchestra-tutorials/
https://lawrence.technology/virtualization/getting-started-tutorial-building-an-open-source-xcp-ng-8-xen-orchestra-virtualization-lab/ -
NIC fallover - this was much easier in Esxi. It took me a night to figure out how to do the bonding thing.
-
The whole "NIC2:" has to be the same "NIC2:" on every machine was a pain in the a##. Again the way esxi does it is easier.
-
Figuring our the proper terminology to properly create a local repository
Find the disk ID of the “sdb” or “cciss/c0d1”disk
ll /dev/disk/by-id
use gdisk to create partions
"xe sr-create host-uuid=c691140b-966e-43b1-8022-1d1e05081b5b content-type=user name-label="Local EXT4 SR-SSD1" shared=false device-config:device=/dev/disk/by-id/scsi-364cd98f07c7ef8002d2c3c86296c4242-part1 type=ext"-
Expanding an existing drive (i.e. after you grow a raid drive) was a tough (I have post on this site that shows how I did it).
-
Moving a VM from esxi to XCP-NG was just long and a few vomited in the process and had to be re-done. In some cases I used the built in XCP-NG migration and, in others (the huge VMs) I figured out how to do it via Clonezilla (much, much faster once I got the hang of it).
-
list item Having to shut down a running VM to increase the disk size is a bit of a PITA but its not that big of a deal.
-
Over committing memory...I still don't have a great grasp on the one.
Before I made the move, I did a ton of speed tests of esxi vs XCP-NG. About 60% were slightly faster on Esxi and 40% were faster on XCP-NG. In the end, the differences were negligible.
With all that said, I think XCP-NG is much easier to use than esxi and I like it better. Vcenter seemed to last about six months and then always died and had to be rebuilt (and the restore utility was about as reliable as gas station sushi). With XOA, it always works and is much faster than Vcenter.
The backup is awesome. With esxi I was using Nakivo.
Just my two cents!
-
-
@archw The network issues might be a deal breaker for us. It's comical how there's no automation or even well written instructions on how to "match" network interfaces. To say that, "in order to have a pool, all the host networks must be identical" is just insane. We don't have identical hosts or network cards throughout our infrastructure, because we upgrade it throughout the years. I've been trying in a test environment to re-order and match up network interfaces, but it's proving impossible. I don't see how anyone could use this product outside of a home lab scenario, which also makes me believe that's where it's used most.
-
@fakenuze we had the same challenge but managed to take care of this by changing the nic names.
Check out this article
https://support.citrix.com/s/article/CTX135809-how-to-change-order-of-nics-in-xenserver?language=en_US -
XCP-NG support/licensing ended up being too expensive for us to justify it. We would have been paying about the same amount as we were paying for VMWare before the price hike.
To get the multiyear pricing you have to pay for all those years up front. VMWare did not have a same restriction pre price hike.
-
In the FWIW department, I've now done four sites. Network thing I mentioned above took about an hour to figure out and was a bit confusing. Once I did one (and made a little cheat sheet on how I did it), the rest of the sites were easy.
Arch
-
Our three main problems are:
snapshots snapshots snapshots
just a pain on a thick lvm iscsi sr.
-
@rfx77 said in Switching to XCP-NG, want to hear your problems:
Our three main problems are:
snapshots snapshots snapshots
just a pain on a thick lvm iscsi sr.
We switched from iSCSI to NFS and never looked back, the performance is pretty good, thin provisioning, snapshots coalescale pretty fast and life is good.
The setup is rather easy as well as there is no need to tweak the XCP's multipathing configuration. -
@rfx77 said in Switching to XCP-NG, want to hear your problems:
Our three main problems are:
snapshots snapshots snapshots
just a pain on a thick lvm iscsi sr.
Thin based SRs are recommended for a reason as they give better performance and also take up less storage space. You can get really good performance if using NFS 4.0 or higher (best when done using NFS 4.2 and using pNFS to its fullest extent).
With full flash storage target using NFS 4.2 and pNFS to its fullest extent and you'll have the capacity to benefit from features which have been developed, specifically to be paired with SSD storage.
-
Also CBT might help to reduce the coalesce work needed in general.
-
@john-c
NFS is no solution compared tu fast iSCSI Storage. To get in the performance-range of out ISCSI Flashsystem you have to buy Netapp which costs you three times as much.If you have to pay this much for a storage to get the same faetures as before you can stay on vmware.
-
@rfx77 said in Switching to XCP-NG, want to hear your problems:
@john-c
NFS is no solution compared tu fast iSCSI Storage. To get in the performance-range of out ISCSI Flashsystem you have to buy Netapp which costs you three times as much.If you have to pay this much for a storage to get the same faetures as before you can stay on vmware.
Does your current storage support NFS?
If yes, you should give it a try and atleast benchmark it, maybe you'll be suprised. -
@rfx77 use cbt backups and no coalesce and snapshot issues anymore.
-
@nikade No our current storage does not support nfs. but which enterprise grade-storage beside top end Netapp and Dell PowerStor really do??
