There is still no guarantee that even modified drivers will work, however, due to the way NVIDIA drivers function. There are a lot of people, who have spent a lot of time, to get it to work - with some success; but, everything I've read points to this being something you'll have to screw around with and "maintain" regularly to keep it working. If you're expecting it to be a smooth, easy, and "permanent" process, I would recommend not even starting to try it on Xen (Server or XCP-NG).

If you REALLY need to do this with VMs, KVM and unRAID both have built-in capabilities to do this - which seem to work a bit better than others. vmWare is supposed to have a similar feature; but, I've not gotten it to work. I will tell you, that in messing around with this on all the hypervisors, even on KVM and unRAID, some card\driver combinations just will not work, and, you'll honestly waste more time screwing around with it than just spinning up a dedicated box or grabbing an AMD card. I'm not an AMD guy, and I didn't choose that route myself; but, they at least do not have this "limitation" in their driver stack.

Don't think NVIDIA will come around and fix it, either - their forums are rife with this topic; and, they've pretty much said "we don't have any plans to fix this bug [that we actually introduced on purpose]".

@fohdeesha I'm going to be honest with you. I've had terrible luck with FreeNAS more times than I can count. It's always been a mix of hardware compatibility issues, appearance of reliability, a not-so-great interface, and a slew of others. I've found unRAID to suffer from similar things. I know a lot of people who use it; and, online, it's obviously widely used by a very large audience - it just doesn't seem to work well for me. I just recently spent at least a couple days trying to use it in front of a fiber-channel array to no avail.

I am about to drain and rebuild an x86-based storage appliance (currently running XPenology) in the next few weeks, so, I may go ahead and try it on this piece of hardware and see where it gets me. I loved OpenFiler back in the day, and, had pretty good luck with Nexenta for a while; but, one is just asking to be compromised using it in 2020, and the other doesn't scale well with modern storage without spending a fortune. XPenology works great, and gets you all the ease of Synology; but, it is very finicky about hardware. I ultimately had to settle with a solution that I really don't like; but, it works for now (Windows Server on vmWare, with the FC array passed through as a RAW disk).

@beagle The simple answer here, which has been implied by others, is...

You DO NOT, EVER, install other "services" on your hypervisor in a production environment - no matter what your cost, convenience, etc. "desires" are.

That's just not how this is all supposed to work; and, any technical guy worth his job position will know right away to not only recommend against setting it up like that, but, will adamantly decline being "forced" to do so - consequences be damned.

What is even worse, is you mentioned a 20 TB requirement for file server storage. There is literally only one solution for this...

A DEDICATED file server - whether that be a VM with enough attached storage (VDIs using workarounds, iSCSI\NFS\SMB direct attached), or you build a back-end NAS system that either passes off the data directly, or utilizes a front-end file server "proxy".

You also claim a need for high-performance - this automatically dismisses any "cheap" solution that aligns with some of your concepts above. Do it right...the first time.

There is a huge difference between "finding a creative solution" and "ignorantly (intentionally or not) pursuing an unrealistic and ill-advised idea".

@Otis2772 Where is your time synchronization coming from? Time differences can be caused by a slew of things including power settings (VM or not), CPU availability, networking connectivity, etc. Are they all the same OS or different OSes?

On a 1.78 TB storage repository, you don't have nearly enough disk space to be snapshotting a 832 GB disk. You will need to increase the size of your storage repository sufficiently to avoid such issues. While a snapshot should only take twice the amount of space as the disk you are snapshotting, overhead and other activity on the storage repository means you should probably aim for having at least twice the size of the disk you are snapshotting actually FREE on the storage repository. E.g. to snapshot a 1 TB disk, you should have 2 TB of free space on the storage repository.

Note: I am aware of all the nuances of snapshot trees, etc. - it's just not worth getting into here when the simple recommendation above prevents issues such as these.

This often occurs when you're having storage issues (whether they are readily apparent or not), that may or may not be related to networking intermittency, that fills up the log files. We can't say what it is without seeing logs; but, just FYI that that is the reason I have most often seen this occur. Check your /var/log directory.

@slavD I'm in agreement with you. I wanted to replace my multiple gaming rigs with multiple VMs with GPUs (since I already run the server); but, I spent at least a month messing around with various options and never came up with a stable, reliable, and "not overly complicated" way to do so without drastically changing my approach to hardware - the easiest being going with AMD GPUs and increasing my power consumption over 9000%. It's why I've ultimately paused that project for now until some better solution comes along, or until I can "empty" my existing host and repurpose it with a solution that will work (probably unRAID despite my concerns with other parts of it).

There is still no guarantee that even modified drivers will work, however, due to the way NVIDIA drivers function. There are a lot of people, who have spent a lot of time, to get it to work - with some success; but, everything I've read points to this being something you'll have to screw around with and "maintain" regularly to keep it working. If you're expecting it to be a smooth, easy, and "permanent" process, I would recommend not even starting to try it on Xen (Server or XCP-NG).

If you REALLY need to do this with VMs, KVM and unRAID both have built-in capabilities to do this - which seem to work a bit better than others. vmWare is supposed to have a similar feature; but, I've not gotten it to work. I will tell you, that in messing around with this on all the hypervisors, even on KVM and unRAID, some card\driver combinations just will not work, and, you'll honestly waste more time screwing around with it than just spinning up a dedicated box or grabbing an AMD card. I'm not an AMD guy, and I didn't choose that route myself; but, they at least do not have this "limitation" in their driver stack.

Don't think NVIDIA will come around and fix it, either - their forums are rife with this topic; and, they've pretty much said "we don't have any plans to fix this bug [that we actually introduced on purpose]".

@imad2nsi This is starting to remind me of my 4-week foray into messing around with unRAID, coming out of it with the conclusion that it was more trouble than it was worth most of the time (with the various types of hardware I tried it on).

@fibrewire If you're looking at a free way to do so, you can install XOA from the sources and do it through that. I honestly don't miss the updates portion of XCP-ng Center (even after having used it for years in XenCenter). It was very clunky and time-consuming; and, I ultimately ended up scripting a lot of it locally on hosts instead. This is obviously much easier now that it can be done through a simple "yum" command instead of having to pull all the separate packages and install them like XenServer does.

XOA will obviously provide that feature through the paid versions as well (if you've already got it). I'm not big on pushing people to XOA from XCP-ng Center, as I still use XCP-ng Center way more than XOA (at least until they push out the new version with the tree view so I can give it a try); but, that's the easy answer to your question.

Yes. That would be a way of doing so. If you can afford the downtime, you could also use a backup solution (XOA, Alike DR, etc.) to backup the VMs from both hosts, build the pool from scratch using both hosts in a clean state, and then restore back into the new pool (this way gives you a chance to not only start fresh; but, implement any big "lessons learned" or "new ideas" you've got on your list).

That is correct. My recommendation would be to migrate the VMs from one host to the other, then wipe that host and bring it into the pool (or just bring it into the pool). I'm not sure it wipes the actual data of those VMs off the local storage repository; but, it will certainly clear out the metadata of all the VMs on that host. You can't recover that metadata into a pool that I'm aware of.

@Appollonius @borzel They've been saying that it'll be relevant for 20 years, though.