@CodeMercenary I am using V4 on the XO-Server to our backup remotes and it seems to work just fine. However using V4 as a storage SR was nothing but problems, as @nikade mentioned we had tons of NFS Server not responding issues which would lock up hosts and VMs causing downtime. Since moving to v3 that hasn't happened.

Checking a host's NFS retransmissions stats after 9 days of uptime i see we have had some retransmissions but they have not caused any downtime or even any timeout messages to appear in dmesg on the host.

[xcpng-prd-02 ~]# nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
268513028   169        268537542

From what a gather from this blog post from redhat (https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat) it seems like that amount of retransmissions is VERY low and not an issue.

Thats fine, we've got a lot more and I haven't seen any "nfs server not responding" in dmesg yet.
Using NFS v3 for both SR and backups now for a couple of years and it's been great, I think I had issues once or twice in like 5-6 years on the backup SR where the vhd file got locked by dom0, Vates helped out there as always and it was resolved quickly.

Checking a host's NFS retransmissions stats after 9 days of uptime i see we have had some retransmissions but they have not caused any downtime or even any timeout messages to appear in dmesg on the host.

[xcpng-prd-02 ~]# nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
268513028   169        268537542

From what a gather from this blog post from redhat (https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat) it seems like that amount of retransmissions is VERY low and not an issue.

Maybe I should give NFS another chance but force it to use v3.

In my case, I'd get backups working on NFS and then several days later a backup would fail. Then backups fail every day until I intervene, usually by rebooting the XO VM. Sometimes I'd then have to do cleanup, like releasing a VDI or something. Then it may or may not start working again but if it did start working I'd have another failure a few days later. It's been 2.5 weeks since I switched it to SMB and have had no failures. That's definitely the longest I've gone without a failure from a delta backup to a networked drive.

Note, I also have backups going to a local drive mounted in XO so with all those remote failures I always had a clean backup somewhere. This was in the process of trying to decide if I could trust sending delta backups to a network remote rather than using full backups to a local remote. My initial feelings were that the delta backups didn't work reliably but now I believe the issue was with NFS, not with deltas specifically.

@rfx77 As a follow up v4 did cause more downtime for us, i switched over everything to v3 which has been much better so far. Going to be curious to see how a controller failover goes during a firmware update as v3 is stateless vs v4 and iscsi being stateful protocols.

We're also using v3 and failover on our Dell Powerstore's are seamless.
Haven't tried v4 since we had A LOT of "nfs server not responding" issues with it and immediately went back to v3.

Our Issue with commvault was that despite it mounts the snapshots in the proxy VM it has to read them as a whole. So Inc Backups take nearly as long as Full backups. When you want to backup 20TB+ Incs every night, Thats not acceptable. So the Lack of CBT is really a big deal when doing VM Backups.

Performance with CommVault would be not that big of a problem since you can do multiple streams and multiple VMs simultanously but we could not get past 600MB/s which is a Xen Problem as of our Testing. With HyperV we see Performance in access of 1.6GB/s in the same Scenarios. But also this performance would be much to slow to do Incs every night.

We tested XO CBT but it was not stable and for what i read in the posts there seems to be problems with CBT and Live-Migration so that the CBT state seems to be lost. Also a big Problem for us.

When we used CommVault to backup multiple VMs we ran into Blue-Screens of the Xen Toolstack in the Windows VMs when attaching or detaching Snapshot-VDIs to the Proxy VMs. There clearly is a bug in the Xen Windows Drivers. So we had to reduce concurrency which reduced backup speed.

We didn't do a short test with xen, we migrated out internal production Cluster to it from VMWare and used it for about 4 month now (30+ VMs on iSCSI Storage with 3 Nodes) and after we ran into more and more Problems we had many discussions with our team and we had to ask ourselfes what Xen brings to the table that is worth the drawbacks. The only scenario where it fits for us is where we have to map physical hardware into VMs.

We decided to come back in some time to see how the SMAPIv3 drivers work out and if there is a better support for shared storage.

Im hoping to eventually get to the same point with XCP-NG, be it with XOA backups or with something like commvault. Our NFS mounts last dropped Thursday night during backups and have been fine since so its VERY intermittent. I think if it happens again i will begin moving to NFS3 backed SRs and hope that solves it.

What issues with commvault did you run into due to its lack of native Xen CBT? Was their own internal change block tracking not reliable? I have a feeling the xen portion of the product does not see a lot of development attention from reading their documentation.

@rfx77 Makes sense. I was thinking of giving the Commvault trial a try with XCP-NG since it looked like while they don't use Xens' CBT, they still tracked changed blocks within their helper VM while still doing dedupe.

Backups have been the biggest set back in our move from VMware, i knew going in i would miss Veeam more than i'd miss Vmware itself.

As for NFS 4.1 vs 3, if the timeouts return this week i think i will give v3 a try if it worked more reliability for you.

Yeah, Veeam really is the king of backups. We're backing up about 50 VM's with Veeam from our vmware clusters and man it is sooo fast and reliable, i've seen backups go at 7Gbit/s which is incredible.

@nikade I am not using any custom mount options other than "hard" to do a hard nfs mount to prevent data lose when drops like these happen.

Did you have to use custom mount options with V3 as well then or just with V4? I may try moving VMs over to a V3 mount from V4 to see if that helps stabilize things.

In XOA we got some mount options from @yannik but in XCP we have not had to use any special options when mounting the NFS SR (as long as we're using NFS 3).

Backups have been the biggest set back in our move from VMware, i knew going in i would miss Veeam more than i'd miss Vmware itself.

As for NFS 4.1 vs 3, if the timeouts return this week i think i will give v3 a try if it worked more reliability for you.

The Lack of CBT Support in CommVault was a major problem. We cannot use XO for backups because it lacks 90% of the features we now have with CommVault (Dedup, Tape, Agents, IntelliSnap of VM disks, AUX Copys,..) and we dont want to combine it with CommVault agents since this complicates our Backups and dies not make real sense.

HyperV with Clustering is free in our setup so it was not a big discussion there. We also can utilize the full potential (performance and feature wise) of our SAN and Network which always was a major issue with Xen.

We keep XCP-NG for now on some specialized installs where we need to map physical hardware into a vm.

Did you have to use custom mount options with V3 as well then or just with V4? I may try moving VMs over to a V3 mount from V4 to see if that helps stabilize things.

@nikade Interesting to hear we are not the only ones experiencing timeouts with V4 We are on NFS 4.1 with the Pure array on the latest firmware. I initially wanted to use v4 as it is a stateful protocol and thought it may handle controller failovers better due to that. Perhaps i should try V3 and see if it fairs better. We had okay luck with V4 on TrueNAS but never really ran it under any extreme load. It will run for 2 or 3 days without issue then suddenly NFS drops appear in dmesg on the hosts.

For example:

dmesg -T from the host shows the following with 10.174.199.25 being the array. 

[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying
[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying
[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying
[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying

Followed by recovery after some time:

[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK
[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK
[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK
[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK

Is that what you were seeing on your powerstore with V4 as well?

We actually had some timeouts like these on our "slower" TrueNAS, we got help from @yannik I think to tweak the mount options when our backups failed.
Ever since they tweaked the mount options it has been rock solid and we havent seen any timeouts in dmesg.

I've tried to tweak the mount options, but I didn't have much luck, thats the reason why I asked if you had played around with it.

For example:

dmesg -T from the host shows the following with 10.174.199.25 being the array. 

[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying
[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying
[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying
[Thu Aug 15 01:28:30 2024] nfs: server 10.174.199.25 not responding, still trying

Followed by recovery after some time:

[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK
[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK
[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK
[Thu Aug 15 01:29:49 2024] nfs: server 10.174.199.25 OK

Is that what you were seeing on your powerstore with V4 as well?

Have you tried to experiment with the mount options?
Just curious, since we didnt have any luck with NFS 4 on both TrueNAS and our Dell Powerstore, the latter is NVME allflash and the TrueNAS boxes are 10K SAS with SSD as cache.

Also i would recommend you to look into alike a3 backup, we use it with high io vms with 2tb disk size, no issues there and u need only one snapshot per vm.

With that said we are also having issues with Pure's NFS implementation and how it interacts with XCP-NG, causing storage timeouts for us. According to them the array, when under load is disconnecting hosts due to "expired NFS leases" we are currently working with them to stabilize that, perhaps then i can revisit full CBT backups. Weirdly enough the disconnects do not appear to happen under regular VM operations even under high load, we mostly run into these during backup runs with CBT enabled runs increasing the chance of it happening.

What we ideally will end up with is local backups, offsite backups and archival backups. This is what we had with Veeam prior. Doing this with traditional XOA backups would result in 3 snapshots per VM (1 for each job) which lead me to looking into commvault since it does things a bit differently. CBT would also solve this as well i assume.

Trying CBT has been a mixed bag with snapshot coalesces taking anywhere from seconds up to 2-3 hours on larger VMs (with looping timeouts in SMlog) and backup failures if a VM has moved between hosts in the same pool.,and in the worst cases causing our NFS based storage (Pure storage) to expire all NFS leases to the pool master trying to coalesce causing NFS timeouts and storage outages. We have Pure looking into why this is happening for us now but until that's figured out we are back to CBT-less backups that leave a snapshots behind.

From what i have read Commvault uses their proxy VM to track changed blocks without leaving the snapshot in place? Sounds like it may be worth checking out.

we also tried it on our main pool with iscsi storage and had mostly the same problems like others above.

we decided to wait a litte longer before we give it another try on our main production system mainly due to some hanging snapshots,...

in the future we are planning to use it as main vm-level backup solution for standalone hosts in combination with commvault agents in the vm.

our current test status i that we try to use XO as a docker container which would fit better in out existing backup infrastructure as we could easily integrate it in our linux backup appliances (see it like a linux NAS). The docker tests are running very well and we dont have any issues at the moment. even file-level-recovery is working well in this scenario.