Good evening all,
Just a quick update. It's been a couple of days now after the rebuild and everything seems to be humming along fine, so I believe this topic can be marked as resolved. I can confidently conclude that this wasn't an XCP-ng issue, although the error message seems a bit misleading.
@olivierlambert Yes sir, it is and I'm glad I confirmed this for myself. Thanks also for helping me understand how the VM Suspend process works. Hopefully this post helps other newbies with the same understanding in the future.
@olivierlambert I didn't time it. I'm leaving home to drop the kids off at school, then off to work afterwards. I'll run it again and time it and report back.
@olivierlambert Oh nice, hope it's not too cold over there. It's 9AM over here in Maryland and a bit nippy but not too cold.
Nope, no host or pool logs available (both are empty). Then again, I cleared all the alerts from the dashboard earlier this morning, so not sure if that's what wiped all the logs. I'm not too worried, as this is a testing environment so I'm using this host to do a lot of learning.
@johnny I think the Vates VMS (whether you go with paid support or not) would fit your use case perfectly. Since each host is essentially a pool of one, you would pretty much have multiple pools managed by a single Xen Orchestra instance (giving you a single pane of glass, if you will). You could augment this with XCP-ng Center.
As @olivierlambert correctly inquired, you should contact the Vates Team and have an in-depth discussion into your situation.
@olivierlambert Yes sir, it is and I'm glad I confirmed this for myself. Thanks also for helping me understand how the VM Suspend process works. Hopefully this post helps other newbies with the same understanding in the future.
Here's the latest, and probably last, test. Disabling compression had no appreciable impact on performance. I am now fully convinced that SYNC is the major player here.
@olivierlambert Oh okay, thanks for responding.
So I turned off SYNC and COMPRESSION on the dataset and retested (by suspending 11 VMs), I immediately noticed a whopping performance improvement (essentially sustained wire speeds AND 50% faster completion time*) :
Gonna retest with SYNC disabled and COMPRESSION re-enabled and see if it degrades performance; standby for another report.
@olivierlambert Yes, I do plan on testing with SYNC disabled and then again with several permutations of dataset changes on the TrueNAS side (like compression on/off, etc.).
Do you guys have a best practices document for setting up an NFS SR using TrueNAS? I browsed through the published XCP-ng documentation site but didn't find anything specific to TrueNAS or maybe I missed it.
Sharing this so others might benefit from what I'm learning.
So, I looked at the network performance on TrueNAS during the Smart Reboot of the second XCP-ng host (screenshot below). What I saw seems to suggest that I'm getting near wire speed during READ operations. However, WRITE operations seem to be hitting a ceiling and I have a feeling it might be due to me having SYNC enabled on the dataset.
@olivierlambert Aaah, I've always wondered what the Suspend SR on the Advanced tab of the Pool meant......now I know.
So, the NIC capacity/pipe between the hosts and the SR does really matter here.
Good-day Folks,
I need help understanding the VM Suspend process: why does it take so long for XO to suspend a VM, triggered by a Smart Reboot of a Host?
My Environment:
HOSTs: XCP-ng 2-node pool at v8.3.0 on HP (ProLiant DL360p Gen8)
XO: Community Edition at commit c5ba7
NETWORKING: 1Gbps Management only
STORAGE: Shared NFS Storage Repository (hosted on a separate TrueNAS Server)
Today, while applying the latest host patches, I wanted to try doing a Smart Reboot. After pressing the button and acknowledging the prompt that VMs will be suspended and then un-suspended after the reboot, and quickly navigated to the Tasks page to monitor the progress. I immediately saw one VM quickly show progress from 0%, 10%, 30%.......100% and boom, it was suspended. The others, however, not so much. As you can see from the two screenshots below, they are still pending suspension and the Estimated End keeps shifting to the right.
I don't have a 10Gb Storage Network between the hosts yet (working on it). However, I didn't think that should have such an impact. Anyway, I don't think I have a proper understanding of how the suspend operation should be working, so if anyone cares to educate me, I would really appreciate it. Thank you.
Screenshot Taking After Pressing Smart Reboot on the Master Host
Screenshot Taking After Pressing Smart Reboot on the Master Host (10-minutes later)
@olivierlambert Yep.
Here's the video of the test - https://www.youtube.com/watch?v=tiVdR74PNjw
@olivierlambert Just ran the test on both XOCE and XOA, and looks like it takes about 8 minutes to get to 8.6GB downloaded before the transfer rate drops to zero. I also observed the transfer rate drop to zero at 8.3GB for a short while before picking back up and stalling at the 8 minute mark.
I made a video of the test - it's uploading to my YouTube channel now, when it finishes I'll update this post with the link.