slow vm with high steal
-
Hello, I'm a bit lost because I'm new to xcpng and virtual machines in general. I installed xcpng onto a supermicro xeon-d 1521 a 4 core 8 thread cpu with 128 gb of ram and 2 tb nvme ssd for the vm. There is only one vm currently on the machine.
The issue is the vm is slow. When I create a vm it takes over an hour installing ubuntu server. Then then I start the vm it takes over 10 min to get to the login prompt.
From what I found googling I found I should look at mpstat -P ALL. iowait is at around 0.22 so I'm assuming that the ssd is fine. But I dont know why the steal is can be as high as 75.
There is the only one vm on the machine. The host machine steal is at 1.7.
I'm at a lost as why the vm is so slow and what more to do to diagnose what might be causing the slowdown.
Thanks for any help.
John
-
@aHguf5QP Did you try running top and xentop from the host command line to see about resource utilization (CPU and memory), as well is iostat to look at the storage I/O resource utilization? It may take more memory and/or CPU cores for dom0.
Some of slowness can be from lack of resources, some can be inherent in the storage (RAID configuration, connectivity bottlenecks, buffer and other settings, etc.). If this is a single SSD drive then you're at the mercy of what the controller and drive itself can manage on the storage end, but there may be system settings that could help.
Just 4 cores and 8 threads is not much for running a server, plus you didn't mention what the clock speed is, which also factors in quite a bit in performance. -
@tjkreidl said in slow vm with high steal:
is iost
Linux 4.19.0+1 (xcp-ng-iptyubfo) 07/20/2023 x86_64 (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
5.85 0.00 4.83 0.01 1.32 87.99Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
nvme0n1 1.74 53.49 9.16 819462 140304
sdb 3.50 6.69 19.69 102444 301689
sda 3.79 13.69 19.69 209780 301689
md127 4.62 20.34 18.92 311612 289808
dm-0 1.76 53.45 9.23 818878 141344
tda 0.19 8.10 0.00 124140 4xentop
dom-0 cpu usage is ranging around 150% to 20%. memory usage is 5.8%.the only vm there cpu usage is 1% and memory is 3%.
as for the ssd speeds.
I ran dd if=/dev/zero of=~/test bs=512 count=10000the dd command ran at 15.8 mb/s so I'm assuming its not the ssd i'm having trouble with.
That said, I took a look at the tasks and found a lot of this error:
Xapi#getResource /rrd_updates
I googled it and found a post about host being behind a nat and not on the same subnet. Xen orchestra and xcpng are on the same subnet. So I'm lost as to what is going on here.
Here is the post I was looking at: https://xcp-ng.org/forum/topic/4926/thousands-of-tasks-xapi-getresource-rrd_updates-on-xcp-ng-01-0/7
thanks.
-
additionally the cpu is suppose to run with a base frequency of 2.4 ghz. So I'm assuming that this cpu is fast enough.
-
@aHguf5QP 2.4 GHz is pretty slow by current standards, but should be OK for a single VM. How many VCPUs are dedicated to dom0?
As to I/O, dd is not a very accurate testing mechanism, but may not be the reason for slowness.
With the I/O, iostat should show if there are bottlenecks.
Storage should always be on a separate subnet as a VM or any other networks. -
The storage is local on the machine. Its a m.2 2tb nvme ssd. So there is no network involved when the vm is writting to disk.
Dom0 has 8 vcpus. So it has everything. and the vm has 2 vcpus.
This is a home server and so it does not need to be powerful. I'm moving up from a pi so I'm assuming that this xeon-d is better.
That said, I dont know what to do now. Is there a checklist to go through to diagnose xcpng servers?
-
@aHguf5QP Sharing the VCPUs with the VM may be art of it, esp. if dom0 sometimes runs 150%. You may be just up against the limitations of the system.
XenTools (guest tools) is installed OK, right? Diagnosing slow performance can be a long and tedious process. You can google for various suggestions, but each system is unique.
The system disk is the same as used for the VM disk? That could be also somewhat of a limitation since they haave to share the I/O from the same physical device.. -
I just installed the xen guest tools. I'm rebooting the vm. Its taking a while.
The vm has their own nvme ssd and the xcpng is on a sata ssd. So there should be no contention.
That said. I'm adding another vm with ubuntu server just to see how long it takes.
-
@aHguf5QP Xen guest tools can make quite a difference in performance. See if that helps.
Good to hear dom0 is on a totally separate device than the one that contains any VMs. -
I rebooted the vm twice and both time it took 9 min. memory usage went up to 4gb during the boot process and returned to under 500 mb once the vm started. So the guess tools did not help.
I'm still waiting for the second vm to install ubuntu.
At this point I'm looking to reinstall xcpng.
I dont know what is going on here.
-
It sounds really slow to install on this hardware (even if it's not crazy fast, it shouldn't be that long). I would take a look at usual logs (like
dmesg) to see if there's a problem on the hardware. I would also check to have latest BIOS and firmware installed on that machine.
