<![CDATA[VM's going really slow after 3 - 4 weeks]]>Evening All,

Would be grateful if anyone can offer guidance on how to fault find this issue. Or solve it :). The google Fu I have used thus far hasn't really help.

All appear to be working great. However over a period of about 3 - 4 weeks the VM's slow down. For example the Windows 10 VM will become noticeably slower.

Rebooting the physical server instantly brings the VM's back to their zippy self.

This is a pretty much an out of the box installation
I haven't found anything obviously wrong.
The only thing I have seen is that some of the CPU's are getting very busy when the issue presents. Whereas when "all" appears OK all CPU's are pretty much around 12% usage.

I have just applied the following pool patch's

  • Linux firmware 20190314
  • microcode_cti 2.1
  • xen-dom0-libs 4.13.4
  • xen-dom0-tools 4.13.4
  • xen-hypervisor 4.13.4
  • xen-libs 4.13.4
  • xen-tools 4.13.4

Below is the spec of the physical and XCP setup

**Physical Server**
CPU model	Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
GPUs	MGA G200EH
Core (socket)	32 (2)
Hyper-threading (SMT)	Enabled
Manufacturer info	HP (ProLiant DL380p Gen8)
BIOS info	HP (P70)

**Network**
HP Ethernet 1Gb 4-port 366FLR Adapter

**Storage**
Smart Array P420i Controller
Raid 5
4 x ST9900805SS

**XCP-ng**
Build Date 2022-02-11
Version: 8.2
DBV: 0.0.1

17.4 GB RAM available (48.0 GB total)

Local storage 	Ext	No	36% (893.9 GB used)	2.4 TB	2.9 TB

**Networking**
Dedicate NIC 0 for management
LACP for all VM's using NIC 2 & 3

**VM's**
CheckMK - Ubuntu Bionic Beaver 18.04 (1): using 3.0 GB
graylog: using 4.0 GB
Windows Server 2012 R2 (64-bit) (1): using 3.0 GB
Windows 10 (64-bit) : using 5.0 GB
Exchange: using 4.0 GB
DC: using 4.0 GB
XO Ubuntu Focal Fossa 20.04: using 3.0 GB

Vendor: GenuineIntel
Model: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
Speed: 2892 MHz

Many thanks

]]>https://xcp-ng.org/forum/topic/6318/vm-s-going-really-slow-after-3-4-weeksRSS for NodeThu, 14 May 2026 16:59:32 GMTTue, 13 Sep 2022 19:11:35 GMT60<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Sat, 04 Mar 2023 17:12:50 GMT]]>@olivierlambert said in VM's going really slow after 3 - 4 weeks:

Oh wow. So something was wrong with the hardware, specifically the memory, and potentially more errors and error correction trying to keep up, slowing down everything. Is it a good recap?

To be honest I am not entirely sure.
I tested the memory my self in the early stages of this issue with memtest. In the past memtest has flagged memory issue that HP built in diags hasn't but it ran clean. 😞

The server was rebooted prior to replacing the memory out of curiosity and everything came up OK no errors.

So we assume that something was wrong with the memory. And now we have are fingers crossed that that is the end if it 🙂

]]>https://xcp-ng.org/forum/post/59561https://xcp-ng.org/forum/post/59561Sat, 04 Mar 2023 17:12:50 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Sat, 04 Mar 2023 17:03:28 GMT]]>Oh wow. So something was wrong with the hardware, specifically the memory, and potentially more errors and error correction trying to keep up, slowing down everything. Is it a good recap?

]]>https://xcp-ng.org/forum/post/59560https://xcp-ng.org/forum/post/59560Sat, 04 Mar 2023 17:03:28 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Sat, 04 Mar 2023 17:01:51 GMT]]>@Berrick
So its been a while since my last post on this topic but finally I think we have bottomed this issue out! 🙂

So the long and the short.
Nothing we tried, checked or diagnostics ran up to this point showed any issue.
Then one Monday morning there was one alert from the iLO about "Corrected Memory Error threshold exceeded"

As this server was due a memory upgrade to LV memory soon and we would need to reboot to clear the alert we brought the upgrade forward.

Since the new memory was installed, tempting fate here, it has been OK

Thanks to all that offered advice.

Kind regards

]]>https://xcp-ng.org/forum/post/59558https://xcp-ng.org/forum/post/59558Sat, 04 Mar 2023 17:01:51 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Wed, 04 Jan 2023 16:13:31 GMT]]>Thanks for all the replies.
I will be checking when the issue starts again in a few weeks and report back.

From previous xentop output I don't believe Dom0 is showing high CPU or Memory.

I also dont think there is a problem with the physical disks.

]]>https://xcp-ng.org/forum/post/56802https://xcp-ng.org/forum/post/56802Wed, 04 Jan 2023 16:13:31 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Wed, 04 Jan 2023 14:39:25 GMT]]>@EddieCh08666741 Indeed. Run top and make sure you have adequate resources allocated for dom0 for both memory and CPU. There should be no swapping to speak of and the CPU should be nowhere close to 100% in use.

]]>https://xcp-ng.org/forum/post/56799https://xcp-ng.org/forum/post/56799Wed, 04 Jan 2023 14:39:25 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 19 Dec 2022 04:27:31 GMT]]>hi Berrick, most of the issues that happen like this is due to IO i believe. I've been using XCP ng for many years and I have about 100 VMs. I love xcp and its the best out there. There are few instance that my VM starts to crawl to be unusable.

  1. One of the disk fails in the raid and cause my VMs on that particular server to crawl.
  2. Too little memory provisioned. Heavy usage.

Hope my above helps

]]>https://xcp-ng.org/forum/post/56312https://xcp-ng.org/forum/post/56312Mon, 19 Dec 2022 04:27:31 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Sun, 18 Dec 2022 13:52:01 GMT]]>@Berrick Run iostat and see if anything shows up as slow or saturated with your storage I/O.

]]>https://xcp-ng.org/forum/post/56296https://xcp-ng.org/forum/post/56296Sun, 18 Dec 2022 13:52:01 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Sat, 17 Dec 2022 11:19:00 GMT]]>can't say that time from reboot a cause of problem, but sometimes i'm feel something about it. Even ssh connection takes tooooo long, like 10 sec in local network. Same for http speed for small services without any load.

]]>https://xcp-ng.org/forum/post/56283https://xcp-ng.org/forum/post/56283Sat, 17 Dec 2022 11:19:00 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Fri, 16 Dec 2022 20:01:59 GMT]]>Adding @fohdeesha in the loop

]]>https://xcp-ng.org/forum/post/56271https://xcp-ng.org/forum/post/56271Fri, 16 Dec 2022 20:01:59 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Fri, 16 Dec 2022 19:46:23 GMT]]>Seasons greetings to all,

So the slowness of is creeping back. Still cant find any thing obvious
Latest Xen Top

4ccf84eb-8d2e-4434-bafc-cd38c43007f2-image.png

Due to business pressures we have had to reboot the physical server.
So now have to wait for the issue to re occur.

there were 5 more patch's which have been applied.

If anyone has any other suggestion of what to check, specific logs to search etc, etc they would be gratefully received

Kind regards

]]>https://xcp-ng.org/forum/post/56267https://xcp-ng.org/forum/post/56267Fri, 16 Dec 2022 19:46:23 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Wed, 09 Nov 2022 17:26:54 GMT]]>@Gheppy You have to give dom0 enough CPU power for various reaasons: 1) to be able to not be bogged down, itself, 2) to allow VMs to be able to interact with dom0 reasonably fast, 3) to provide enough resources to deal with storage and network I/O. Showing essentially no swap space in use is good!

]]>https://xcp-ng.org/forum/post/54785https://xcp-ng.org/forum/post/54785Wed, 09 Nov 2022 17:26:54 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Wed, 09 Nov 2022 16:17:45 GMT]]>@tjkreidl
I increased the RAM for Xen to 10Gb and it is much better, maximum 156% on CPU on xentop.
Thank you

]]>https://xcp-ng.org/forum/post/54780https://xcp-ng.org/forum/post/54780Wed, 09 Nov 2022 16:17:45 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Wed, 09 Nov 2022 06:49:19 GMT]]>This is a top img, swap it seems to be ok but memory is 103 Mb free of 7.4Gb.
I will increase the memory for Xen. Thank you
a4628398-2b29-4965-ae22-76500a7c151c-image.png

]]>https://xcp-ng.org/forum/post/54747https://xcp-ng.org/forum/post/54747Wed, 09 Nov 2022 06:49:19 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Tue, 08 Nov 2022 19:43:42 GMT]]>@Gheppy If you run top, how is the dom0 CPU load and how is the swap space? If dom0 is saturated it may need more memory allocated to it.

]]>https://xcp-ng.org/forum/post/54732https://xcp-ng.org/forum/post/54732Tue, 08 Nov 2022 19:43:42 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Tue, 08 Nov 2022 06:50:52 GMT]]>There is free space, on real disks I have:

  • 4 x 400Gb SSD = 800Gb RAID 10, on which I have XCP-ng and the rest of the disk is with the first disk of the VM where is the operating system, which has 350 Gb allocated in which I have 2 partitions with 150 Gb SO and 200 Gb SQL log (for some speed). The occupied space being for SO 65Gb and for SQL log of 120Gb.

  • 14 x 1200Gb = 8.4Tb RAID 10, on which I have the rest of the database divided on 3 virtual disks of 2Tb with soft RAID in the OS ( which is windows ),the configuration is like this because of the XCP-ng limit of 2Tb per vdisk. The database currently has 3.7 Tb.

]]>https://xcp-ng.org/forum/post/54711https://xcp-ng.org/forum/post/54711Tue, 08 Nov 2022 06:50:52 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 20:11:40 GMT]]>@Gheppy What is free disk space like on this VM?

This is what caused the CPU's in our graylog server to go nuts. It had almost run out of free disk space. The only thing I have done was to increase free disk space by clearing out some of the older logs.

72b922e6-3c6a-457e-984a-271e5ff95961-image.png

]]>https://xcp-ng.org/forum/post/54700https://xcp-ng.org/forum/post/54700Mon, 07 Nov 2022 20:11:40 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 17:40:55 GMT]]>My server is HP DL380 G9, CPU 2 x E5-2620 v3 @ 2.40GHz with 4xSSD and 16xHDD 2.5" and 128Gb RAM.
System ( XCP-ng and OS ) is on 4 x SSD RAID 10, DB is on 14 x HDD RAID 10.

]]>https://xcp-ng.org/forum/post/54696https://xcp-ng.org/forum/post/54696Mon, 07 Nov 2022 17:40:55 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 17:27:22 GMT]]>What's the hardware behind it by the way?

]]>https://xcp-ng.org/forum/post/54695https://xcp-ng.org/forum/post/54695Mon, 07 Nov 2022 17:27:22 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 17:20:37 GMT]]>@Danp said in VM's going really slow after 3 - 4 weeks:

@Gheppy I was surprised to see a graylog VM using so much CPU, but maybe it's normal. 🤷

Not sure if Gheppy DB is graylog. Mine was and when I searched for an answer as to why the strange CPU utilization came up with the same answer Oliver supplied.

I would like to point out, as I didn't earlier, that the graylog cpu utilization in the xentop image I up loaded has been fixed so CPU util is much much less now.

However, the CPU utilization of that vm was also at the high levels after a server reboot so dont think its the answer to why all vm's slowly slow down

]]>https://xcp-ng.org/forum/post/54693https://xcp-ng.org/forum/post/54693Mon, 07 Nov 2022 17:20:37 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 16:18:53 GMT]]>Yes, it is a VM with a heavily used database.
At the moment I am trying to convince them to buy an XO license for this server.
I work for a public service and I have no say when it comes to money.

]]>https://xcp-ng.org/forum/post/54691https://xcp-ng.org/forum/post/54691Mon, 07 Nov 2022 16:18:53 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 14:21:29 GMT]]>@Gheppy I was surprised to see a graylog VM using so much CPU, but maybe it's normal. 🤷

]]>https://xcp-ng.org/forum/post/54684https://xcp-ng.org/forum/post/54684Mon, 07 Nov 2022 14:21:29 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 13:24:24 GMT]]>So (roughly and IIRC), you are using 6,23 vCPUs at 100%.

]]>https://xcp-ng.org/forum/post/54683https://xcp-ng.org/forum/post/54683Mon, 07 Nov 2022 13:24:24 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 13:23:00 GMT]]>@olivierlambert
Server has 24 core and the VM 20

]]>https://xcp-ng.org/forum/post/54682https://xcp-ng.org/forum/post/54682Mon, 07 Nov 2022 13:23:00 GMT<![CDATA[Reply to VM's going really slow after 3 - 4 weeks on Mon, 07 Nov 2022 13:20:39 GMT]]>How many vCPUs you have assigned to this VM?

]]>https://xcp-ng.org/forum/post/54681https://xcp-ng.org/forum/post/54681Mon, 07 Nov 2022 13:20:39 GMT