@olivierlambert said in VM's going really slow after 3 - 4 weeks:

Oh wow. So something was wrong with the hardware, specifically the memory, and potentially more errors and error correction trying to keep up, slowing down everything. Is it a good recap?

To be honest I am not entirely sure.
I tested the memory my self in the early stages of this issue with memtest. In the past memtest has flagged memory issue that HP built in diags hasn't but it ran clean.

The server was rebooted prior to replacing the memory out of curiosity and everything came up OK no errors.

So we assume that something was wrong with the memory. And now we have are fingers crossed that that is the end if it

Hi Oliver,

My apologies I have wasted your time (though I did learn about the XOA so not totally wasted)

The old adage "If you want a job done properly. Do it yourself" is still very true.

Long story short when I looked into the issue myself, my colleague was using the wrong XO installation. This VPN isn't site to site so for now we have vmware running 2 nat'ed VMs, XO and a windows client. The windows client accesses XO and XO goes out over the hosts vpn connection to the remote site.

When using this correctly everything is OK

Case closed

@olivierlambert said in VM's going really slow after 3 - 4 weeks:

Oh wow. So something was wrong with the hardware, specifically the memory, and potentially more errors and error correction trying to keep up, slowing down everything. Is it a good recap?

To be honest I am not entirely sure.
I tested the memory my self in the early stages of this issue with memtest. In the past memtest has flagged memory issue that HP built in diags hasn't but it ran clean.

The server was rebooted prior to replacing the memory out of curiosity and everything came up OK no errors.

So we assume that something was wrong with the memory. And now we have are fingers crossed that that is the end if it

@Berrick
So its been a while since my last post on this topic but finally I think we have bottomed this issue out!

So the long and the short.
Nothing we tried, checked or diagnostics ran up to this point showed any issue.
Then one Monday morning there was one alert from the iLO about "Corrected Memory Error threshold exceeded"

As this server was due a memory upgrade to LV memory soon and we would need to reboot to clear the alert we brought the upgrade forward.

Since the new memory was installed, tempting fate here, it has been OK

Thanks to all that offered advice.

Kind regards

Thanks for all the replies.
I will be checking when the issue starts again in a few weeks and report back.

From previous xentop output I don't believe Dom0 is showing high CPU or Memory.

I also dont think there is a problem with the physical disks.

@christopher-petzel
Many thanks saved me from hours of searching for a fix

Seasons greetings to all,

So the slowness of is creeping back. Still cant find any thing obvious
Latest Xen Top

Due to business pressures we have had to reboot the physical server.
So now have to wait for the issue to re occur.

there were 5 more patch's which have been applied.

If anyone has any other suggestion of what to check, specific logs to search etc, etc they would be gratefully received

Kind regards

@Gheppy What is free disk space like on this VM?

This is what caused the CPU's in our graylog server to go nuts. It had almost run out of free disk space. The only thing I have done was to increase free disk space by clearing out some of the older logs.

@Danp said in VM's going really slow after 3 - 4 weeks:

@Gheppy I was surprised to see a graylog VM using so much CPU, but maybe it's normal.

Not sure if Gheppy DB is graylog. Mine was and when I searched for an answer as to why the strange CPU utilization came up with the same answer Oliver supplied.

I would like to point out, as I didn't earlier, that the graylog cpu utilization in the xentop image I up loaded has been fixed so CPU util is much much less now.

However, the CPU utilization of that vm was also at the high levels after a server reboot so dont think its the answer to why all vm's slowly slow down

@mjtbrady

We ran xentop after you suggested it but dont believe it shows anything?

Just recently we had notification of another lot of patch's which have been applied.

So now we wait!

Thanks for the replies.

@olivierlambert I have looked at the logs but as mentioned I can seen nothing which to me indicates an issue. But to my peers; I could be talking rubbish.

As all VM's appear to suffer and rebooting the physical server corrects the "issue" short term, am thinking what is common to all, hence Dom0.

As testament to xneserver/xcp it has pretty much just worked and until now, any issue's that have occurred I have been able to find the answer. So haven't really learnt a whole lot regards trouble shooting.

@Anonabhar No, I dont believe so. See below

# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/sda6                               partition       1048572 0       -2

# grep Swap /proc/meminfo
SwapCached:            0 kB
SwapTotal:       1048572 kB
SwapFree:        1048572 kB

Hi, So Only been about 2 weeks since last server reboot and already VM's are noticeably slower.

Still struggling to make any head way regards how to fault find this issue....

Is Dom0 the most likely culprit. Or could it be storage?