Oh wow. So something was wrong with the hardware, specifically the memory, and potentially more errors and error correction trying to keep up, slowing down everything. Is it a good recap?
To be honest I am not entirely sure.
I tested the memory my self in the early stages of this issue with memtest. In the past memtest has flagged memory issue that HP built in diags hasn't but it ran clean.
The server was rebooted prior to replacing the memory out of curiosity and everything came up OK no errors.
So we assume that something was wrong with the memory. And now we have are fingers crossed that that is the end if it