@olivierlambert said in VM's going really slow after 3 - 4 weeks:

Oh wow. So something was wrong with the hardware, specifically the memory, and potentially more errors and error correction trying to keep up, slowing down everything. Is it a good recap?

To be honest I am not entirely sure.
I tested the memory my self in the early stages of this issue with memtest. In the past memtest has flagged memory issue that HP built in diags hasn't but it ran clean. 😞

The server was rebooted prior to replacing the memory out of curiosity and everything came up OK no errors.

So we assume that something was wrong with the memory. And now we have are fingers crossed that that is the end if it 🙂

In Xen Orchestra, XCP-ng Center isn't a supported client for XCP-ng.

@Danp You mean to try to allow UEFI bios during the boot for this istance of Debian11?

@olivierlambert i have opened the new post, thank you for your support https://xcp-ng.org/forum/topic/7015/stuck-on-boot-hard-disk-success

Make sure you don't install the KB5022842 update. Below is a link to an article with more info.

https://www.bleepingcomputer.com/news/microsoft/emergency-vmware-esxi-update-fixes-windows-server-2022-vm-boot-issues/

The graph is doing average for the last 5 seconds You are presuming there's a perfect mapping between 1x vCPU and 1x physical CPU. Xen could schedule one vCPU into various physical CPUs. Likely, one of your vCPU load is spread on more physical CPUs.

Remember: for Xen, your whole VM is like a process. It will be scheduled on "real" CPUs in a way that might be not "what you see" from the guest.

Smells like an ACPI/firmware bug, HPE is "known" to be a tad worst than Dell on this.

Also as @Danp said, when Googling this error, it seems a recurring issue that is affecting other distro too.

"the BIOS has corrupted hw-PMU resources" gets displayed as the OS is trying to use a performance counter the BIOS is using. You can either ignore the message, or turn off performance monitoring in the BIOS as shown in this advisory: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c03265132

HA is enabled at the pool level. Read this blog post I wrote 9 years ago but still relevant: https://xen-orchestra.com/blog/xenserver-and-vm-high-availability/

On second thought, I was not clear in the beginning. I don't expect to see 1 socket with 128 vCPU in the VM, but maybe 2 sockets with splitted vCPUs between them if I assign more than 64 to the VM. Initially I had 1 socket with 64 CPU and after turning ACPI off (either in grub or in the VM config itself), secondary socket appeared in the VM with the rest of the CPUs. More funny, turning ACPI off while running the cloud kernel of Debian makes the VM see one CPU only.

Also something to keep in mind: It's not only about NUMA (which is different since 2nd Epyc gen, as they have all memory channels on an IO-Die and only split the caches now), it's also about memory bandwith!

So it adds more complexity and depends on the needs of your workload.
If it benefits from high memory bandwith, a 2nd socket doubles it (technically)!

I updated XCP-ng to 8.2.1 yesterday.
I will write if issue not gone.

@olivierlambert Yes, make sense 🙂

There's ongoing effort to avoid opening file descriptors, to avoid this situation. I'll consult whether the current patches avoid this situation in template-export / vm-import

https://github.com/xapi-project/xen-api/pull/4877

snwoods created this issue in xapi-project/xen-api open CP-32622: avoid using select and instead use epoll #4877

Thanks for the feedback @nvs 🙂 Glad to see we fixed those issues!

@fdrcrtl @andSmv @olivierlambert
Bump, ty

@Andrew I'm using the same. I only tested it on one of the monitors here. I have a few more in the field I will test it on later. Before doing this I would always get a "resolution out of bounds" error on the monitors. (one 4:3 1024 and one 16:9 1366)

Hi,

"Native" in Xen context doesn't mean anything since the Dom0 is already a VM. So even running Docker in the dom0 won't be "native" 😉

RunX is still tech preview yes. It doesn't answer all the question.

What do you want to achieve functionally speaking?