I'm having an issue when comparing CPU performance between two seemingly identical xcp-ng hosts. Both hosts have the same physical hardware (Dell Chassis, Mobo, CPU, Memory, Storage and Firmware versions).

When I create a VM on each host with the same software/vCPU count/memory, one of the hosts performs considerably better in sysbench.

All workloads/benchmarks that I am testing with are configured for using a single CPU thread (not multi-threading).

I have found that on the host that is performing worst, that if I run sysbench in dom0 the results are much better than if I run sysbench in domU.

On the faster hosts I do not see this difference, both domU and dom0 sysbenches are very similar.

This leads me to believe that the issues are probably not hardware related as dom0 is performing as expected and similar across all, it is only domU on this specific host that is not performing as well as it could be.

I am trying to figure out what could be different between these hosts/vm's causing the performance difference.

I believe I have ruled out the following factors:

• Physical hardware difference
• Hyper threading (compared with enabled and disabled and little - no performance difference)
• CPU topology for the VM (both VM's are running the same CPU topology)
• Virtualisation mode (both VM's are PVHVM running same drivers)
• VM level OS issue (tried fresh VM and different versions on both good and slow hosts)

I believe these hosts were both upgraded from an earlier version of Xen (possibly both from different versions as they were not originally setup at the same time).

Both hosts are currently running XCP-NG 7.6.0.

Is there a chance that the migration from older Xen versions would carry over a legacy configuration within XCP that is causing the performance difference?

Does anyone know of any other possibly causes for the performance difference?

Any clues on what to check / where to go next with my research? I really don't want to just format and forget / I need to learn from this issue as we may have other hosts with the same problem I've not found yet.

Thanks in advance.