Epyc VM to VM networking slow
-
@olivierlambert said in Epyc VM to VM networking slow:
No obvious solution yet, it's likely due to an architecture problem on AMD, because of CCDs and how CPUs are made. So the solution (if there's any) will be likely a sum of various small improvements to make it bearable.
I'm going to Santa Clara to discuss that with AMD directly (among other things).
Do we have other data to back this? The issue is not really common outside of Xen. I do hope some solution comes out from the meeting with AMD.
-
If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.
-
@olivierlambert
Can we rule out extra_guest_irqs as the root cause of this problem?https://docs.xcp-ng.org/compute/#nvme-storage-devices-on-linux -
It's probably completely unrelated, but feel free to test
-
@olivierlambert said in Epyc VM to VM networking slow:
If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.
Hi, is there anything new to report on this? We have very powerful machines, but unfortunately limited by this stubborn issue.
-
@Forza Dito. A 15.000€ EPYC HP monster is slower than a 1.600€ Protectli Intel...
This is a joke and had we known this we'd NEVER jumped on the AMD wagon
-
@Forza said in Epyc VM to VM networking slow:
olivierlambert said in Epyc VM to VM networking slow:
If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.
Hi, is there anything new to report on this? We have very powerful machines, but unfortunately limited by this stubborn issue.
Can you test https://xcp-ng.org/forum/topic/10862/early-testable-pvh-support ?
We observe very significant improvements on AMD EPYC with PVH.
We're still pin-pointing the issue with HVM, the current hypothesis is a issue regarding memory typing (grant-table accessed as uncacheable(UC) which is very slow) related to grant-table positionning in HVM.
-
@TeddyAstie Unfortunately not. This is a production pool on 8.2.1 so I do not want to try too experimental things.
Do we know if the issue happens on plain Xen on a modern (6.12-15) dom0 kernel?
-
It's on any Xen and Linux version. Vates is now the spearhead on finding the problem and a solution, there's no upstream with a fix anywhere.
-
OK, thanks for the update. I would be interesting to hear what AMD said about this issue.
-
Our most promising lead is that's due to the fact they do not have a feature Intel got, called iPAT.
In very short (and probably too short to be entirely correct), is the fact that the grant tables in the guest (used to securely communicate between -in that case- the VM and the Dom0) is not cached by AMD CPU. And on AMD, there's no way to force a cache attribute on a guest memory access, unlike with Intel. So the grant table requests are not cached on AMD vs Intel, explain at least a part of the performance difference.
What's next? Roger from Xen project pointed us in that direction, and he did a very crude patch that demonstrated that we tested internally, showing that's a promising lead (x5 perf in VM->Dom0 and near twice between VMs). Right now, we have multiple people working internally to make a "real" patch or at least something to "workaround" the issue if possible.
So it's been few weeks since then, we are trying to figure (at Vates, again) what would be the best approach for AMD CPUs, to make a patch that could land upstream.
-
@olivierlambert I know that I am late to this thread, but I would like to ask if there is any realistic time estimate for a workable fix, or even for a temporary patch or workaround?
We have been trialing XCP-ng with the compiled version of Xen Orchestra as a potential replacement for VMware in a fairly large multi-datacenter environment, before doing an "official" proof of concept. My concern is that we've done all of our testing on our freshly retired older Intel virtualization hosts - as we've just completed replacing everything with AMD EPYC-based servers. Until now, our only matter of concern has been the 2TB virtual disk limit. I feel that this is going to be a much larger issue for us. It sounds as if I need to pull in a pool of EPYC systems to expand our testing.
Thanks in advance for any input or guesstimates that you may be able to provide. -
@linuxmoose What kind of workloads are you needing network wise though? Like we aren't talking about unusable performance, it's just not as good as Intel.
Unless you're doing higher bandwidth stuff I don't really foresee it posing much of an issue.
-
@linuxmoose said in Epyc VM to VM networking slow:
I feel that this is going to be a much larger issue for us.
Before that I would strongly encourage to test if it's really a problem, because it's really not in 90% of the use cases we've seen.
-
@olivierlambert Biggest issue is when it presents itself during backup, which are SLOW compared to Intel.
We've been at that since the beginning of our Vmware-XCPNG switch. Unfortunately (in a hindsight we chose AMD EPYC. Huge mistake!!)
-
I wasn't addressing you but @linuxmoose having theoretical concerns.
-
@olivierlambert Just added my pratical experience, which might interest. AND that was not a shot AGAINST xcpng. FAR FROM IT.
Switch, do it, tomorrow! Just don't choose EPYC.
-
Yes but I don't think there's a need to add another "layer on the cake" about this, you already know we are convinced already about getting it fixed in priority and investing nearly half a million already to fix it.
It's really rare when it could become a blocker, and the proof is it took years to people to even notice (even us, having an EPYC production infrastructure).
-
@olivierlambert OK, won't comment any more... You can delete my comment at your leisure.
-
That's not what I meant either. I was just answering initially about the theoretical concern, mostly following @planedrop logical question, in other words "before fearing something, test it for real and see the real impact toward your requirements".
We already know you are affected, it's like you were afraid we wouldn't care about your use case while we are already deeply invested into getting a solution to a tricky problem that could affect some people in some situations.
