@Forza We have tried all the settings avail to tweak on our hardware, We have a full big twin chassi we have dedicated to vates doing testing of this issue and the first roughly month was spent going over settings together with vates and making sure everything was tweaked properly.

AMD is involved themself and if anyone knows AMD firmware settings and tweaking it would be AMD.

In all seriousness it is a very good suggestion but it has been looked into and unfortunately do minimal or no difference.

@nicols Hello, you are not alone in this. We have delayed the deployment of our core ISP routing as virtual routers due to this.

We have an XOA premium and xcp-ng enterprise subscription and an open case on this.

I cannot go into detail about what is being said in there but i can say that the approach @olivierlambert and vates have is very good and they have my full confidence on this one.

@manilx i can say this affects all vm traffic as we have vm out to switch over to a second hardware host into a VM there and we see the limit there aswell just the roof is twice as high as inside the same HW box

Either way. Vates is VERY well aware of how serious this is and are taking good efforts to fix it.

Heya!

Just chiming in that we (WDMAB) Are keeping tabs on this thread as well as our ongoing support ticket with you guys.

Saw our result up on the list.
If we can do ANYTHING further to assist then please do tell us. We are available 24/7 to solve this issue since it is very heavily impacting to our new production deployment.

Regards.
Mathias W.

@manilx it is deffo interesting to see more proof that this but may be wider than expected.

@john-c That would only work for someone where it is possible to have non epyc machines handle this tho Unfortunately not an option for us.

@olivierlambert Sure but the traffic still goes to a VM and through it and then out, That is also affected by the issue. Any traffic through a VM is affected by this bug as we have established earlier.

@olivierlambert Define "Outside of master"? Because since backup traffic passes through the XOA VM if it is hitting this issue with VM traffic then that does not help does it?? Or do you mean outside of the pool entirely? As in a physical machine?

@manilx Oh i absolutely agree that it is an issue... Maybe i could see that due to backups being handled by the XOA VM then whatever is causing our slowdowns for network between VMs (And out of VMs to external) might impact the networking and/or the process of the XOA backup process too.

What do you think @olivierlambert is these perhaps directly related? It sure would explain our very low backup speeds aswell that we see, (we have fully loaded synology FS2500's (all flash) with write intensive SSDs in.)

@manilx i dont think it is directly related due to just how low it is. But we also see similar "Lower than expected speeds" on backups

@timewasted

Spread network heavy VMs across the cluster as it is a per physical host limit and also changed our design a bit where we intended to have all levels of routers virtual we split out the core routers from that and they are physical.

@LennertvdBerg they are still trying to figure this one out.

And an estimated full fix is not in sight just yet from what i know. Atleast i havent been informed in my ticket with them about this. But i do know they are still working very hard on this.

@sluflyer06 This test does not say anything other than that you have a 10G nic and we already knew that the limit for latest gen amd's are just above 10G. If you insert an 25 G nic then you can only use half of that capacity likely and for some of us that are using this in actual datacenters that is a pretty critical issue.even more so when it seems the limit is shared per host so that 4 VMs running on same host if the limit is 12gbit means you get 3 gbit per vm. And when you realize lots of us may have 20-40 VMs per server that all use a decent portion of network it is suddenly really scary whenn you realize that is 300-600 mbit per server.

Or even worse when you realize that for those that have earlier gens of amd platform where the limit is 2-4 gbit ish.. now you re looking at 100-200 mbit per vm which suddenly is not very unobtainable for even a smaller provider during peak use times.

It is great that the issue is not triggered for you as your bottleneck is elsewhere, but it is a very serious issue for several of us.

With that said, Vates is handling it as good as anyone could request and i thank them for the attention given and the dedication to solving it.

It is a NASTY bug and very situational for it to have been discovered.

I can unfortunately share that from ongoing ticket investigations in this, It is far more deeply rooted than something that a patch of going from one major kernel to another will "just fix" There are multiple leads being investigated and multiple vendors involved.