MS-01 performance issues w/ Intel 226 NICs

xerxist

This is probably the same issue as I see on my NUC11 and NUC13.

Try turning off ASPM in the BIOS and see if you still have the same issue.

cowboy

@xerxist It's good you commented. I finally figured this out and had intended to come back to update this post but forgot. Yes, that was the magic secret that solved it all.

Now, if only we could get Thunderbolt Networking support in XCP-NG, it would be perfect, but I can imagine that's not a high priority for the Vates team.

None the less, my 3 X MS-01 cluster (not setup as a pool.... yet) continues to work very well for it's first month in service.

xerxist

@cowboy

Good stuff

I wish I could use it but still waiting for them to start using a new kernel. As on the NUC13 I can’t use my igpu for passthrough on an old kernel. Need something above 5.15

Andrew

@cowboy Did you use pcie_aspm=off in GRUB or disable it in the BIOS?

xerxist

@Andrew

BIOS but if you dont have to option there you could use that.

Andrew

@xerxist If you have time to test options, it would be interesting to see if other choices fix the problem (disable and off should be the same).

pcie_aspm=disable
pcie_aspm=off
pcie_aspm.policy=performance

cowboy

@Andrew I flipped it off in BIOS, worked like a charm. BTW, I'm only on a 1G switch with my 226's. Looking forward to getting a faster network switching at home now.

Looking forward to eventually test driving XOSTOR on this cluster.

xerxist

@cowboy

Just got 2 new ones. 2,5Gbit from TP-Link Omada SG3210X-M2.

Working pretty good.

xerxist

@Andrew

I’m good with the BIOS setting for my NUC11 and 13. I wish Intel would fix ASPM, but still not on the latest BIOS. Just can’t run xcp-ng yet because of the older kernel and passthrough issue on the NUC13.

It will probably come soon as 4.19 is out of support pretty soon. Was actually hoping there would be a alternate kernel for testing already.

bugsy

@Andrew said:

pcie_aspm=disable

Just to help anyone who would run into it.
Disabling ASPM via dom0 settings/kernel did not resolve the issue. Had to disable it in BIOS (NUC13)

After kernel level disable it did show:

lspci -vv -s 55:00.0 | grep -E 'LnkCap|LnkCtl|LnkSta'
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <2us, L1 <4us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

Only after BIOS disable it showed

lspci -vv -s 55:00.0 | grep -E 'LnkCap|LnkCtl|LnkSta'
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <2us, L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

Running on
NUC13ANBi7

Hope it helps anyone running into this problem.