Mellanox ConnectX-3 - Card not working

Andrew

@Pyroteq If you goal is to get 10G ethernet then buy a newer better supported card... (like Intel x540)

The ConnectX-3 is rather old and odd. You can try a firmware update/reflash to the card as old firmware on the Mellanox cards is a known problem. There are people using them with XCP.

Also, there are several versions of the Mellanox card, including dedicated FCoE.

If you MUST use Mellanox then use the newer (but still old) ConnectecX-4 cards (10/25G ethernet).

Pyroteq

I think I'm getting a bit closer...

lspci -k
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
        Subsystem: Mellanox Technologies Device 0055
        Kernel modules: mlx4_core

lsmod | grep mlx
mlx4_core             352256  0
devlink                77824  1 mlx4_core

dmesg | grep mlx
[    7.304630] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    7.304646] mlx4_core: Initializing 0000:04:00.0
[    7.304799] mlx4_core 0000:04:00.0: Missing UAR, aborting

Seems the driver is failing to load. Google suggests it could be caused by SR-IOV (pretty sure I've got all my virtualisation options on in the BIOS) or I might be able to play around with the grub configuration to get it working.

The reason I went with this card in particular was because of the form factor. In this machine I've got 2 graphics cards and a TV tuner card already. I have a 1X slot (which basically nothing will fit in) and the bottom X16 slot only runs at 4X (which the ConnectX3 is a 4x PCI-e 3.0 card).

Supposedly another possible fix is to edit the firmware and flash a firmware that limits or removes SR-IOV functions from the card.

EDIT - Took out of hypervisor and installed into my gaming rig. Different motherboard, but same chipset (AMD X570). Windows 11 picked up the card instantly without need for driver installation. Plugged DAC into a 10Gbit SFP+ Juniper switch and Windows reads 10Gbit speed.

So card is definitely working properly but XCP-NG 8.3 doesn't like it.

Anonabhar

@Pyroteq Just as a point of reference, I use Mellanox ConnectX-3 cards all the time in my home lab for 40G Ethernet connection to my SAN. No problems with them at all. But as everyone has mentioned before, be careful of the firmware and model.. Some of the cards will only work in IB mode and others can work in IB or ETH mode.

[09:43 xcp-ng-GHF4 ~]# lsmod | grep mlx
mlx4_en               135168  0 
mlx4_core             352256  1 mlx4_en
devlink                77824  2 mlx4_core,mlx4_en
[09:44 xcp-ng-GHF4 ~]#

[09:44 xcp-ng-GHF4 ~]# dmesg | grep mlx
[   16.566265] mlx4_core: Mellanox ConnectX core driver v4.0-0
[   16.566299] mlx4_core: Initializing 0000:03:00.0
[   22.903721] mlx4_core 0000:03:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[   22.910404] mlx4_core 0000:03:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[   23.122348] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[   23.122566] mlx4_en 0000:03:00.0: Activating port:1
[   23.127040] mlx4_en: 0000:03:00.0: Port 1: Using 16 TX rings
[   23.127042] mlx4_en: 0000:03:00.0: Port 1: Using 16 RX rings
[   23.127454] mlx4_en: 0000:03:00.0: Port 1: Initializing port
[   23.128547] mlx4_en 0000:03:00.0: registered PHC clock
[   23.128902] mlx4_en 0000:03:00.0: Activating port:2
[   23.131517] mlx4_en: 0000:03:00.0: Port 2: Using 16 TX rings
[   23.131518] mlx4_en: 0000:03:00.0: Port 2: Using 16 RX rings
[   23.131678] mlx4_en: 0000:03:00.0: Port 2: Initializing port
[   27.210288] mlx4_core 0000:03:00.0 side-9894-eth2: renamed from eth2
[   27.244783] mlx4_core 0000:03:00.0 side-701-eth3: renamed from eth3
[   35.418878] mlx4_core 0000:03:00.0 eth0: renamed from side-9894-eth2
[   35.582138] mlx4_core 0000:03:00.0 eth4: renamed from side-701-eth3
[   38.171360] mlx4_en: eth4: Steering Mode 1
[   38.189737] mlx4_en: eth4: Link Up
[   38.726934] mlx4_en: eth0: Steering Mode 1
[   38.744587] mlx4_en: eth0: Link Up
[   41.586228] mlx4_en: eth4: Steering Mode 1
[   41.756191] mlx4_en: eth0: Steering Mode 1

Pyroteq

@Anonabhar

I just edited my post as you posted. I just tried it on Windows 11 and it worked instantly. This particular card doesn't support Infiniband, it's ethernet only so I don't believe that should be relevant in this case. It's the CX311A EN model.

The problem seems to lie in the drivers on XCP-NG or in the firmware of the card.

I'm not sure if I should be upgrading or downgrading the firmware or as some people suggested editing the firmware and limiting some aspects of the card to try and get it working.

I saw some posts about PCI=REALLOC=OFF in Grub from the Red Hat community support page, but wouldn't I want this on for a hypervisor?

olivierlambert

I'm not aware of existing issues with those cards in the past (except IB vs Ethernet) So that's weird.

Anonabhar

@Pyroteq I have seen others mention that its a lack of physical resources being allocated to the card.

As far as the firmware.. I think fw-ConnectX3-rel-2_42 is the latest for that series of card.

Pyroteq

@Anonabhar seems to be an issue with the machine itself.

Tested my gaming rig and my server.

They're both AMD X570 chipsets.

Loaded up Ubuntu 22.04 live USB on each system. On my gaming rig it worked straight away without issue. Connected DAC to switch and it reported a 10gbit connection. On the server it has the same UAR Missing error message in dmesg and didn't show up as a network interface but appeared in lspci just as before.

Either I'm missing a setting hidden away in the BIOS or there's some very weird hardware conflict or incompatibility.

I'm installing Ubuntu to a portal SSD now and then I'll try out the firmware tools and see if I have any luck.

olivierlambert

Good idea, I have the impression there's something weird about this setup

Pyroteq

@olivierlambert

In the end the solution was so simple... I was up until 2am trying to fix this. Updating the firmware, messing up Grub on my XCP-NG installation and having to recover off an XCP-NG 8.2 backup, hours wasted troubleshooting the Mellanox firmware tools (During installation you have to enter your password multiple times and there's no prompt or anything - No idea how someone discovered that work around).

I saw lots of people mentioning adding pci=realloc=off to kernel boot parameters and I tried that without success.

Just for the lolz I figured what if I turned it on instead?

pci=realloc=on

dmesg | grep UAR

Can no longer see the UAR missing error from before... Seems promising...

lspci -k | grep -A 2 Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
      Subsystem: Mellanox Technologies Device 0055
      Kernel driver in use: mlx4_core
      Kernel modules: mlx4_core

Kernel driver is in use now, before it wasn't.

Rescan PIF in XOA and I can see it! Shows connected at 10gbit, is able to get IP address via DHCP, etc.

Everything seems to be working...

Now I've just gotta test that all my other PCI-e devices are passing through ok with this option enabled.

olivierlambert

That's weird, never heard about this option to be needed to make it work, does it ring a bell @fohdeesha ?