XCP-ng 8.3: Broadcom BCM57414 `bnxt_en` Driver Fails to Probe on HPE DL380a Gen12

yannsionneau

FYI this is not being forgotten, we still have a card about this assigned to someone, it will just take a bit more time than I thought before we work on this.
I'll send updates here when we have new elements.
Regards,
Yann

maximsachs

@yannsionneau Thanks for the update! We are eagerly awaiting your findings! Thanks for looking into it.

maximsachs

Hi @yannsionneau, have you had the time to investigate this issue some more? We would love to hear if there is already a solution.

ThierryEscande

Hi @maximsachs,

Sorry for the delay.

From what you describe, the driver seems to be probed in all cases but I have a doubt regarding the driver from xcp-ng 8.2. I rebuilt it specifically for 8.3. To completely eliminate a driver issue, can you try this RPM ?

From the host, this can be done by running the following commands:

$ wget https://nextcloud.vates.tech/public.php/dav/files/R33Dwpt5gjy6CCr/broadcom-bnxt-en-1.10.0_216.0.119.1-1.0.82srcs.0.xcpng8.3.x86_64.rpm
$ yum update ./broadcom-bnxt-en-1.10.0_216.0.119.1-1.0.82srcs.0.xcpng8.3.x86_64.rpm

Also, since this xcpng-8.2 release seems to have support for device IDs that have been removed from the 8.3 one, can you give the output of the following shell commands:

$ lspci -nn -s 0001:86:00.0

and

$ lspci -nn -s 0001:86:00.1

Regards,
Thierry

yannsionneau

Hello @maximsachs,

on top of trying what @thierryescande told you, could you also give a try to this ISO installer please?
We have done some internal investigations and we think that the issue could be because of an issue in Xen handling of PCI devices that are on 0001 segment which seems to be your case.

Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).

unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683

Regards,
Yann

maximsachs

@yannsionneau @thierryescande

Thank you very much for the updates!

We are scheduling some downtime on the services in the upcoming weeks, then I hope to be able to try out all your suggestions. Our fallback plan is to switch the Broadcom network cards for other Melanox cards.

I will be posting the results once we have them!

maximsachs

Unfortunately, we will have a bit of delay from our side. We are only able to get sufficient downtime for testing with the servers in the first half of August. So I will have to wait until then to try out the fixes.

yannsionneau

@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.

Thanks!
Regards,
Yann

maximsachs

Hi we have reached our scheduled downtime window!

I ran the PCI-e commands to check as @thierryescande requested:

[15:36 REDACTED ~]# lspci -nn -s 0001:86:00.0
0001:86:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)
[15:46 REDACTED ~]# lspci -nn -s 0001:86:00.1
0001:86:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)

The custom ISO from @yannsionneau was able to succesfully upgrade our 8.2 install to 8.3! The installer ran through just as a "normal" iso would.

Below is the DMESG output, it seems to also confirm that the root cause may be the removal of the device IDs on 0001.

dmesg_custom_iso_redacted.txt

Thank you very much for your great advice and help!

Kind regards,
Maxim

olivierlambert

Very cool, thanks for the feedback @maximsachs !