XCP-ng 8.3: Broadcom BCM57414 `bnxt_en` Driver Fails to Probe on HPE DL380a Gen12

yann

That's indeed more on @Team-Hypervisor-Kernel side. Maybe a driver/firmware version mismatch?

yannsionneau

Hi,

Someone from the Hypervisor & Kernel team will have a look shortly, we'll get back to you with our findings.

Thanks a lot for the very detailed report!

Yann

yannsionneau

FYI this is not being forgotten, we still have a card about this assigned to someone, it will just take a bit more time than I thought before we work on this.
I'll send updates here when we have new elements.
Regards,
Yann

maximsachs

@yannsionneau Thanks for the update! We are eagerly awaiting your findings! Thanks for looking into it.

maximsachs

Hi @yannsionneau, have you had the time to investigate this issue some more? We would love to hear if there is already a solution.

ThierryEscande

Hi @maximsachs,

Sorry for the delay.

From what you describe, the driver seems to be probed in all cases but I have a doubt regarding the driver from xcp-ng 8.2. I rebuilt it specifically for 8.3. To completely eliminate a driver issue, can you try this RPM ?

From the host, this can be done by running the following commands:

$ wget https://nextcloud.vates.tech/public.php/dav/files/R33Dwpt5gjy6CCr/broadcom-bnxt-en-1.10.0_216.0.119.1-1.0.82srcs.0.xcpng8.3.x86_64.rpm
$ yum update ./broadcom-bnxt-en-1.10.0_216.0.119.1-1.0.82srcs.0.xcpng8.3.x86_64.rpm

Also, since this xcpng-8.2 release seems to have support for device IDs that have been removed from the 8.3 one, can you give the output of the following shell commands:

$ lspci -nn -s 0001:86:00.0

and

$ lspci -nn -s 0001:86:00.1

Regards,
Thierry

yannsionneau

Hello @maximsachs,

on top of trying what @thierryescande told you, could you also give a try to this ISO installer please?
We have done some internal investigations and we think that the issue could be because of an issue in Xen handling of PCI devices that are on 0001 segment which seems to be your case.

Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).

unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683

Regards,
Yann

maximsachs

@yannsionneau @thierryescande

Thank you very much for the updates!

We are scheduling some downtime on the services in the upcoming weeks, then I hope to be able to try out all your suggestions. Our fallback plan is to switch the Broadcom network cards for other Melanox cards.

I will be posting the results once we have them!

maximsachs

Unfortunately, we will have a bit of delay from our side. We are only able to get sufficient downtime for testing with the servers in the first half of August. So I will have to wait until then to try out the fixes.

yannsionneau

@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.

Thanks!
Regards,
Yann