XCP-ng 8.3 and Dell R660 - crash during boot, halts remainder of installer process (bnxt_en?)
-
Hi,
Yes, it's pretty standard hardware. My first reflex would be to run a memtest on this config first.
-
@umbradark Hello, there are new fixed drivers in the latest bnxt_en driver disk. Could you specify
modprobe.blacklist=bnxt_enthen apply the linked driver disk to see if it works? -
@umbradark Maybe too obvious, but is your boot configuration set up to be BIOS or EUFI mode?
-
Broadcom hardware never disappoints.
-
@dinhngtu Thanks for the suggestion. I'll give this a shot today.
-
@tjkreidl Hi, thank you for the reply. I've tried both BIOS and UEFI boot methods. No change, same driver panic during boot (which then appears to hang the subsequent systemd units).
-
All - thank you for the feedback. It appears the issue is the embedded
BCM5720BCM57504, and not the other NICs. I was able to boot the installer and upgrade an xcp-ng 8.2.1 host by disabling the BCM57504 NIC in BIOS.We don't use the embedded NICs, so I'm tempted to just leave it disabled and move on for the time being. I'll experiment with the updated driver at a later time, as this may be needed by other users.And yes, Broadcom always coming through with the quality products.
-
@umbradark said in XCP-ng 8.3 and Dell R660 - crash during boot, halts remainder of installer process (bnxt_en?):
And yes, Broadcom always coming through with the quality products.
hahaha. True. I'm not even surprised, their drivers (and even some NIC hardware) are really shitty.
Glad you "fixed" it
-
@olivierlambert I've always preferred Intel NICs.
-
FYI, this NIC worked until recently. We suspect new hardware revision and/or a new firmware in it, that is causing a bug with the current XCP-ng driver.
-
@olivierlambert when planning our infrastructure, Vates engineer told us to avoid Broadcom nics but to go Intel or nvidia/Mellanox.
we bought Mellanox
This is the way.
-
@olivierlambert I heard back from Dell, and they suggested I should "approach Vates and have the OS validated through Dell Product Group". Has Dell ever contacted you or Vates about anything like this?
-
@umbradark Nope but we'll be happy to do so.
-
[Removing, as proper solution is described below]
-
@umbradark or anyone from Vates.
I have not read all the details so I apologize up front. Not sure if this is helpful or note but I have at work 2 Dell R660 with 8.3 running no issues this was from a fresh install. I did not have to disable any nic in bios but i am not using the BCM5720 but it is there and active if i wanted to.
So far everything is stable that I can tell. If you need me to pull any information that will help just let me know. Below is my hardware information.
-
Hi everyone,
Apologies for the earlier confusion about the NIC model — the actual problematic hardware is the Broadcom BCM57504 quad-port NIC built into some Dell PowerEdge R660 systems.
When attempting a clean install of XCP-ng 8.3 (June 2025 ISO) with this card enabled, the installer crashes or fails to detect any network interfaces.
The issue appears to stem from the olderbnxt_endriver included in the installer.
Once the host is updated with the newer Broadcom driver (broadcom-bnxt-en-1.10.3_232.0.155.5-1.xcpng8.3), the system becomes completely stable and all ports enumerate correctly.For reference, blacklisting the
bnxt_enmodule during installation did not allow the installer to proceed — the only working method is described below.
Workaround / Installation Procedure
Step 1 — BIOS Pre-Install
- Reboot and press F2 to enter BIOS setup → System Setup → Integrated Devices.
- Disable only the Integrated Network Card 1 (the BCM57504 quad-port NIC).
- Embedded NIC1/NIC2 can remain enabled.
- Save and exit BIOS.
Step 2 — Install XCP-ng 8.3
- Boot from the June 2025 XCP-ng 8.3 installer ISO.
- Proceed with a normal installation.
- With the BCM57504 disabled, the installer will complete normally using any other available NIC.
Step 3 — Update the Host (Critical)
After the first boot, run the following commands:
yum clean all yum update -yConfirm that the Broadcom driver has been updated to:
broadcom-bnxt-en-1.10.3_232.0.155.5-1.xcpng8.3.x86_64This updated driver resolves the crash and detection issues.
Note:
Testing confirmed that using the F9 driver disk method during install does not resolve the issue — the installer still fails to enumerate the BCM57504 properly, even when the updated driver ISO is loaded before installation.
The only reliable method is to complete installation with the NIC disabled, perform a fullyum update, and then re-enable the NIC afterward.
Step 4 — Re-enable the Integrated NIC
- Shut down the host cleanly:
shutdown -h now - Re-enter BIOS (F2 → Integrated Devices) and re-enable:
Integrated Network Card 1 → Enabled - Save and power on.
At this point, the BCM57504 will enumerate correctly and the host will boot without any kernel panics or NIC detection issues.
Notes
- This behavior is isolated to the BCM57504 and the older
bnxt_endriver shipped with the June 2025 installer. - The issue does not occur once the updated driver is installed via
yum update. - Other Broadcom adapters, such as the BCM57414, install normally.
- Blacklisting the driver or loading a driver disk with F9 during install does not bypass the issue — the NIC must be disabled in BIOS until after updating.
Hopefully this helps anyone else running into the same issue with R660 systems and the Broadcom 57504 NIC.
Once the host is updated and the NIC is re-enabled, everything works as expected. -
@umbradark said in XCP-ng 8.3 and Dell R660 - crash during boot, halts remainder of installer process (bnxt_en?):
Blacklisting the driver during install does not bypass the issue — the installer still fails to proceed without the NIC disabled in BIOS.
any idea if there is a problem with BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller ?
I have two clusters of 3 hosts to upgrade to 8.3 soon
-
Thanks for your feedback @umbradark
Let me reping the @Team-OS-Platform-Release so we can think about a good strategy to reduce the needed steps
-
@olivierlambert Thanks so much for the quick response and for looping in the Team-OS-Platform-Release folks — I really appreciate it.
I’d be more than happy to test an updated installer ISO if one is released. There’s a good chance I’ll be setting up at least one more R660 with the same hardware configuration in the near future, so this would be a perfect opportunity to validate the fix.
Thanks again for all your help and attention on this.
-
I'm going to build an updated installer with an updated
bnxt_endriver, as more and more servers require it.
