XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Test results for Dell Poweredge R770 with NVMe drives

    Scheduled Pinned Locked Moved Hardware
    26 Posts 7 Posters 1.9k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      I think it's related to a Broadcom NIC issue that we identified. @stormi will be able to give a test ISO to verify this hypothesis 🙂

      P 1 Reply Last reply Reply Quote 0
      • M Offline
        mik
        last edited by

        in my first XCP-ng test server, I have this Broadcom network adapters (Dell R760):

        a) Broadcom NetXtreme Gigabit Ethernet (BCM5720)
        b) Broadcom Adv. Quad 25Gb Ethernet
        c) Broadcom Gigabit Ethernet BCM5719
        d) Broadcom BCM57504 4x25G SFP28 PCIE
        e) Broadcom Gigabit Ethernet BCM5719

        adapters at point a) ad b) are integrated (but I can try to disable them using BIOS),
        the other adapters can be disabled also extracting them (PCI).

        if useful I can do that and identify which one is causing 8.3 installation boot stuck

        1 Reply Last reply Reply Quote 0
        • P Offline
          Pilow @olivierlambert
          last edited by

          @olivierlambert eager to test this new ISO, we have two XCP clusters in 8.2 that need upgrading in 8.3 with these cards :

          • BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller

          Do you think we would be impacted ?

          These same servers also have I350 Gigabit Network Connectioncard (quad port)

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            I think it's only the Broadcom NIC

            1 Reply Last reply Reply Quote 0
            • A Offline
              acebmxer
              last edited by

              FYI we have the Broadcom NetXtreme Gigabit Ethernet (BCM5720) in Dell R660 and Broadcom Adv. Dual 10GBASE-T Ethernet in Dell r640.

              Currently have no issues with ether nic in XCP-NG 8.3

              1 Reply Last reply Reply Quote 0
              • M Offline
                mik
                last edited by

                This post is deleted!
                1 Reply Last reply Reply Quote 0
                • M Offline
                  mik
                  last edited by

                  sorry for unwanted bold characters in previous post

                  I do confirm, 8.3 LTS installation boot sequence hungs with these two adapters:

                  Broadcom 57504 25G SFP28 Quad Port Adapter OCP 3.0 NIC
                  firmware version: 23.31.18.10 (Dell p/n 540-BCRX)

                  Broadcom 57504 Quad Port 10/25GbE SFP28 Adapter, PCIe Full Height
                  firmware version: 23.31.18.10 (Dell p/n 540-BDCZ)

                  Leaving enabled only these ones, we have NO HUNGS during 8.3 boot:

                  Broadcom 5720 Dual Port 1GbE LOM
                  firmware version: 23.31.1 (Dell p/n 540-BDKD)

                  Broadcom 5719 Quad Port 1GbE BASE-T Adapter PCIe Low Profile V2
                  firmware version: 23.31.1 (Dell p/n 540-BDRK)----

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Yes, that's why our new ISO coming soon will get a more recent version of the driver, fixing the problem 🙂 Hang on!

                    P 1 Reply Last reply Reply Quote 0
                    • M Offline
                      mik
                      last edited by

                      many thanks, to start our evaluation XCP-ng + XOA we wait new ISO

                      XCP-ng maximum hardware nic quantity (eth ports) per host is 16 ...correct ?

                      1 Reply Last reply Reply Quote 0
                      • P Offline
                        Pilow @olivierlambert
                        last edited by

                        @olivierlambert said in Test results for Dell Poweredge R770 with NVMe drives:

                        Hang on!

                        no pun intended ? 😃

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          No pun, just wait for everyone to get back from our yearly Vates internal event 😉

                          1 Reply Last reply Reply Quote 0
                          • M Offline
                            MarcVdS
                            last edited by

                            Hi,
                            I'm having the exact same issue.
                            Dell R650xs
                            BOSS with M2 in RAID1
                            Broadcom 25Gbps NIC's

                            Glad to find out it's not an isolated issue. All is working fine in 8.2.1 in production for almost 2 years now, but of course 8.2.1 is out of service so I'm hoping the new ISO will be out soon.

                            thanks,
                            Marc

                            DanpD 1 Reply Last reply Reply Quote 0
                            • DanpD Offline
                              Danp Pro Support Team @MarcVdS
                              last edited by

                              @MarcVdS The new ISO is already available. See this post for more details.

                              M 1 Reply Last reply Reply Quote 0
                              • M Offline
                                MarcVdS @Danp
                                last edited by

                                @Danp
                                OK thx, will test that right away!

                                1 Reply Last reply Reply Quote 1
                                • Y Offline
                                  yllar
                                  last edited by yllar

                                  Tested the newest iso (xcp-ng-8.3.0-20250606.2.iso) with Dell R770 and Broadcom 57508 100GbE QSFP56 Dual Port Adapter, PCIe Full Height

                                  It still fails to recognize the NIC, relevant log lines:

                                  [ 57.592461] Broadcom NetXtreme-C/E/S driver bnxt_en v1.10.3-232.0.155.5+
                                  [ 61.400879] bnxt_en 0001:5c:00.0 (unnamed net_device) (uninitialized): Firmware not responding, rc: -16 status: 0x8000
                                  [ 61.421312] bnxt_en: probe of 0001:5c:00.0 failed with error -16

                                  By the way, it works ok with 8.2.1 iso

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    That's interesting 🤔 Is your firmware fully up to date on the NIC?

                                    Y 1 Reply Last reply Reply Quote 0
                                    • Y Offline
                                      yllar @olivierlambert
                                      last edited by

                                      @olivierlambert Yes, the NIC firmware is the latest.
                                      Strange that it works with 8.2.1

                                      Y 1 Reply Last reply Reply Quote 0
                                      • Y Offline
                                        yllar @yllar
                                        last edited by

                                        Relevant lines in 8.2.1 where it succeeds:

                                        [ 12.242658] Broadcom NetXtreme-C/E driver bnxt_en v1.10.0-216.0.119.1
                                        [ 12.274753] bnxt_en 0001:5c:00.0 eth0: Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem 2affff010000, node addr 40:5b:7f:6e:22:00
                                        [ 12.274767] bnxt_en 0001:5c:00.0: 252.048 Gb/s available PCIe bandwidth (16 GT/s x16 link)
                                        [ 12.306588] bnxt_en 0001:5c:00.1 eth1: Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem 2affff000000, node addr 40:5b:7f:6e:22:01
                                        [ 12.306599] bnxt_en 0001:5c:00.1: 252.048 Gb/s available PCIe bandwidth (16 GT/s x16 link)
                                        [ 15.100023] bnxt_en 0001:5c:00.1 side-1659-eth1: renamed from eth1
                                        [ 15.117296] bnxt_en 0001:5c:00.0 side-3513-eth0: renamed from eth0
                                        [ 16.878049] bnxt_en 0001:5c:00.0 eth0: renamed from side-3513-eth0
                                        [ 16.970941] bnxt_en 0001:5c:00.1 eth1: renamed from side-1659-eth1
                                        [ 20.216480] bnxt_en 0001:5c:00.1 eth1: NIC Link is Up, 100000 Mbps full duplex, Flow control: ON - receive & transmit
                                        [ 20.216487] bnxt_en 0001:5c:00.1 eth1: FEC autoneg off encodings: None
                                        [ 20.734402] bnxt_en 0001:5c:00.0 eth0: NIC Link is Up, 100000 Mbps full duplex, Flow control: none
                                        [ 20.734409] bnxt_en 0001:5c:00.0 eth0: FEC autoneg off encodings: None

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Offline
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Not entirely strange, a driver version can make the difference.

                                          1 Reply Last reply Reply Quote 0
                                          • Y Offline
                                            yllar
                                            last edited by yllar

                                            Also works with alt driver (broadcom-bnxt-en-alt-1.10.3_231.0.162.0-1.xcpng8.2.x86_64) in 8.2.1:

                                            # yum install broadcom-bnxt-en-alt
# reboot

# modinfo bnxt_en
filename:       /lib/modules/4.19.0+1/override/bnxt_en.ko
version:        1.10.3-231.0.162.0
description:    Broadcom NetXtreme-C/E/S network driver
license:        GPL
srcversion:     533BB7E5866E52F63B9ACCB
alias:          pci:v000014E4d0000D800sv*sd*bc*sc*i*
...
alias:          pci:v000014E4d00001604sv*sd*bc*sc*i*
depends:        devlink,ipv6
retpoline:      Y
name:           bnxt_en
vermagic:       4.19.0+1 SMP mod_unload modversions 

                                            # dmesg|grep bnxt_en
[   15.047312] Broadcom NetXtreme-C/E/S driver bnxt_en v1.10.3-231.0.162.0
[   15.048046] bnxt_en 0001:5c:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 60 seconds, may trigger hung task watchdog
[   15.215453] bnxt_en 0001:5c:00.0 eth0: Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem 2affff010000, node addr 40:5b:7f:6e:22:00
[   15.215469] bnxt_en 0001:5c:00.0: 252.048 Gb/s available PCIe bandwidth (16 GT/s x16 link)
[   15.216606] bnxt_en 0001:5c:00.1 (unnamed net_device) (uninitialized): Device requests max timeout of 60 seconds, may trigger hung task watchdog
[   15.265361] bnxt_en 0001:5c:00.1 eth1: Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem 2affff000000, node addr 40:5b:7f:6e:22:01
[   15.265372] bnxt_en 0001:5c:00.1: 252.048 Gb/s available PCIe bandwidth (16 GT/s x16 link)
[   15.350702] bnxt_en 0001:5c:00.1 side-5183-eth1: renamed from eth1
[   15.385121] bnxt_en 0001:5c:00.0 side-3345-eth0: renamed from eth0
[   17.400142] bnxt_en 0001:5c:00.0 eth0: renamed from side-3345-eth0
[   17.493640] bnxt_en 0001:5c:00.1 eth1: renamed from side-5183-eth1
[   20.614447] bnxt_en 0001:5c:00.1 eth1: NIC Link is Up, 100000 Mbps (NRZ) full duplex, Flow control: none
[   20.614454] bnxt_en 0001:5c:00.1 eth1: FEC autoneg off encoding: None
[   21.113687] bnxt_en 0001:5c:00.0 eth0: NIC Link is Up, 100000 Mbps (NRZ) full duplex, Flow control: ON - receive & transmit
[   21.113694] bnxt_en 0001:5c:00.0 eth0: FEC autoneg off encoding: None
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post