XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    PCIe NIC passing through - dropped packets, no communication

    Scheduled Pinned Locked Moved Hardware
    8 Posts 3 Posters 492 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • I
      icompit
      last edited by

      I'm looking for help with setting up PCIe NIC passthrough to VM. I tried with several Linux distributions on clean XCP-NG 8.2 and 8.3 and with different 10GbE network cards (Intel/Emulex/Mellanox).

      I've used Supermicro and Gigabyte server class motherboard and Xeon and Ryzen CPUs.

      The outcome is the same in each case. The device is nicely passed to the VM it appears correctly and except MSI and MSI-X interrupt related errors displayed during boot, the device seems to be operational at guest OS.

      The main problem is that I can't communicate through those interfaces.
      MAC addresses are visible on the switch but all RX packets are dropped, at least this is reported in interfaces counters on guest OS.

      Additionally despite the fact the I hided NIC for dom0 there are still bridge interfaces related with those passthrough interfaces operational and their MAC addresses are also visible on switch side.

      Can someone help me here?
      THX in advance for sharing some of your wisdom.

      D 1 Reply Last reply Reply Quote 0
      • D
        DustinB @icompit
        last edited by

        @icompit Are you trying to pass this card through to the VM so there is a physical card assigned to this VM?

        I 1 Reply Last reply Reply Quote 0
        • I
          icompit @DustinB
          last edited by

          @DustinB

          From the beginning:

          • I have bare metal system with 2 NICs on motherboard and 2 NICs installed in PCIe slots
          • I have installed XCP-NG on top of this hardware and I time to have 2 NICs from motherboard in use for hypervisor and two from PCIe slots passed through to VM.

          I'm using lspci -D to find devices which I need to passthrough at XCP-NG console

          0000:01:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
          0000:03:00.0 Non-Volatile memory controller: Sandisk Corp Device 5041 (rev 01)
          0000:04:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
          0000:06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
          0000:06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

          and with command:

          /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:01:00.0)(0000:04:00.0)"

          I prepare them to passthrough.

          After reboot I check does the devices are ready:

          *[00:12 xs-node0 ~]# xl pic-assignable-list
          0000:01:00.0
          0000:04:00.0 *

          So from this side all looks OK.

          Now I'm adding those devices to VM with command:

          xe vm-param-set other-config:pci=0/0000:01:00.0,0/0000:04:00.0 uuid=xxxx

          I can see at XO that both cards are configured for the VM

          10c50f6f-1fc2-43c3-b824-03002741acc5-image.png

          and when I start the VM I can see the cards:

          643ff2eb-1401-42f8-930f-9a8311e5c6a2-image.png

          f85ea65d-261e-4c1b-b547-81dbf6c4e05a-image.png

          ed344301-c9ad-4383-b294-3bcf876c8253-image.png

          Both links are UP and at this point all looks OK.

          dfa02d14-8d2a-43fb-914d-728ab79c1463-image.png

          Unfortunate when I try to ping the host which is on the same network to which cards are connected I get:

          e4440a57-d8e2-44fc-8934-467696999d1c-image.png

          When I try to ping the TX counter on VM interface increase... but seems that package doesn't leave the box because RX counter doesn't increase on switch side.

          I can add that Management agent 7.30.0-2 has been installed on VM.

          5599c7f4-4e2a-4f51-aa38-d34f28ada91c-image.png

          The 3rd interface which is configured on this VM works OK.

          cc28efdc-b933-423f-955d-947341ede5c0-image.png

          I 1 Reply Last reply Reply Quote 0
          • I
            icompit @icompit
            last edited by

            I've made clean install of xcp-ng 8.2.1 and applied all the patches available to this day again to collect all the details.

            d1bdbcaf-b439-4c83-908e-e76f11d0599b-image.png

            After installation:
            711f8898-9d68-4e3c-a9ff-4ee97a485cd9-image.png

            98d701ba-4b2d-4e6f-8960-df2e6e4b71d6-image.png

            5f1e65bf-0373-447c-a682-351e6a7105b5-image.png

            4ddb87f2-7fc5-4eb7-b1c7-a7579d2d435f-image.png

            After reboot:
            1b532ff3-87ce-4f91-b34b-55bf7565a5a7-image.png

            NIC are hidden from dom0, the interfaces eth2 and eth3 are not visible but the xenbr2 and xenbr3 are still there and are operational.

            8e1d71be-cb8f-4a1c-bc5a-9f1b06b2edc2-image.png

            d9c36a31-2589-4506-8622-644f5be128e0-image.png

            3626bee2-936f-4c2f-a094-76c5e7002be6-image.png

            This is a bit strange for me as I thought those interfaces should be not present and this should lead that both xenbr interfaces at lest should be down... without operational connection.

            I see the MAC addresses fot those interfaces on switch side.

            54cf01f0-31f7-4148-87cd-d8a956d1ccc3-image.png

            e1c6ab92-1598-405b-a025-327a6ecaf9a7-image.png

            Anyway both interfaces are available for passthrough.

            ad235ded-57e9-4610-b947-4b27d8060047-image.png

            I've bonded eth1 and eth2 before setting up the VM target for NIC for passthrough.

            9a4f9d11-2ff7-4024-83d6-d4f814f4f0bf-image.png

            VM installation:

            abbdc474-2848-4ade-905b-ce82ebc90848-image.png

            Before I start the VM and proceed with installation I will add those cards.

            39041e19-e02b-4488-bd91-ec331e3e055c-image.png |

            I see at XO that that NICs are assigned for the VM.

            ced29b54-d428-4d08-ac6f-a9c1d6fe97d3-image.png

            During installation cards are available:
            a7fd0997-d384-41ab-9551-a7c8b9fda77b-image.png

            But what is worth to mention on switch side apart from MAC of card I'm starting to see also MAC from xenbr2 and xenbr3 interfaces.

            7930f48e-68ec-452d-9cd8-efca5e37c59c-image.png

            67cf3851-32e4-4300-8978-add35e316692-image.png

            After starting VM:

            faef3885-5dc8-4313-b995-f5e76a4f674b-image.png

            4572dbf4-b581-4138-ba95-ebb50a021f78-image.png

            8528ea86-d1d1-4a70-9261-03b9f830c43a-image.png

            bd58b4f8-2448-4ef6-be10-9482a2747e4c-image.png

            When I start ping to the host in the network connected to the passthrough NIC on switch side I can see only:

            12b70a0a-3dba-48b2-96df-8b6dfef5dbbb-image.png

            No packet received on VM level, counters doesn't increase.

            Is there something which I miss to configure on hypervisor level???

            I 1 Reply Last reply Reply Quote 0
            • I
              icompit @icompit
              last edited by

              Problem solved with adding the kernel parameter wit command:

              /opt/xensource/libexec/xen-cmdline --set-dom0 "pci=realloc=on"

              1 Reply Last reply Reply Quote 0
              • olivierlambertO
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by olivierlambert

                @icompit said in PCIe NIC passing through - dropped packets, no communication:

                --set-dom0 "pci=realloc=on"

                Did you had a message in xl dmesg to put you in that right direction? I'm curious about this workaround, first time I heard it. Nice catch 🙂

                I 1 Reply Last reply Reply Quote 0
                • I
                  icompit @olivierlambert
                  last edited by

                  @olivierlambert

                  No, the logs from hypervisor as well as from dom0 didn't lead me to this.

                  Logs from VM OS while trying use those NICs generated some logs which after checking on the Internet lead to RHEL support case where adding this parameter to kernel was a confirmed solution. At XCP-NG forum there was also a thread which mentioned this option for passthrough device but it was related with GPU if I recall correctly.

                  Anyway this has helped me with Intel NICs.

                  Still no luck with Marvel FastlinQ NICs but there the problem is more related with driver included in kernel on guest OS. I'm still trying to solve it.

                  I didn't test this yet with Mellanox cards.

                  I'm trying to squeeze my lab environment as much as I can to save on electricity... the prices went crazy nowadays.

                  I'm trying to use StarWind VSAN (free license for 3 nodes and HA storage) and those NICs with SATA controller and NVMe ssd I needed to passthrough to VM. Server nodes are so powerful today so even single socket server can handle quite a lot..

                  I did some tests today with two LUNs presented over multi path iSCSI and this may work quite nice.

                  One instance of StarWind VSAN on one XCP-NG node, one on second and communication over dedicated NICs.. HA and replication on interfaces connected directly between StarWind nodes without switch.

                  I don't need much in terms of computing power and storage spece.. It might be perfect solution of two node cluster setups which may fit many of small customers or labs as my.
                  Of course this require more testing... but my lab is perfect place to experiment 🙂

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by olivierlambert

                    I quickly discussed with Xen developers and they are curious about your issue. Can you do a lspci -vvvv in your Dom0, first while you have still the option pci=realloc, and once without it, and see if we can spot any difference regarding your NIC. This might be helpful 🙂 (you can provide both outputs at https://paste.vates.tech)

                    1 Reply Last reply Reply Quote 0
                    • FionnF Fionn referenced this topic
                    • First post
                      Last post