XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    PCIe Pass-through lanes and lane performance

    Scheduled Pinned Locked Moved Compute
    39 Posts 12 Posters 4.6k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A Offline
      andriy.sultanov Vates 🪐 XAPI & Network Team @JamesG
      last edited by

      @JamesG it should be compatible, yes.

      you can check the current command-line value with /opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide and add all 81-84 devices with --set-dom0

      J 1 Reply Last reply Reply Quote 0
      • J Online
        JamesG @andriy.sultanov
        last edited by

        @andriy.sultanov First problem...

        /opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide
xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)

        That looks good...

        xl pci-assignable-list
libxl: error: libxl_pci.c:444:libxl_device_pci_assignable_list: Looks like pciback driver not loaded

        Not so much...

        I did go ahead and assign those PCI devices to the guest, but of course it won't start, complaining that the pciback driver isn't loaded.

        J 1 Reply Last reply Reply Quote 0
        • J Online
          JamesG @JamesG
          last edited by

          Nevermind...Typo...Rebooting now to see...

          J 1 Reply Last reply Reply Quote 0
          • J Online
            JamesG @JamesG
            last edited by

            Argh...

            Okay...First up:

            lspci:

            81:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01)
            82:01.0 PCI bridge: Intel Corporation Device e2f0
            82:02.0 PCI bridge: Intel Corporation Device e2f1
            83:00.0 VGA compatible controller: Intel Corporation Device e212
            84:00.0 Audio device: Intel Corporation Device e2f7

            Execute the hide:

            /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82:01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)"

            Reboot and verify the hide:

            xl pci-assignable-list
            0000:84:00.0
            0000:82:01.0
            0000:83:00.0
            0000:81:00.0
            0000:82:02.0

            Looks good....make the assignment:

            xe vm-param-set other-config:pci=0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0 uuid=9d76f849-16bf-0414-9072-6904176bbb01

            Verify:

            xe vm-param-list uuid=9d76f849-16bf-0414-9072-6904176bbb01

            other-config (MRW): pci: 0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0; import_task: OpaqueRef:9e15d1aa-5e79-79d9-af81-4dc299725217; xo:9d76f849: {"creation":{"date":"2025-12-16T19:19:14.724Z","template":"1f8728f9-6354-4a68-adc3-222c34f2bb94","user":"ae9cf9c9-bbcc-4dc9-b1bb-d9fb1b657c60"}}; base_template_name: Windows 11; mac_seed: 5891089b-e9ec-3050-30e5-564db3e6e170; install-methods: cdrom

            Starting up the VM results in this:

              "params": [
    "xenopsd internal error: Cannot_add(0000:82:02.0, Device_common.QMP_Error(1, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000006-1\\\"}\"))"
  ],

            I'm gonna strip it all back, reboot and start over.

            J 1 Reply Last reply Reply Quote 0
            • J Online
              JamesG @JamesG
              last edited by

              Yep...

              Stripped it all out....Rebooted...

              Set the pcihideback for dom0...Rebooted.

              Verified all devices were assignable.

              Assigned to guest VM, booted...No go.

              Initially the log said it couldn't assign 82:02.0
              So I removed it.
              Then it couldn't assign 82:01.0
              So I removed it.
              Then it couldn't assign 81:00.0
              So I removed it.

              Now the VM works...But we're effectively back to just passing through the GPU itself, and I guess the audio device.

              No functional change.

              TeddyAstieT 1 Reply Last reply Reply Quote 0
              • TeddyAstieT Online
                TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @JamesG
                last edited by

                @JamesG

                You can't PCI Passthrough a PCI bridge (and even if you could, it wouldn't be very useful), as it's a part of PCI internals and not really something the guest requires to use the device.

                As I asked previously, is there any issue (aside display like PCIe 1.0 1x) with the device, is it behavior correctly or having the expected performance ?

                J 1 Reply Last reply Reply Quote 0
                • J Online
                  JamesG @TeddyAstie
                  last edited by

                  @TeddyAstie From what I've tried...It won't work at all in Linux.

                  It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least.

                  I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs.

                  Side tangent...
                  I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere.

                  1 Reply Last reply Reply Quote 0
                  • pandusenP Offline
                    pandusen @andriy.sultanov
                    last edited by pandusen

                    @andriy.sultanov said:
                    I'll try to see why other devices are omitted and how this can be fixed.

                    Any news on this?
                    I have been fighting with getting an Intel Arc Pro B50 to work on Linux for months and i suspect that, the fact that the audio device isn't present, is what makes Linux fail to load the driver correctly.

                    Going the xen-cmdline way, seams to disable xe passthrough both in command line and in XO gui, causing all kinds of issues.

                    Windows seems to be able to initialise the GPU driver without the audio device, but it is in some kind of borked state. You can send workload to the GPU but the drivers are complaining about "no-pci compatibility".

                    XCP-ng 8.3 (latest patches)
                    Fedora server 44 - Kernel 7.1.5-201
                    on latest Arc firmware as well.

                    A 1 Reply Last reply Reply Quote 0
                    • A Offline
                      andriy.sultanov Vates 🪐 XAPI & Network Team @pandusen
                      last edited by

                      @pandusen As Teddy said above, you can't passthrough a PCI bridge, so there's no PCI devices xapi shouldn't omit here.

                      But "going the xen-cmdline way" shouldn't break anything, that's what xe pci-disable-dom0-access does behind the scenes. What issues did you see? Which steps did you follow?

                      J 1 Reply Last reply Reply Quote 0
                      • J Online
                        JamesG @andriy.sultanov
                        last edited by

                        @andriy.sultanov It's possible that this is just a fundamental issue with trying to use (maybe just Battlemage) Intel GPU's in XCP-ng.

                        The card itself appears to be a PCIe bridge in front of the actual GPU hardware on the card. If the driver on the guest requires "the whole thing" sent to it, and you can't pass-through the whole bridge and everything behind it, then there's no real way to use these GPU's in a guest on XCP-ng.

                        The lack of recent kernel or sufficient backports in the XCP-ng kernel, means the SRIOV features of Intel GPU's are lost as well.

                        All of this stuff works in Proxmox with seemingly little issues.

                        James

                        1 Reply Last reply Reply Quote 0

                        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                        With your input, this post could be even better 💗

                        Register Login
                        • First post
                          Last post