XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    RTX A4000 GPU passthrough failure on XCP-ng

    Scheduled Pinned Locked Moved Hardware
    2 Posts 2 Posters 372 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • U Offline
      Unvocal2
      last edited by

      I’m encountering major issues getting PCIe GPU passthrough working on my homely setup:

      Host hardware & firmware
      • Dell PowerEdge R740 (dual Xeon Gold 6230R)
      • Single NVIDIA RTX A4000 GPU
      • BIOS settings confirmed: Virtualization enabled, IOMMU enabled, “Above 4G Decode” enabled
      • XCP-ng host (dom0) is not using the GPU; both functions (graphics + audio) are assigned to the VM

      VM Guest details
      • Guest OS: Arch Linux (also tested Windows)
      • VM config: Both PCI devices (GPU + HDMI/Audio) attached via passthrough
      • On Arch: kernel cmdline includes pcie_aspm=off nvidia.NVreg_EnableGpuFirmware=0
      • On host VM platform flags set: pci-msitranslate=true, pci-power_mgmt=false, device-model=qemu-upstream-compat, UEFI/OVMF enabled

      Symptoms
      • On Arch: repeated log entries such as:

      NVRM: gpuHandleSanityCheckRegReadError_GM107: Possible bad register read …
      NVRM: GSP failed to halt with GFW_BOOT …
      RmInitAdapter: Cannot initialize GSP firmware RM

      •	On Windows: Device Manager shows “Error 43” for the GPU.

      What I’ve done so far
      • Verified IOMMU groups; both functions isolated and passed through correctly
      • Checked FLR support for the GPU: GPU core supports FLR, audio function does not
      • Tried disabling ASPM/power management in host & guest
      • Tried older NVIDIA driver versions (including 510.xx branch)
      • Verified large BARs are present in guest lspci -vvv output
      • Uploaded full dmesg logs + BIOS dump from Redfish for review

      Attachments
      • dmesg_errors.log
      • BIOS dump from Redfish[link text]

      Request for help
      If you have successfully passed through an RTX A4000 on XCP-ng (to Linux or Windows guest), can you share:
      • XCP-ng version, guest OS, driver version
      • VM platform flags (especially any non-default settings)
      • Any custom vBIOS or device reboot/reset tweaks you used
      • Any additional steps you found necessary for stability

      I’m pretty much out of ideas at this point and would appreciate any working configurations or suggestions.
      dmesg_errors.txt
      BIOS_Current_Settings.txt

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hi,

        Are you sure your GPU is correctly fed via the power connector?

        1 Reply Last reply Reply Quote 0

        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

        With your input, this post could be even better 💗

        Register Login
        • First post
          Last post