AMD Radeon Vega M GH Passthrough

AmandaBeuno

Good day,

Apologies if this question was asked before. I tried searching but to no avail.

The Hades Canyon comes with both the Intel HD 630 graphics chipset along with AMD's Radeon Vega M GH one.

I have XCP-NG 8.0 installed and it seems that there are no drivers for the Radeon graphics as I cannot add it to be passed through to a VM.

I believe at this stage that the kernel needs the driver to be installed there first before being able to pass it through (please correct me if I am wrong).

I noticed that the same question was asked on Ubuntu forums here https://ubuntuforums.org/showthread.php?t=2400400&page=2 and I believe if it is there fore Ubuntu, it should be more-or-less available for a CentOS dervied XCP-NG, right ?

I would totally appreciate any pointers on how to get the AMD graphics passed through to VMs.

All the best,
Amanda

r1

PCI passthrough works like hiding that device from Dom0 (kernel) and being directly accessible to VM (kernel). You won't need a driver of it in Dom0 as Dom0 never sees it.

Make sure that the device is visible within VM via lspci or similar. Also I think, you'd need to enable VT-d in BIOS.

AmandaBeuno

@r1 Thank you so much for your reply. I really am glad that you are looking at this issue. Totally appreciate your help.

The thing is, I followed this walk-through to the T. https://github.com/xcp-ng/xcp/wiki/PCI-Passtrough

And I run

xl pci-assignable-list
0000:01:00.0   <- VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Polaris 22 XT [Radeon RX Vega M GH] (rev c0)
0000:01:00.1   <- Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Polaris 22 HDMI Audio

However, when I try to start the VM in question, I get the following error:

"Failed","Starting VM 'Wind'
Internal error: xenopsd internal error: Device.PCI.Cannot_add(_, _)
Time: 00:00:04","TestLab","Feb 3, 2020 1:34 PM"

Please let me know if there are any logs that I could add here to help figure out what the problem is.

All the best,
Amanda

amp88

@AmandaBeuno I had the same error as you're getting now ("Internal error: xenopsd internal error: Device.PCI.Cannot_add(_, _)" when attempting to perform PCI passthrough of an HBA card on a system which didn't have the required virtualisation feature (VT-d) enabled.

As r1 mentioned above, have you made sure that your system has virtualisation support enabled in the BIOS/UEFI? I've never used a Hades Canyon NUC before, but from this BIOS overview video it appears you want to enter the BIOS/UEFI at boot and then navigate to the Advanced -> Security tab. Then make sure under "Security Features" both "Intel Virtualization Technology" and "Intel VT for Directed I/O (VT-d)" are enabled.

Intel Hades Canyon NUC NUC8i7HVK BIOS Walkthrough (relevant section at 2:17, link is timestamped)

stormi

By the way, could someone edit https://github.com/xcp-ng/xcp/wiki/PCI-Passtrough to add information about the VT-d requirement?

amp88

@stormi said in AMD Radeon Vega M GH Passthrough:

By the way, could someone edit https://github.com/xcp-ng/xcp/wiki/PCI-Passtrough to add information about the VT-d requirement?

I was thinking about doing it, but I'm not sure whether it should be added to the guide as a "prerequisite" step at the beginning (e.g. check it's enabled before you begin the guide) or in a "troubleshooting" step at the end (e.g. if you experience this error, check it's enabled). Which approach is better probably depends on the ratio of systems where it's enabled by default, which I would believe/assume is relatively high with enterprise equipment and relatively low with consumer equipment.

stormi

I'd say prerequisite in that case. You can add that you think that it's probably already enabled in enterprise equipment.

amp88

@stormi said in AMD Radeon Vega M GH Passthrough:

I'd say prerequisite in that case. You can add that you think that it's probably already enabled in enterprise equipment.

OK, I've added a prerequisite section. I tried to keep the edit as concise as possible whilst still conveying the necessary information.

stormi

Looks very good to me, thanks!

imad2nsi

I have the same problem for my hades canyon nuc8i7hvk.
I followed the PCI Passtrough wiki, both Intel Virtualization and VT-d are enabled.
Tested multiples guest OS (kernel >= 4.18 support for AMD VEGA), the VM starts but as soon as I get output over HDMI every thing hangs including the Dom0.
After a hard reset, nothing related to passtrough in theses log files: SMlog, xensource.log and hypervisor.log.
The problem is explained in the Intel documentation (Link below page 63): "lack of Access Control Service (ACS)"
https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e3-1200v3-spec-update.pdf
There is a success story using Fedora and ACS patched kernel to change IOMMU groups.
Please is there a way to disable the kernel ACS check for XCP-ng ?

jtbw911

@imad2nsi This is starting to remind me of my 4-week foray into messing around with unRAID, coming out of it with the conclusion that it was more trouble than it was worth most of the time (with the various types of hardware I tried it on).

old_guy

@imad2nsi
Thanks for your research. I built a system around a Xeon E3-1226v3 with VT-D support for the express purpose of using PCI passthrough. Now I see I've wasted my time and money.

stormi

If I understand correctly, PCI pass-through is discouraged by Intel with this family of processors, but there's a way to make it work with kernel patches?