JamesG

@TeddyAstie

Thought...If you want access to a server with a B50, I think I could setup VPN access and put a host online with IPMI access for you to do whatever you want with.

You will have an AMD Epyc 7302 with 64GB of ram some flash and HDD storage, and an Intel B50. PM me and we can work out the details of what you need to experiment with, and I can setup some remote access for you to get into the system.

I will gladly work with you guys any which way I can to move this forward.

@TeddyAstie I think there's alot of interest in the Intel platform because of the lack of ridiculous licensing for virtualization features. The problem is that you need a pretty current 6.x kernel to use them and XCP-ng/XenServer is just stuck on a really aging kernel right now.

As exposed in this thread, the PCIe pass-through isn't really working properly under XCP-ng.

Even if pass-through worked, that's kinda lame because the real purpose would be to chop up the GPU into SR-IOV instances so multiple workloads could have a GPU.

The Proxmox community is already rockin' and rollin' with this. I watched a video a couple of days ago where a B70 was chopped up into Qty.4, 8GB GPU's via SR-IOV and passed to clients.

No ridiculous mega-$$$$ licensing. It just worked.

@andriy.sultanov It's possible that this is just a fundamental issue with trying to use (maybe just Battlemage) Intel GPU's in XCP-ng.

The card itself appears to be a PCIe bridge in front of the actual GPU hardware on the card. If the driver on the guest requires "the whole thing" sent to it, and you can't pass-through the whole bridge and everything behind it, then there's no real way to use these GPU's in a guest on XCP-ng.

The lack of recent kernel or sufficient backports in the XCP-ng kernel, means the SRIOV features of Intel GPU's are lost as well.

All of this stuff works in Proxmox with seemingly little issues.

James

@TeddyAstie From what I've tried...It won't work at all in Linux.

It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least.

I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs.

Side tangent...
I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere.

Yep...

Stripped it all out....Rebooted...

Set the pcihideback for dom0...Rebooted.

Verified all devices were assignable.

Assigned to guest VM, booted...No go.

Initially the log said it couldn't assign 82:02.0
So I removed it.
Then it couldn't assign 82:01.0
So I removed it.
Then it couldn't assign 81:00.0
So I removed it.

Now the VM works...But we're effectively back to just passing through the GPU itself, and I guess the audio device.

No functional change.

Argh...

Okay...First up:

lspci:

81:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01)
82:01.0 PCI bridge: Intel Corporation Device e2f0
82:02.0 PCI bridge: Intel Corporation Device e2f1
83:00.0 VGA compatible controller: Intel Corporation Device e212
84:00.0 Audio device: Intel Corporation Device e2f7

Execute the hide:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82:01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)"

Reboot and verify the hide:

xl pci-assignable-list
0000:84:00.0
0000:82:01.0
0000:83:00.0
0000:81:00.0
0000:82:02.0

Looks good....make the assignment:

xe vm-param-set other-config:pci=0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0 uuid=9d76f849-16bf-0414-9072-6904176bbb01

Verify:

xe vm-param-list uuid=9d76f849-16bf-0414-9072-6904176bbb01

other-config (MRW): pci: 0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0; import_task: OpaqueRef:9e15d1aa-5e79-79d9-af81-4dc299725217; xo:9d76f849: {"creation":{"date":"2025-12-16T19:19:14.724Z","template":"1f8728f9-6354-4a68-adc3-222c34f2bb94","user":"ae9cf9c9-bbcc-4dc9-b1bb-d9fb1b657c60"}}; base_template_name: Windows 11; mac_seed: 5891089b-e9ec-3050-30e5-564db3e6e170; install-methods: cdrom

Starting up the VM results in this:

  "params": [
    "xenopsd internal error: Cannot_add(0000:82:02.0, Device_common.QMP_Error(1, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000006-1\\\"}\"))"
  ],

I'm gonna strip it all back, reboot and start over.

Nevermind...Typo...Rebooting now to see...

@andriy.sultanov First problem...

/opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide
xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)

That looks good...

xl pci-assignable-list
libxl: error: libxl_pci.c:444:libxl_device_pci_assignable_list: Looks like pciback driver not loaded

Not so much...

I did go ahead and assign those PCI devices to the guest, but of course it won't start, complaining that the pciback driver isn't loaded.

@andriy.sultanov Absolutely. I wasn't sure if the old way was incompatible with the new way or not.

So the old 8.2 method will still work without conflict in 8.3??

If so, I'll have an answer for you shortly.

@MajorP93 I'm pretty sure you can migrate between dissimilar pools as long as the CPU architecture is the same. You can't go from Pool A with AMD to Pool B with Intel, but you should be able to go from Pool A AMD to Pool B AMD with the caveat that the CPU generation of the destination pool has to be equal or beyond that of the source pool.

If I understand it right, Tackyone wants to dumb down all VM's to have the least common denominator CPU instructions so he can migrate any guest to any host. I personally think that's a bad direction, but I understand his desire...Pool a bunch of random systems laying around into a common farm. I think that's a bit too low-end of a goal for Vates to spend any time on myself. That seems exclusively homelab to me and counter to moving forward in the industry.