pandusen

@john.c
Oh, that is most certainly an AI response.. No wonder I was frustrated with the questions.
Why do you feel obligated to infuse this slop into a thread on a support forum, with no regard of it being helpful or not?

@john.c said:
also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

Yes I did among a ton of other diagnoses, a billion times.. I am talking about findings i did over a long period(months). I am not going to sit down and write a book about something I gave up on almost a month ago. I know what I need to get it working, but I could not get to that state in XCP-ng. I could however get there using Proxmox. But that was on another system though, the x99 6900k. I am running as bare metal now... The former(my current xcp-ng box) was on a ROMED8-2T Epyc Milan. I tested that bare metal on a USB, and it also worked.

Yes Above 4G has to be enabled otherwise reBar wont report a bar-0 or bar-1 at 16gb in linux, I got it to report that at one point (Sorry, I was not aware of the importence at that point, I just thought I had missed something, and was at a perfectly normal point in the process), but the guest driver still had issues(So it was no success anyway). All this led me to other findings, both here and in other discussions about people not succeeding with intel pass-through in xcp-ng. Again, I only mentioned those points, because I could see that Teddy, basically hadn't tested it beyond what I had already succeed at. "speculation" I am sry, but that was really I could get to.. I have searched everywhere, I have had Claud and GPT chew on it and looked through Intel's documention as well as my own hardware. The only thing I couldnt see, was what was happening under the xen hood.

I will look through my flyby notes and if I find something of interest, or perhaps in regards to getting that reBar to report 16gb on the guest, I will post it here!

My current Ubuntu bare metal, works as expected, I get a 16x8 on PCI v3 wich is expected from the 8 lanes electrical of the Arc pro b50. reBar is working and reporting correctly, and the performance numbers match what others have measured in a healthy system. Yes I have run plenty of heavy inference and compute.. mostly inference.. I have not used it for transcoding media yet, as that was a job it was suppose to have on the xcp-ng server.. It is even running a rtx 2080 card in the same box as well, which is running isolated along side the intel card and they are juggling data back and forth every day. No issues.

But as I also said, I might suspect PCI grouping of certain motherboards, not playing well with xcp-ng, which might explain your "Its working on my computer" phenomenon.

I am not currently at a point where I feel like taking my AI server apart and put the Intel card back in the xcp-ng server.. Maybe when something has actually changed, I would be willing to give it a shot again and I promise at that point, I will be sure to have some notes with me.

@MajorP93 said:

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

I’m not using the 1x1 as a definitive indicator, just listing it as one of the observed anomalies. Also, this is what the thread is about.

Yes, I ran workloads, but only in Windows. The drivers not working in Linux might or might not be from the audio device not being passed through.

Sure, it might not make sense, but these Arc architectures and drivers are a strange new breed. Some people have claimed, for instance, that the Windows drivers can fail to update the firmware if the audio device isn’t detected.

It’s all speculation. I’m not pointing fingers; I’m simply listing what I’ve observed. Nevertheless, all these points are abnormal and unusual compared to other platforms (AMD and NVIDIA).

@john.c

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

Could very well be John. Was fighting for months with this, so my memory is a bit mushy on that part, but i think I got it to map the full 16gb... I later moved my card to a bare metal sidecar, and things just worked. Even though I hacked the bios to activate reBar on an old x99 board.

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

I never got to a point of playing with transcoding for media nor AI Inferencing engines, as the tools and drivers never got stable. The drivers were aware of the card and loaded, but as soon as the compute stack tried to talk to it, the driver reset / crashed.

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

@stormi I am glad to hear there is progress, and I never doubted that there was interest and will from the Dev team, it was just stated, by Oliver himself, that the focus was mostly on the classic hypervisor features and that resources for edge cases were scarce. The numbers might not be spot on, but it was something similar.. Anyways I know you care and that the team always take interest in home-lab scenarios. And that, is why I keep recommending XCP-ng. You are good and motivated people, you care and interact with the community and you have a product that is simple in structure and free of most of the technical bureaucracy that the other ecosystems have.

And yes, this particular Intel Arc issue, is not only on XCP-ng, The intel toolbox and drivers for this platform is currently a mess for Linux.

And you are right to take home-labbers under your wing and prioritize it. As I'm sure you know, although it might not be of direct importance for revenue, they do have a huge impact on popularity. I dont think Proxmox would have had the traction it has, without them

Thank you for replying, and thank you to the team for their hard work!

@stormi I dont doubt that the team works hard. The GPU passthrough support has been of lower priority for a long time, it has been said many times before by the team and Oliver. 99 percent of the dev team is working security and basic hypervisor, and is not prioritized for edge cases. Which i guess that PCI passthrough is for GPUs Please dont ask me to dig up the sources. It has always been the case with Xen.... Proxmox, KVM and vmware has the edge there, and that should be well known. But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.

@JamesG Yeah, its scary that GPU passthrough still has a low priority for XCP-ng, during the boom of AI and local LLM Inferencing. I my self am starting to plan a move to Proxmox, though I do not really wish to move, cause I like XCP way better and have been with them for a long time... But that is just comfort...In the end, everything it comes down to is the capability and performance of the running VM guests.

@andriy.sultanov

@andriy.sultanov said:

@pandusen As Teddy said above, you can't passthrough a PCI bridge, so there's no PCI devices xapi shouldn't omit here.

I am not trying to pass through the bridge only the end points. The Intel arc's have 2 end points: The GPU and the Sound device. "xe pci-list" only reveals the GPU, not the sound device. (this works for nvidia and AMD)

But "going the xen-cmdline way" shouldn't break anything, that's what xe pci-disable-dom0-access does behind the scenes. What issues did you see? Which steps did you follow?

the sound device is available in the lspci list and can be passed through using CLI.
But doing so, (using CLI for passtrough) undoes everything done using xe or the passthrough gui in XO. and results in this:
https://xcp-ng.org/forum/topic/10609/xcp-ng-8.3-pci-passthrough-issue
so yes, its does break something.