09:00.0 "Multimedia video controller [0400]" "Conexant Systems, Inc. [14f1]" "CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb [8880]" -r04 "Hauppauge computer works Inc. [0070]" "Device [6a28]"
0a:00.0 "Multimedia video controller [0400]" "Conexant Systems, Inc. [14f1]" "CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb [8880]" -r04 "Hauppauge computer works Inc. [0070]" "Device [6b28]"
My issue isn't that I can't pass through the card at all, I can still do it via CLI just fine so there isn't really any real problem preventing me from hiding the card from dom0 and then attaching it to my Plex VM. I was just curious why some devices are listed with xe pci-list and others aren't because I would have thought that it would essentially just reflect whatever shows in lspci.
Having it show in xe pci-list would just mean it's easier to attach/detach from a VM as it would then show in the webUI.
Does XCP-NG require the built in driver support to list the devices in pci-list?? Because the TV tuner doesn't work natively on Linux without downloading the drivers so I guess that could be a possibility?
Correct. The webUI seems to reflect whatever is in xe pci-list, but what appears in this list seems somewhat random when compared to the full lspci list.
Perhaps xe pci-list could be made to be a bit more broad in what it can detect?
I have a TV tuner card I use with my Plex Media Server.
I've successfully used the command line to pass this device through to the VM and it works fine, but recently I upgraded to XCP-NG 8.3 and saw that there is now new options in XO to make pass through much easier so I figured I'd give it a try.
Unfortunately it appears that not all PCI devices are listed including my TV Tuner card.
Is there a reason some devices are detected and others aren't?
lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 7
01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
05:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2000] (rev a1)
05:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 16)
07:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
08:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
08:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
09:00.0 Multimedia video controller: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb (rev 04)
0a:00.0 Multimedia video controller: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb (rev 04)
0b:00.0 Serial controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller
0b:00.1 Serial controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller
0b:00.2 Parallel controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
0c:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0e:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0f:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. FURY Renegade NVMe SSD with heatsink (rev 01)
10:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c8)
10:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller
10:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
10:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
10:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
10:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
11:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
11:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81
xe pci-list
uuid ( RO) : bcc76d1d-c85d-67c1-dfc4-83ece0267ade
vendor-name ( RO): Realtek Semiconductor Co., Ltd.
device-name ( RO): RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
pci-id ( RO): 0000:06:00.0
uuid ( RO) : 29611fb5-e276-5de1-6cf1-6e1ecdcac271
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD/ATI]
device-name ( RO): Cezanne [Radeon Vega Series / Radeon Vega Mobile Series]
pci-id ( RO): 0000:10:00.0
uuid ( RO) : c0c7d8d9-ae57-7ccd-ee29-d6e6a999d86d
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): FCH SATA Controller [AHCI mode]
pci-id ( RO): 0000:11:00.1
uuid ( RO) : a90f2a27-f722-2371-9757-d920f20e2c8c
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): FCH SATA Controller [AHCI mode]
pci-id ( RO): 0000:11:00.0
uuid ( RO) : 316e2f1f-4684-76da-36d6-fe9d92a629ff
vendor-name ( RO): NVIDIA Corporation
device-name ( RO): GP106GL [Quadro P2000]
pci-id ( RO): 0000:05:00.0
uuid ( RO) : a96bf51a-9ab2-b1d3-1f9b-6861057ad6e9
vendor-name ( RO): NVIDIA Corporation
device-name ( RO): GP106 High Definition Audio Controller
pci-id ( RO): 0000:01:00.1
uuid ( RO) : b21b6fcf-14d2-6dfc-6bd0-930e55533c29
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): Family 17h/19h HD Audio Controller
pci-id ( RO): 0000:10:00.6
uuid ( RO) : 839b273a-6a05-bf48-ffd4-7f9f5abf14ae
vendor-name ( RO): NVIDIA Corporation
device-name ( RO): GP106 High Definition Audio Controller
pci-id ( RO): 0000:05:00.1
uuid ( RO) : b7823f9f-f222-fe1c-f7ef-0706e9b220fd
vendor-name ( RO): NVIDIA Corporation
device-name ( RO): GP106 [GeForce GTX 1060 6GB]
pci-id ( RO): 0000:01:00.0
uuid ( RO) : 7d357a0c-65b8-0289-e066-d7406bce1bef
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): Family 17h (Models 10h-1fh) Platform Security Processor
pci-id ( RO): 0000:10:00.2
uuid ( RO) : 100a234c-3a8d-9633-9c8b-6a8b477caa99
vendor-name ( RO): Kingston Technology Company, Inc.
device-name ( RO): FURY Renegade NVMe SSD with heatsink
pci-id ( RO): 0000:0f:00.0
uuid ( RO) : 134b5187-9de7-da61-f2b0-0041c08b1c27
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): Renoir/Cezanne USB 3.1
pci-id ( RO): 0000:10:00.3
uuid ( RO) : b41fd3a4-1494-3f73-ed50-74493289e01d
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): FCH SATA Controller [AHCI mode]
pci-id ( RO): 0000:0e:00.0
uuid ( RO) : bd91ee24-da09-aaef-51ec-a3c4c89ea72d
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): FCH SATA Controller [AHCI mode]
pci-id ( RO): 0000:0d:00.0
uuid ( RO) : 9df35a8c-1343-4353-9799-906a2c015479
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD]
device-name ( RO): Renoir/Cezanne USB 3.1
pci-id ( RO): 0000:10:00.4
uuid ( RO) : 823820df-a14c-ae47-37f2-7cb133496c02
vendor-name ( RO): Mellanox Technologies
device-name ( RO): MT27500 Family [ConnectX-3]
pci-id ( RO): 0000:04:00.0
uuid ( RO) : 655f0cf1-1b7a-63e2-5404-7860795ca712
vendor-name ( RO): Advanced Micro Devices, Inc. [AMD/ATI]
device-name ( RO): Renoir Radeon High Definition Audio Controller
pci-id ( RO): 0000:10:00.1
This is the device in question:
09:00.0 Multimedia video controller: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb (rev 04)
0a:00.0 Multimedia video controller: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb (rev 04)
I have managed to get a serial connection to the second machine with the following grub.cfg configuration.
serial --unit=4 --port=0xb030 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
if [ -s $prefix/grubenv ]; then
load_env
fi
if [ -n "$override_entry" ]; then
set default=$override_entry
fi
menuentry 'XCP-ng' {
search --label --set root root-fsidtl
multiboot2 /boot/xen.gz dom0_mem=4272M,max:4272M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=com5,115200 console=com5,vga vga=mode-0x0311
module2 /boot/vmlinuz-4.19-xen root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=ttyS4,115200 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
module2 /boot/initrd-4.19-xen.img
}
I believe the trick was adding console=ttyS4,115200
Unfortunately, I still don't get the grub boot selection to screen to show on reboot like it does on the first machine.
Can anyone explain why?
I had already tried changing COM1 to COM5 in the multiboot line. Having access to the boot loader could be useful if I ever had issues booting and needed to access the grub command line. Also tried changing COM5 to ttyS4 as well. Is there a specific serial= command I should use instead there?
G'day, can someone please help me before I go insane. I had already typed out a post and then when I went to post it asked me to confirm my email (despite posting before) which made me lose everything I typed which has made this even more frustrating after spending hours trying to get this working...
Ok, so I have 2 hypevisors.
I've managed to get serial connection working on it very easily in just a few minutes.
dmesg | grep tty
[ 0.000000] Command line: root=LABEL=root-pjvhsg ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:21:00.0)
[ 0.436953] Kernel command line: root=LABEL=root-pjvhsg ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:21:00.0)
[ 0.491878] console [tty0] enabled
All I needed to do was edit the grub.cfg file, reboot and it showed up straight away on my laptop connected via serial hub.
cat /boot/efi/EFI/xenserver/grub.cfg
serial --unit=0 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
if [ -s $prefix/grubenv ]; then
load_env
fi
if [ -n "$override_entry" ]; then
set default=$override_entry
fi
menuentry 'XCP-ng' {
search --label --set root root-pjvhsg
multiboot2 /boot/xen.gz dom0_mem=4096M,max:4096M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G com1=115200,8n1 console=com1,vga vga=mode-0x0311
module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pjvhsg ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:21:00.0)
module2 /boot/initrd-4.19-xen.img
}
This one is a bit different as it doesn't have any serial headers on the motherboard so I've bought a PCIe serial/parallel card.
lspci -v | grep -A 14 'Serial\|Parallel'
0b:00.0 Serial controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller (prog-if 02 [16550])
Subsystem: Asix Electronics Corporation (Wrong ID) Device 1000
Flags: fast devsel, IRQ 30
I/O ports at b030 [size=8]
Memory at 91f00000 (32-bit, non-prefetchable) [size=4K]
Memory at 91f01000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [800] Advanced Error Reporting
Kernel driver in use: serial
0b:00.1 Serial controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller (prog-if 02 [16550])
Subsystem: Asix Electronics Corporation (Wrong ID) Device 1000
Flags: fast devsel, IRQ 31
I/O ports at b020 [size=8]
Memory at 91f02000 (32-bit, non-prefetchable) [size=4K]
Memory at 91f03000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Legacy Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: serial
0b:00.2 Parallel controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller (prog-if 03 [IEEE1284])
Subsystem: Asix Electronics Corporation (Wrong ID) Device 2000
Flags: fast devsel, IRQ 11
I/O ports at b010 [disabled] [size=8]
I/O ports at b000 [disabled] [size=8]
Memory at 91f04000 (32-bit, non-prefetchable) [disabled] [size=4K]
Memory at 91f05000 (32-bit, non-prefetchable) [disabled] [size=4K]
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Legacy Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
dmesg | grep tty
[ 0.000000] Command line: root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
[ 0.770474] Kernel command line: root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
[ 0.837499] console [tty0] enabled
[ 6.010629] 0000:0b:00.0: ttyS4 at I/O 0xb030 (irq = 30, base_baud = 115200) is a ST16650V2
[ 6.033192] 0000:0b:00.1: ttyS5 at I/O 0xb020 (irq = 31, base_baud = 115200) is a ST16650V2
I've used screen to determine the connection between the serial hub and machine is ok.
screen /dev/ttyS5 115200
I could see text I entered appear in the putty serial connection when I was typing via SSH so it appears the cable is ok, serial hub is ok and PCIe card is ok.
So I made the exact same changes to my grub.cfg (Slight differences in PCI settings as well as the ports used by COM# and ttyS#)
cat /boot/efi/EFI/xenserver/grub.cfg
serial --unit=5 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
if [ -s $prefix/grubenv ]; then
load_env
fi
if [ -n "$override_entry" ]; then
set default=$override_entry
fi
menuentry 'XCP-ng' {
search --label --set root root-fsidtl
multiboot2 /boot/xen.gz dom0_mem=4272M,max:4272M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G com6=115200,8n1 console=com6,vga vga=mode-0x0311
module2 /boot/vmlinuz-4.19-xen root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
module2 /boot/initrd-4.19-xen.img
}
Upon reset putty shows a single weird character as if it was trying to establish a connection or something but then I don't get any sort of output or input.
Please help.
Cheers.
I was importing a 700GB .VHD created with Microsoft Disk2VHD (Not XVHD).
This is coming from my NAS and only gigabit ethernet (Until more 10gbit cards arrive next week) so it was taking forever. XOA had estimated 21 hours or something. Which was fine, happy to leave it overnight. I left my PC on the tasks page to monitor the progress and left it there.
However at some point when the task was over 5% complete I came back to check the progress and found that the task had just completely disappeared.
I didn't have any sort of notification left on the screen as to what happened or why. If I check in previous tasks I don't see anything. If I check in Pool logs I don't see anything.
I figured maybe the progress bar was inaccurate and the task finished but when I checked too see if the disk was available it's nowhere to be found so it seems the import failed at some point.
Am I missing something? It seems pretty difficult to find the information in XOA and I have no idea what I'd search for in logs or previous tasks. I tried to search for "import" or "nfs" (since that's part of the SR name I was importing it to) but got no results.
Any tips?
EDIT - Pretty sure I understand what happened. Computer went to sleep and lost network connection instead of keeping the session alive. Copying from mapped NAS folder, transferring through XOA on my PC was adding too many jumps so I've just opted to copy the file via command line instead.
Still confused as to why there was no error message or anything under tasks or anything though. It seems as soon as you refresh the page everything disappears.
In the end the solution was so simple... I was up until 2am trying to fix this. Updating the firmware, messing up Grub on my XCP-NG installation and having to recover off an XCP-NG 8.2 backup, hours wasted troubleshooting the Mellanox firmware tools (During installation you have to enter your password multiple times and there's no prompt or anything - No idea how someone discovered that work around).
I saw lots of people mentioning adding pci=realloc=off to kernel boot parameters and I tried that without success.
Just for the lolz I figured what if I turned it on instead?
pci=realloc=on
dmesg | grep UAR
Can no longer see the UAR missing error from before... Seems promising...
lspci -k | grep -A 2 Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies Device 0055
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core
Kernel driver is in use now, before it wasn't.
Rescan PIF in XOA and I can see it! Shows connected at 10gbit, is able to get IP address via DHCP, etc.
Everything seems to be working...
Now I've just gotta test that all my other PCI-e devices are passing through ok with this option enabled.
@Anonabhar seems to be an issue with the machine itself.
Tested my gaming rig and my server.
They're both AMD X570 chipsets.
Loaded up Ubuntu 22.04 live USB on each system. On my gaming rig it worked straight away without issue. Connected DAC to switch and it reported a 10gbit connection. On the server it has the same UAR Missing error message in dmesg and didn't show up as a network interface but appeared in lspci just as before.
Either I'm missing a setting hidden away in the BIOS or there's some very weird hardware conflict or incompatibility.
I'm installing Ubuntu to a portal SSD now and then I'll try out the firmware tools and see if I have any luck.
I just edited my post as you posted. I just tried it on Windows 11 and it worked instantly. This particular card doesn't support Infiniband, it's ethernet only so I don't believe that should be relevant in this case. It's the CX311A EN model.
The problem seems to lie in the drivers on XCP-NG or in the firmware of the card.
I'm not sure if I should be upgrading or downgrading the firmware or as some people suggested editing the firmware and limiting some aspects of the card to try and get it working.
I saw some posts about PCI=REALLOC=OFF in Grub from the Red Hat community support page, but wouldn't I want this on for a hypervisor?
I think I'm getting a bit closer...
lspci -k
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies Device 0055
Kernel modules: mlx4_core
lsmod | grep mlx
mlx4_core 352256 0
devlink 77824 1 mlx4_core
dmesg | grep mlx
[ 7.304630] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 7.304646] mlx4_core: Initializing 0000:04:00.0
[ 7.304799] mlx4_core 0000:04:00.0: Missing UAR, aborting
Seems the driver is failing to load. Google suggests it could be caused by SR-IOV (pretty sure I've got all my virtualisation options on in the BIOS) or I might be able to play around with the grub configuration to get it working.
The reason I went with this card in particular was because of the form factor. In this machine I've got 2 graphics cards and a TV tuner card already. I have a 1X slot (which basically nothing will fit in) and the bottom X16 slot only runs at 4X (which the ConnectX3 is a 4x PCI-e 3.0 card).
Supposedly another possible fix is to edit the firmware and flash a firmware that limits or removes SR-IOV functions from the card.
EDIT - Took out of hypervisor and installed into my gaming rig. Different motherboard, but same chipset (AMD X570). Windows 11 picked up the card instantly without need for driver installation. Plugged DAC into a 10Gbit SFP+ Juniper switch and Windows reads 10Gbit speed.
So card is definitely working properly but XCP-NG 8.3 doesn't like it.
Hi all,
Recently bought a Mellanox ConnectX-3 CX311A from eBay.
Plugged it into my hypervisor XCP-NG 8.3, but can't seem to get the card to work. As far as I know this card is ethernet only so shouldn't require any flashing or anything like that.
I'm using a new DAC from FS.com from the card to the 10GB SFP+ port on my Juniper switch.
Card is detected:
lspci
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
lspci -v | grep Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies Device 0055
I deleted and re-created my PCI passthrough config
xl pci-assignable-list
0000:09:00.0
0000:05:00.1
0000:01:00.0
0000:0a:00.0
0000:05:00.0
0000:01:00.1
Tried PIF scan in GUI and console but when I check PIF list I only see the on board NIC
xe pif-scan host-uuid=ff0c8a58-0feb-4fe1-8cc1-556aad1f8c75
xe pif-list
uuid ( RO) : a902c9a0-77f2-ad66-9f69-814e9bdd6413
device ( RO): eth0
MAC ( RO): d8:5e:d3:2b:23:8b
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): 5e0c47e8-abbd-bc5b-02eb-83b838560f90
host-uuid ( RO): ff0c8a58-0feb-4fe1-8cc1-556aad1f8c75
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
link/ether d8:5e:d3:2b:23:8b brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 92:b5:0c:1e:3e:cc brd ff:ff:ff:ff:ff:ff
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether d8:5e:d3:2b:23:8b brd ff:ff:ff:ff:ff:ff
inet 192.168.1.13/24 brd 192.168.1.255 scope global dynamic xenbr0
valid_lft 81730sec preferred_lft 81730sec
5: vif1.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
7: vif2.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
9: vif3.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: vif4.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether d8:5e:d3:2b:23:8b txqueuelen 1000 (Ethernet)
RX packets 53756 bytes 20202528 (19.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 27516 bytes 6309909 (6.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 9556 bytes 5978566 (5.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9556 bytes 5978566 (5.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vif1.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 3859 bytes 336927 (329.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29344 bytes 9261592 (8.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vif2.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 4605 bytes 937140 (915.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 28965 bytes 8527803 (8.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vif3.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 9936 bytes 1319123 (1.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 33662 bytes 7927207 (7.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vif4.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 7851 bytes 1274556 (1.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 31981 bytes 8681713 (8.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.13 netmask 255.255.255.0 broadcast 192.168.1.255
ether d8:5e:d3:2b:23:8b txqueuelen 1000 (Ethernet)
RX packets 33215 bytes 5737003 (5.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6334 bytes 5412414 (5.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Any ideas?
@Niko1 Some motherboards updated their BIOS for Windows 11 compatibility which may have automatically turned on secure boot, etc, etc.
I can't see any other reason Linux ISO's would all of a sudden fail to boot.
Try the CLI method and see if that fixes it.
Those settings in XOA exist for vGPUs I believe, not for pass through.
I've managed to pass through an Nvidia Quadro P2000 as well as a TV Tuner to a Ubuntu VM for Plex transcoding and free to air TV.
I'm now trying to pass through a graphics card to a Windows VM for personal cloud gaming. After 2 days I've finally managed to get an ancient AMD 6870 to work! But I'd like to figure out why the other cards don't work.
System specs:
AMD Ryzen 5700G
Gigabyte X570 UD
64GB DDR4 RAM
Nvidia Quadro P2000
Hauppauge WinTV QuadHD
[plus a random graphics card]
According to Nvidia a GTX 760 SHOULD be able to be used in a virtual Windows environment since Nvidia driver 465. Upon installing the card and adding it to my Windows VM I can see the card in the device manager with the default Windows drivers installed and it seems to be functioning normally. However when I attempt to install the Nivdia drivers I then get the Code 43 issue. I tried multiple driver versions after 465 with no luck.
When I try the same in a Ubuntu VM it sees the card but it doesn't load the driver. Nvidia-SMI says no devices detected.
Eventually I gave up on the Nvidia card and decided to try an R9 270 from AMD, however with this card the host crashes when attempting to install the GPU drivers on Windows... Tried multiple driver versions but can't seem to keep the host stable when the card attempts to initialise.
I found similar threads on here mentioning their host would crash when trying to load GPU drivers:
https://xcp-ng.org/forum/topic/7012/host-crashes-error-43-nvidia-quadro-k2200-pcie-passthrough-to-vm
https://xcp-ng.org/forum/topic/5872/passed-through-gpu-crashes-host-during-driver-install
I've tried updating from XCP-NG 8.2 to 8.3 to see if that helped anything, but I get the same results as before.
I tried a Ubuntu VM with the R9 270 to see if it is just a Windows error, and upon start up is a warning message saying "Unable to locate a bios ROM" and then a fatal error. lspci shows the card and lshw -c video shows the card but it's "unclaimed".
Both cards never showed any output once XCP-NG had booted.
I was ready to completely give up but figured since I already have the chassis open I'll just toss in an old 6870 to see what happens.
Windows 10 immediately recognises the device in device manager rather than displaying as a "Basic Adapter", which isn't too surprising given its age. I installed the drivers, plugged in a monitor and bam, it was displaying the Windows desktop!
So out of 3 cards I managed to get only 1 working. I'd like to figure out WHY these other cards don't work as the entire point of this is to put in a more capable card in the near future, but obviously I'm not keen on buying a card that may not even work if I can only get 1/3 cards working.
Any ideas on what I can try to get this working more reliably with more modern graphics cards? The main difference I can think of between the cards is that the 6870 has 1GB VRAM while the GTX 760 and R2 270 both have 2GB VRAM. The Quadro P2000 has 5GB of VRAM, but being a workstation card perhaps Xen has better support for it.
Happy to provide any logs, etc, that would be useful.
==UPDATE==
My brother had a spare GTX 1060 6GB card. I put that in and it worked right away. Now able to Stream games from the VM to the TV, etc. Awesome! Still a mystery as why the other cards refused to work.
Unfortunately they don't offer a guide, just a mention of it working.
https://docs.citrix.com/en-us/citrix-hypervisor/graphics/hv-graphics-config.html
Citrix Hypervisor supports Intel’s virtual GPU (GVT-g), a graphics acceleration solution that requires no additional hardware. It uses the Intel Iris Pro feature embedded in certain Intel processors, and a standard Intel GPU driver installed within the VM.
However, they don't actually offer a guide on HOW to set this up besides offering some tips on how to setup BAR in the UEFI settings to change how many vGPU's it will support. I've enabled Above 4G Decoding, Re-size BAR Support and set the Aperture Size to 512MB but that doesn't seem to enable the vGPU options XCP-ng Center or XO. I'm assuming I need to enable it manually on the underlying OS before XCP-ng can see it.
Note that the Intel GPU pass through is GVT-d NOT GVT-g. If you use GVT-d (AKA, just your regular PCIe pass through) obviously then I wouldn't be able to use the GPU for the the XCP-ng console on the machine itself. I'm comfortable on how to set that up but I really need the console available on the hypervisor for my use case.
This video (in German) demonstrates setting it up in XCP-ng center.
Hi, I've got a machine with an Intel i9-10850K with has UHD Graphics 630 iGPU.
From doing a bit of reading it seems this should support vGPU support with Intel GVT-g.
I can find plenty of guides on how to enable this on other Linux distro and with other hypervisors and Citrix even mentions it in their own documentation but I can't actually find a guide on how to do this in Xen/XCP-ng.
Does XCP-ng have an easy method of enabling Intel GVT-g support or do I need to edit Grub config and load modules, etc, etc, like other guides suggest?
Eg: https://wiki.archlinux.org/title/Intel_GVT-g
Supposedly this would allow me to still utilise the iGPU for the XCP-ng console as well as share the iGPU to a Windows guest.
@TheFrisianClause I ended up selling my P400 and buying a P2000 which worked straight away. It's possible the P400 may work in future but I didn't want to wait until then.
Unfortunate waste of money, but I'm pretty happy with the setup now running 2 XCP-NG hosts now with Plex with the P2000 doing transcoding, game servers and other utilities.
Nice to see a bit of action in this thread after a few weeks. It seems both of use are using Ivy Bridge CPU's, I wonder if the issue is related to our CPU's or chipsets?
I notice in the Reddit thread you posted it mentions something about the E3-12xx CPU's:
IMPORTANT ADDITIONAL COMMANDS
You might need to add additional commands to this line, if the passthrough ends up failing. For example, if you're using a similar CPU as I am (Xeon E3-12xx series), which has horrible IOMMU grouping capabilities, and/or you are trying to passthrough a single GPU.
These additional commands essentially tell Proxmox not to utilize the GPUs present for itself, as well as helping to split each PCI device into its own IOMMU group. This is important because, if you try to use a GPU in say, IOMMU group 1, and group 1 also has your CPU grouped together for example, then your GPU passthrough will fail.
Is it possible that same GRUB config is required here? Sorry, I'm probably not much help when it comes to hardware troubleshooting on Linux.
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"
I don't have the faintest idea of what any of this does or if it could help.
Hi all, I'm hoping someone can help me.
Hardware specs:
VM: Ubuntu Server 20.04 LTS
What I was hoping to do was use PCI/GPU passthrough to pass through the GPU and TV Tuner to a Ubuntu server running Plex media server. I was able to get the TV tuner working and it could scan channels, etc, and the GPU was also detected by Ubuntu, however no matter what I did I could not get the latest Nvidia drivers working and some of the hardware details aren't complete.
I tried installing from the official Ubuntu PPA, the Nvidia PPA, I tried upgrading Ubuntu to the latest release 21.10. I tried 21.04. I tried the 20.04 LTS version. I tried older drivers like 430, 450, 460, 465 and 470 and no matter what I did I could not get the GPU to function correctly through XCP-NG.
I know there's no hardware problem as I have pretty much given up and it's currently doing hardware transcoding just fine on a Ubuntu 20.04 install on bare metal.
I'd be willing to give XCP-NG one last try if anyone can give me any suggestions as what could be wrong.
I'm not sure if it's the same thing, but I tried both passing through the PCI devices in the CLI as well as trying the method of selecting the Nvidia GPU from the list in XOA after I had disabled the GPU from the hypervisor. Neither method worked. I also tried assigning the GPU through XCP-NG center Windows program.
This screenshot shows a few different commands. You can see the error messages in dmesg as well as the GPU UUID and Video BIOS appear as blank when inspecting the device information under /proc.
