Try the CLI method and see if that fixes it.

Those settings in XOA exist for vGPUs I believe, not for pass through.

I have managed to get a serial connection to the second machine with the following grub.cfg configuration.

serial --unit=4 --port=0xb030 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
if [ -s $prefix/grubenv ]; then
        load_env
fi

if [ -n "$override_entry" ]; then
        set default=$override_entry
fi

menuentry 'XCP-ng' {
        search --label --set root root-fsidtl
        multiboot2 /boot/xen.gz dom0_mem=4272M,max:4272M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=com5,115200 console=com5,vga vga=mode-0x0311
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=ttyS4,115200 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
        module2 /boot/initrd-4.19-xen.img
}

I believe the trick was adding console=ttyS4,115200

Unfortunately, I still don't get the grub boot selection to screen to show on reboot like it does on the first machine.

Can anyone explain why?

I had already tried changing COM1 to COM5 in the multiboot line. Having access to the boot loader could be useful if I ever had issues booting and needed to access the grub command line. Also tried changing COM5 to ttyS4 as well. Is there a specific serial= command I should use instead there?

G'day, can someone please help me before I go insane. I had already typed out a post and then when I went to post it asked me to confirm my email (despite posting before) which made me lose everything I typed which has made this even more frustrating after spending hours trying to get this working...

Ok, so I have 2 hypevisors.

I've managed to get serial connection working on it very easily in just a few minutes.

Machine 1

dmesg | grep tty
[    0.000000] Command line: root=LABEL=root-pjvhsg ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:21:00.0)
[    0.436953] Kernel command line: root=LABEL=root-pjvhsg ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:21:00.0)
[    0.491878] console [tty0] enabled

All I needed to do was edit the grub.cfg file, reboot and it showed up straight away on my laptop connected via serial hub.

cat /boot/efi/EFI/xenserver/grub.cfg
serial --unit=0 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
if [ -s $prefix/grubenv ]; then
        load_env
fi

if [ -n "$override_entry" ]; then
        set default=$override_entry
fi

menuentry 'XCP-ng' {
        search --label --set root root-pjvhsg
        multiboot2 /boot/xen.gz dom0_mem=4096M,max:4096M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G com1=115200,8n1 console=com1,vga vga=mode-0x0311
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pjvhsg ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:21:00.0)
        module2 /boot/initrd-4.19-xen.img
}

Machine 2

This one is a bit different as it doesn't have any serial headers on the motherboard so I've bought a PCIe serial/parallel card.

lspci -v | grep -A 14 'Serial\|Parallel'
0b:00.0 Serial controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller (prog-if 02 [16550])
        Subsystem: Asix Electronics Corporation (Wrong ID) Device 1000
        Flags: fast devsel, IRQ 30
        I/O ports at b030 [size=8]
        Memory at 91f00000 (32-bit, non-prefetchable) [size=4K]
        Memory at 91f01000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [800] Advanced Error Reporting
        Kernel driver in use: serial

0b:00.1 Serial controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller (prog-if 02 [16550])
        Subsystem: Asix Electronics Corporation (Wrong ID) Device 1000
        Flags: fast devsel, IRQ 31
        I/O ports at b020 [size=8]
        Memory at 91f02000 (32-bit, non-prefetchable) [size=4K]
        Memory at 91f03000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: serial

0b:00.2 Parallel controller: MosChip Semiconductor Technology Ltd. PCIe 9912 Multi-I/O Controller (prog-if 03 [IEEE1284])
        Subsystem: Asix Electronics Corporation (Wrong ID) Device 2000
        Flags: fast devsel, IRQ 11
        I/O ports at b010 [disabled] [size=8]
        I/O ports at b000 [disabled] [size=8]
        Memory at 91f04000 (32-bit, non-prefetchable) [disabled] [size=4K]
        Memory at 91f05000 (32-bit, non-prefetchable) [disabled] [size=4K]
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting

dmesg | grep tty
[    0.000000] Command line: root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
[    0.770474] Kernel command line: root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
[    0.837499] console [tty0] enabled
[    6.010629] 0000:0b:00.0: ttyS4 at I/O 0xb030 (irq = 30, base_baud = 115200) is a ST16650V2
[    6.033192] 0000:0b:00.1: ttyS5 at I/O 0xb020 (irq = 31, base_baud = 115200) is a ST16650V2

I've used screen to determine the connection between the serial hub and machine is ok.

screen /dev/ttyS5 115200

I could see text I entered appear in the putty serial connection when I was typing via SSH so it appears the cable is ok, serial hub is ok and PCIe card is ok.

So I made the exact same changes to my grub.cfg (Slight differences in PCI settings as well as the ports used by COM# and ttyS#)

cat /boot/efi/EFI/xenserver/grub.cfg
serial --unit=5 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
if [ -s $prefix/grubenv ]; then
        load_env
fi

if [ -n "$override_entry" ]; then
        set default=$override_entry
fi

menuentry 'XCP-ng' {
        search --label --set root root-fsidtl
        multiboot2 /boot/xen.gz dom0_mem=4272M,max:4272M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G com6=115200,8n1 console=com6,vga vga=mode-0x0311
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-fsidtl ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:0a:00.0)(0000:09:00.0) pci=realloc=on
        module2 /boot/initrd-4.19-xen.img
}

Upon reset putty shows a single weird character as if it was trying to establish a connection or something but then I don't get any sort of output or input.

Please help.

Cheers.

I was importing a 700GB .VHD created with Microsoft Disk2VHD (Not XVHD).

This is coming from my NAS and only gigabit ethernet (Until more 10gbit cards arrive next week) so it was taking forever. XOA had estimated 21 hours or something. Which was fine, happy to leave it overnight. I left my PC on the tasks page to monitor the progress and left it there.

However at some point when the task was over 5% complete I came back to check the progress and found that the task had just completely disappeared.

I didn't have any sort of notification left on the screen as to what happened or why. If I check in previous tasks I don't see anything. If I check in Pool logs I don't see anything.

I figured maybe the progress bar was inaccurate and the task finished but when I checked too see if the disk was available it's nowhere to be found so it seems the import failed at some point.

Am I missing something? It seems pretty difficult to find the information in XOA and I have no idea what I'd search for in logs or previous tasks. I tried to search for "import" or "nfs" (since that's part of the SR name I was importing it to) but got no results.

Any tips?

EDIT - Pretty sure I understand what happened. Computer went to sleep and lost network connection instead of keeping the session alive. Copying from mapped NAS folder, transferring through XOA on my PC was adding too many jumps so I've just opted to copy the file via command line instead.

Still confused as to why there was no error message or anything under tasks or anything though. It seems as soon as you refresh the page everything disappears.

@olivierlambert

In the end the solution was so simple... I was up until 2am trying to fix this. Updating the firmware, messing up Grub on my XCP-NG installation and having to recover off an XCP-NG 8.2 backup, hours wasted troubleshooting the Mellanox firmware tools (During installation you have to enter your password multiple times and there's no prompt or anything - No idea how someone discovered that work around).

I saw lots of people mentioning adding pci=realloc=off to kernel boot parameters and I tried that without success.

Just for the lolz I figured what if I turned it on instead?

pci=realloc=on

dmesg | grep UAR

Can no longer see the UAR missing error from before... Seems promising...

lspci -k | grep -A 2 Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
      Subsystem: Mellanox Technologies Device 0055
      Kernel driver in use: mlx4_core
      Kernel modules: mlx4_core

Kernel driver is in use now, before it wasn't.

Rescan PIF in XOA and I can see it! Shows connected at 10gbit, is able to get IP address via DHCP, etc.

Everything seems to be working...

Now I've just gotta test that all my other PCI-e devices are passing through ok with this option enabled.

@Anonabhar seems to be an issue with the machine itself.

Tested my gaming rig and my server.

They're both AMD X570 chipsets.

Loaded up Ubuntu 22.04 live USB on each system. On my gaming rig it worked straight away without issue. Connected DAC to switch and it reported a 10gbit connection. On the server it has the same UAR Missing error message in dmesg and didn't show up as a network interface but appeared in lspci just as before.

Either I'm missing a setting hidden away in the BIOS or there's some very weird hardware conflict or incompatibility.

I'm installing Ubuntu to a portal SSD now and then I'll try out the firmware tools and see if I have any luck.

@Anonabhar

I just edited my post as you posted. I just tried it on Windows 11 and it worked instantly. This particular card doesn't support Infiniband, it's ethernet only so I don't believe that should be relevant in this case. It's the CX311A EN model.

The problem seems to lie in the drivers on XCP-NG or in the firmware of the card.

I'm not sure if I should be upgrading or downgrading the firmware or as some people suggested editing the firmware and limiting some aspects of the card to try and get it working.

I saw some posts about PCI=REALLOC=OFF in Grub from the Red Hat community support page, but wouldn't I want this on for a hypervisor?

I think I'm getting a bit closer...

lspci -k
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
        Subsystem: Mellanox Technologies Device 0055
        Kernel modules: mlx4_core

lsmod | grep mlx
mlx4_core             352256  0
devlink                77824  1 mlx4_core

dmesg | grep mlx
[    7.304630] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    7.304646] mlx4_core: Initializing 0000:04:00.0
[    7.304799] mlx4_core 0000:04:00.0: Missing UAR, aborting

Seems the driver is failing to load. Google suggests it could be caused by SR-IOV (pretty sure I've got all my virtualisation options on in the BIOS) or I might be able to play around with the grub configuration to get it working.

The reason I went with this card in particular was because of the form factor. In this machine I've got 2 graphics cards and a TV tuner card already. I have a 1X slot (which basically nothing will fit in) and the bottom X16 slot only runs at 4X (which the ConnectX3 is a 4x PCI-e 3.0 card).

Supposedly another possible fix is to edit the firmware and flash a firmware that limits or removes SR-IOV functions from the card.

EDIT - Took out of hypervisor and installed into my gaming rig. Different motherboard, but same chipset (AMD X570). Windows 11 picked up the card instantly without need for driver installation. Plugged DAC into a 10Gbit SFP+ Juniper switch and Windows reads 10Gbit speed.

So card is definitely working properly but XCP-NG 8.3 doesn't like it.

Hi all,

Recently bought a Mellanox ConnectX-3 CX311A from eBay.

Plugged it into my hypervisor XCP-NG 8.3, but can't seem to get the card to work. As far as I know this card is ethernet only so shouldn't require any flashing or anything like that.

I'm using a new DAC from FS.com from the card to the 10GB SFP+ port on my Juniper switch.

Card is detected:

lspci
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

lspci -v | grep Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
        Subsystem: Mellanox Technologies Device 0055

I deleted and re-created my PCI passthrough config

xl pci-assignable-list
0000:09:00.0
0000:05:00.1
0000:01:00.0
0000:0a:00.0
0000:05:00.0
0000:01:00.1

Tried PIF scan in GUI and console but when I check PIF list I only see the on board NIC

xe pif-scan host-uuid=ff0c8a58-0feb-4fe1-8cc1-556aad1f8c75
xe pif-list
uuid ( RO)                  : a902c9a0-77f2-ad66-9f69-814e9bdd6413
                device ( RO): eth0
                   MAC ( RO): d8:5e:d3:2b:23:8b
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 5e0c47e8-abbd-bc5b-02eb-83b838560f90
             host-uuid ( RO): ff0c8a58-0feb-4fe1-8cc1-556aad1f8c75

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether d8:5e:d3:2b:23:8b brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 92:b5:0c:1e:3e:cc brd ff:ff:ff:ff:ff:ff
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d8:5e:d3:2b:23:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global dynamic xenbr0
       valid_lft 81730sec preferred_lft 81730sec
5: vif1.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
7: vif2.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
9: vif3.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: vif4.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether d8:5e:d3:2b:23:8b  txqueuelen 1000  (Ethernet)
        RX packets 53756  bytes 20202528 (19.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27516  bytes 6309909 (6.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 9556  bytes 5978566 (5.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9556  bytes 5978566 (5.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif1.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 3859  bytes 336927 (329.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29344  bytes 9261592 (8.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 4605  bytes 937140 (915.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28965  bytes 8527803 (8.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif3.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 9936  bytes 1319123 (1.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33662  bytes 7927207 (7.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif4.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 7851  bytes 1274556 (1.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 31981  bytes 8681713 (8.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.13  netmask 255.255.255.0  broadcast 192.168.1.255
        ether d8:5e:d3:2b:23:8b  txqueuelen 1000  (Ethernet)
        RX packets 33215  bytes 5737003 (5.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6334  bytes 5412414 (5.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Any ideas?

@Niko1 Some motherboards updated their BIOS for Windows 11 compatibility which may have automatically turned on secure boot, etc, etc.

I can't see any other reason Linux ISO's would all of a sudden fail to boot.

Try the CLI method and see if that fixes it.

Those settings in XOA exist for vGPUs I believe, not for pass through.