Weird issue with PCIe passthrough and XCP-NG/Xenserver
-
Maybe @r1 or @fohdeesha will be able to assist
-
@PeterPorker3 said in Weird issue with PCIe passthrough and XCP-NG/Xenserver:
I get this error in XCP-ng Center
Can you check on node
# xl dmesgafter you try to start the VM? That should have some more info about the failure. -
I was never succesfully able to get PCI passthrough to work, but for me it was related to a bug in Dell's IOMMU implementation. In the BIOS make sure anything mentioning IOMMU, vt-d, and SR-IOV is enabled. However your CPUs not officially supporting SR-IOV points to your CPU/BIOS not having a proper IOMMU management/implementation either
-
@r1 said in Weird issue with PCIe passthrough and XCP-NG/Xenserver:
Can you check on node # xl dmesg after you try to start the VM?
Thanks for the help, I am going to have to start using the server I am testing this with for something else, I will keep the SSD I have XCP-NG installed on it and when I have time I can try suggestions and solutions, just keep in mind my responses may be delayed. Anyways, here is the information you requested:
__ __ _ _ _ _ _ \ \/ /___ _ __ | || | / / | / | \ // _ \ '_ \ | || |_ | | | | | / \ __/ | | | |__ _|| | |_| | /_/\_\___|_| |_| |_|(_)_|_(_)_| (XEN) [ 0.000000] Xen version 4.11.1-7.3.xcpng8.0 (mockbuild@[unknown]) (gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)) debug=n Thu May 16 19:44:22 CEST 2019 (XEN) [ 0.000000] Latest ChangeSet: 96cbd0893f78, pq 00446b784406 (XEN) [ 0.000000] Bootloader: GRUB 2.02 (XEN) [ 0.000000] Command line: dom0_mem=3488M,max:3488M watchdog ucode=scan dom0_max_vcpus=1-8 crashkernel=256M,below=4G console=vga vga=mode-0x0311 (XEN) [ 0.000000] Xen image load base address: 0xbd400000 (XEN) [ 0.000000] Video information: (XEN) [ 0.000000] VGA is graphics mode 640x480, 16 bpp (XEN) [ 0.000000] VBE/DDC methods: none; EDID transfer time: 0 seconds (XEN) [ 0.000000] EDID info not retrieved because no DDC retrieval method detected (XEN) [ 0.000000] Disc information: (XEN) [ 0.000000] Found 1 MBR signatures (XEN) [ 0.000000] Found 2 EDD information structures (XEN) [ 0.000000] Xen-e820 RAM map: (XEN) [ 0.000000] 0000000000000000 - 0000000000099800 (usable) (XEN) [ 0.000000] 0000000000099800 - 0000000000099c00 (reserved) (XEN) [ 0.000000] 000000000009e000 - 00000000000a0000 (reserved) (XEN) [ 0.000000] 00000000000f0000 - 0000000000100000 (reserved) (XEN) [ 0.000000] 0000000000100000 - 00000000bddac000 (usable) (XEN) [ 0.000000] 00000000bddac000 - 00000000bddde000 (ACPI data) (XEN) [ 0.000000] 00000000bddde000 - 00000000d0000000 (reserved) (XEN) [ 0.000000] 00000000fec00000 - 00000000fee10000 (reserved) (XEN) [ 0.000000] 00000000ff800000 - 0000000100000000 (reserved) (XEN) [ 0.000000] 0000000100000000 - 0000000c3ffff000 (usable) (XEN) [ 0.000000] Kdump: 256MB (262144kB) at 0xad600000 (XEN) [ 0.000000] ACPI: RSDP 000F4F00, 0024 (r2 HP ) (XEN) [ 0.000000] ACPI: XSDT BDDAECC0, 00F4 (r1 HP ProLiant 2 ▒ 162E) (XEN) [ 0.000000] ACPI: FACP BDDAEE00, 00F4 (r3 HP ProLiant 2 ▒ 162E) (XEN) [ 0.000000] ACPI: DSDT BDDAEF00, 26DC (r1 HP DSDT 1 INTL 20030228) (XEN) [ 0.000000] ACPI: FACS BDDAC140, 0040 (XEN) [ 0.000000] ACPI: SPCR BDDAC180, 0050 (r1 HP SPCRRBSU 1 ▒ 162E) (XEN) [ 0.000000] ACPI: MCFG BDDAC200, 003C (r1 HP ProLiant 1 0) (XEN) [ 0.000000] ACPI: HPET BDDAC240, 0038 (r1 HP ProLiant 2 ▒ 162E) (XEN) [ 0.000000] ACPI: FFFF BDDAC280, 0064 (r2 HP ProLiant 2 ▒ 162E) (XEN) [ 0.000000] ACPI: SPMI BDDAC300, 0040 (r5 HP ProLiant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: ERST BDDAC340, 0230 (r1 HP ProLiant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: APIC BDDAC580, 026A (r1 HP ProLiant 2 0) (XEN) [ 0.000000] ACPI: SRAT BDDAC800, 0750 (r1 HP Proliant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: FFFF BDDACF80, 0176 (r1 HP ProLiant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: BERT BDDAD100, 0030 (r1 HP ProLiant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: HEST BDDAD140, 00BC (r1 HP ProLiant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: DMAR BDDAD200, 0480 (r1 HP ProLiant 1 ▒ 162E) (XEN) [ 0.000000] ACPI: SLIT BDDAEC00, 0030 (r1 HP ProLiant 1 0) (XEN) [ 0.000000] ACPI: PCCT BDDAEC40, 006E (r1 HP Proliant 1 PH 504D) (XEN) [ 0.000000] ACPI: SSDT BDDB1600, 07EA (r1 HP DEV_PCI1 1 INTL 20120503) (XEN) [ 0.000000] ACPI: SSDT BDDB1E00, 0103 (r3 HP CRSPCI0 2 HP 1) (XEN) [ 0.000000] ACPI: SSDT BDDB1F40, 0098 (r3 HP CRSPCI1 2 HP 1) (XEN) [ 0.000000] ACPI: SSDT BDDB2000, 019A (r2 HP riser0 2 INTL 20030228) (XEN) [ 0.000000] ACPI: SSDT BDDB21C0, 01D4 (r3 HP riser1b 2 INTL 20030228) (XEN) [ 0.000000] ACPI: SSDT BDDB23C0, 01D5 (r3 HP riser2a 2 INTL 20030228) (XEN) [ 0.000000] ACPI: SSDT BDDB25C0, 0386 (r3 HP riser3a 2 INTL 20030228) (XEN) [ 0.000000] ACPI: SSDT BDDB2980, 0BB9 (r1 HP pcc 1 INTL 20120503) (XEN) [ 0.000000] ACPI: SSDT BDDB3540, 0377 (r1 HP pmab 1 INTL 20120503) (XEN) [ 0.000000] ACPI: SSDT BDDB38C0, 5524 (r1 HP pcc2 1 INTL 20120503) (XEN) [ 0.000000] ACPI: SSDT BDDB8E00, 4604 (r1 INTEL PPM RCM 1 INTL 20061109) (XEN) [ 0.000000] System RAM: 49117MB (50296080kB) (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 00 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 01 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 02 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 03 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 04 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 05 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 06 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 07 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 08 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 09 -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 0a -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 0b -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 0c -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 0d -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 0e -> Node 0 (XEN) [ 0.000000] SRAT: PXM 0 -> APIC 0f -> Node 0 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 20 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 21 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 22 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 23 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 24 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 25 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 26 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 27 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 28 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 29 -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 2a -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 2b -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 2c -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 2d -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 2e -> Node 1 (XEN) [ 0.000000] SRAT: PXM 1 -> APIC 2f -> Node 1 (XEN) [ 0.000000] SRAT: Node 0 PXM 0 0-640000000 (XEN) [ 0.000000] SRAT: Node 1 PXM 1 640000000-c40000000 (XEN) [ 0.000000] Domain heap initialised DMA width 32 bits (XEN) [ 0.000000] vesafb: framebuffer at 0xf5000000, mapped to 0xffff82c000201000, using 2048k, total 8192k (XEN) [ 0.000000] vesafb: mode is 640x480x16, linelength=1280, font 8x8 (XEN) [ 0.000000] vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0 (XEN) [ 0.000000] CPU Vendor: Intel, Family 6 (0x6), Model 45 (0x2d), Stepping 7 (raw 000206d7) (XEN) [ 0.000000] found SMP MP-table at 000f4f80 (XEN) [ 0.000000] DMI 2.8 present. (XEN) [ 0.000000] Using APIC driver default (XEN) [ 0.000000] ACPI: PM-Timer IO Port: 0x908 (32 bits) (XEN) [ 0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:904,1:0], pm1x_evt[1:900,1:0] (XEN) [ 0.000000] ACPI: wakeup_vec[bddac14c], vec_size[20] (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x20] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0x22] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x24] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x26] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x28] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x2a] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x2c] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2e] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0x21] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0x23] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x25] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x27] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x29] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x2b] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x2d] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x2f] enabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] disabled) (XEN) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) (XEN) [ 0.000000] Overriding APIC driver with bigsmp (XEN) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) (XEN) [ 0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 (XEN) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec10000] gsi_base[24]) (XEN) [ 0.000000] IOAPIC[1]: apic_id 0, version 32, address 0xfec10000, GSI 24-47 (XEN) [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48]) (XEN) [ 0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71 (XEN) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) (XEN) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) [ 0.000000] Enabling APIC mode: Phys. Using 3 I/O APICs (XEN) [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 (XEN) [ 0.000000] [VT-D] RMRR address range e8000..e8fff not in reserved memory; need "iommu_inclusive_mapping=1"? (XEN) [ 0.000000] Xen ERST support is initialized. (XEN) [ 0.000000] HEST: Table parsing has been initialized (XEN) [ 0.000000] Using ACPI (MADT) for SMP configuration information (XEN) [ 0.000000] SMP: Allowing 64 CPUs (32 hotplug CPUs) (XEN) [ 0.000000] IRQ limits: 72 GSI, 6088 MSI/MSI-X (XEN) [ 0.000000] Not enabling x2APIC (upon firmware request) (XEN) [ 0.000000] XSM Framework v1.0.0 initialized (XEN) [ 0.000000] Initialising XSM SILO mode (XEN) [ 0.000000] microcode: CPU0 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] xstate: size: 0x340 and states: 0x7 (XEN) [ 0.000000] CPU0: Intel machine check reporting enabled (XEN) [ 0.000000] Speculative mitigation facilities: (XEN) [ 0.000000] Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD MD_CLEAR (XEN) [ 0.000000] Compiled-in support: INDIRECT_THUNK SHADOW_PAGING (XEN) [ 0.000000] Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- SSBD-, Other: IBPB L1D_FLUSH VERW (XEN) [ 0.000000] L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 46, Safe address 300000000000 (XEN) [ 0.000000] Support for VMs: PV: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR, HVM: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR (XEN) [ 0.000000] XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID) (XEN) [ 0.000000] PV L1TF shadowing: Dom0 disabled, DomU enabled (XEN) [ 0.000000] Using scheduler: SMP Credit Scheduler (credit) (XEN) [ 0.000000] Platform timer is 14.318MHz HPET (XEN) [ 2.696110] Detected 1795.689 MHz processor. (XEN) [ 2.712697] Initing memory sharing. (XEN) [ 2.725055] alt table ffff82d080439698 -> ffff82d08043b642 (XEN) [ 2.738548] PCI: MCFG configuration 0: base c0000000 segment 0000 buses 00 - ff (XEN) [ 2.763079] PCI: MCFG area at c0000000 reserved in E820 (XEN) [ 2.775468] PCI: Using MCFG for segment 0000 bus 00-ff (XEN) [ 2.788229] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB. (XEN) [ 2.801297] Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB. (XEN) [ 2.814378] Intel VT-d Snoop Control enabled. (XEN) [ 2.826691] Intel VT-d Dom0 DMA Passthrough not enabled. (XEN) [ 2.839293] Intel VT-d Queued Invalidation enabled. (XEN) [ 2.851581] Intel VT-d Interrupt Remapping enabled. (XEN) [ 2.863879] Intel VT-d Posted Interrupt not enabled. (XEN) [ 2.876375] Intel VT-d Shared EPT tables enabled. (XEN) [ 2.904771] I/O virtualisation enabled (XEN) [ 2.916814] - Dom0 mode: Relaxed (XEN) [ 2.928776] Interrupt remapping enabled (XEN) [ 2.940701] Enabled directed EOI with ioapic_ack_old on! (XEN) [ 2.953134] ENABLING IO-APIC IRQs (XEN) [ 2.964964] -> Using old ACK method (XEN) [ 2.977073] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) [ 3.441737] Defaulting to alternative key handling; send 'A' to switch to normal mode. (XEN) [ 3.464861] Allocated console ring of 128 KiB. (XEN) [ 3.476563] VMX: Supported advanced features: (XEN) [ 3.488259] - APIC MMIO access virtualisation (XEN) [ 3.500173] - APIC TPR shadow (XEN) [ 3.511610] - Extended Page Tables (EPT) (XEN) [ 3.523023] - Virtual-Processor Identifiers (VPID) (XEN) [ 3.534472] - Virtual NMI (XEN) [ 3.545691] - MSR direct-access bitmap (XEN) [ 3.556929] - Unrestricted Guest (XEN) [ 3.568118] HVM: ASIDs enabled. (XEN) [ 3.579235] HVM: VMX enabled (XEN) [ 3.590231] HVM: Hardware Assisted Paging (HAP) detected (XEN) [ 3.601359] HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) [ 0.000000] microcode: CPU2 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU4 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU6 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU8 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU10 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU12 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU14 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU16 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU18 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU20 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU22 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU24 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU26 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU28 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 0.000000] microcode: CPU30 updated from revision 0x714 to 0x718, date = 2019-05-21 (XEN) [ 4.636141] Brought up 32 CPUs (XEN) [ 4.647850] Testing NMI watchdog on all CPUs: ok (XEN) [ 4.719879] build-id: 647b367a562165f4bf2d967551255bee1b477a7c (XEN) [ 4.750814] ACPI sleep modes: S3 (XEN) [ 4.761477] VPMU: disabled (XEN) [ 4.771810] mcheck_poll: Machine check polling timer started. (XEN) [ 4.782410] Dom0 has maximum 1224 PIRQs (XEN) [ 4.792941] csched_alloc_domdata: setting dom 0 as the privileged domain (XEN) [ 4.813537] NX (Execute Disable) protection active (XEN) [ 4.824174] *** Building a PV Dom0 *** (XEN) [ 5.148812] Xen kernel: 64-bit, lsb, compat32 (XEN) [ 5.159413] Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x302c000 (XEN) [ 5.180913] PHYSICAL MEMORY ARRANGEMENT: (XEN) [ 5.191417] Dom0 alloc.: 0000000c1c000000->0000000c20000000 (871807 pages to be allocated) (XEN) [ 5.212505] Init. ramdisk: 0000000c3eb7f000->0000000c3fdffabf (XEN) [ 5.223462] VIRTUAL MEMORY ARRANGEMENT: (XEN) [ 5.234119] Loaded kernel: ffffffff81000000->ffffffff8302c000 (XEN) [ 5.245308] Init. ramdisk: 0000000000000000->0000000000000000 (XEN) [ 5.256214] Phys-Mach map: 0000008000000000->00000080006d0000 (XEN) [ 5.267151] Start info: ffffffff8302c000->ffffffff8302c4b4 (XEN) [ 5.278151] Xenstore ring: 0000000000000000->0000000000000000 (XEN) [ 5.289222] Console ring: 0000000000000000->0000000000000000 (XEN) [ 5.300372] Page tables: ffffffff8302d000->ffffffff8304a000 (XEN) [ 5.311615] Boot stack: ffffffff8304a000->ffffffff8304b000 (XEN) [ 5.322956] TOTAL: ffffffff80000000->ffffffff83400000 (XEN) [ 5.334315] ENTRY ADDRESS: ffffffff82427180 (XEN) [ 5.346942] Dom0 has maximum 8 VCPUs (XEN) [ 9.814959] Masked UR signaling on 0000:00:00.0 (XEN) [ 9.826187] Masked UR signaling on 0000:00:01.0 (XEN) [ 9.837336] Masked UR signaling on 0000:00:01.1 (XEN) [ 9.848492] Masked UR signaling on 0000:00:03.0 (XEN) [ 9.859569] Masked UR signaling on 0000:00:03.1 (XEN) [ 9.871004] Masked UR signaling on 0000:00:03.2 (XEN) [ 9.882008] Masked UR signaling on 0000:00:03.3 (XEN) [ 9.893036] Masked VT-d error signaling on 0000:00:05.0 (XEN) [ 9.908003] Masked UR signaling on 0000:20:00.0 (XEN) [ 9.918982] Masked UR signaling on 0000:20:01.0 (XEN) [ 9.929883] Masked UR signaling on 0000:20:01.1 (XEN) [ 9.940792] Masked UR signaling on 0000:20:03.0 (XEN) [ 9.951619] Masked UR signaling on 0000:20:03.1 (XEN) [ 9.962448] Masked UR signaling on 0000:20:03.2 (XEN) [ 9.973200] Masked UR signaling on 0000:20:03.3 (XEN) [ 9.983974] Masked VT-d error signaling on 0000:20:05.0 (XEN) [ 10.025774] Initial low memory virq threshold set at 0x4000 pages. (XEN) [ 10.036616] Scrubbing free RAM in background (XEN) [ 10.047305] Std. Loglevel: Errors, warnings and info (XEN) [ 10.058217] Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) [ 10.069406] *************************************************** (XEN) [ 10.080275] Booted on L1TF-vulnerable hardware with SMT/Hyperthreading (XEN) [ 10.091173] enabled. Please assess your configuration and choose an (XEN) [ 10.102131] explicit 'smt=<bool>' setting. See XSA-273. (XEN) [ 10.113008] *************************************************** (XEN) [ 10.124330] Booted on MLPDS/MFBDS-vulnerable hardware with SMT/Hyperthreading (XEN) [ 10.145566] enabled. Mitigations will not be fully effective. Please (XEN) [ 10.156568] choose an explicit smt=<bool> setting. See XSA-297. (XEN) [ 10.167608] *************************************************** (XEN) [ 10.178746] 3... 2... 1... (XEN) [ 13.189825] Xen is relinquishing VGA console. (XEN) [ 13.264258] *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) [ 13.264695] Freed 532kB init memory (XEN) [ 18.417211] Found masked UR signaling on 0000:00:00.0 (XEN) [ 18.417921] Found masked UR signaling on 0000:00:01.0 (XEN) [ 18.418608] Found masked UR signaling on 0000:00:01.1 (XEN) [ 18.419328] Found masked UR signaling on 0000:00:03.0 (XEN) [ 18.420016] Found masked UR signaling on 0000:00:03.1 (XEN) [ 18.420724] Found masked UR signaling on 0000:00:03.2 (XEN) [ 18.421381] Found masked UR signaling on 0000:00:03.3 (XEN) [ 18.427226] Masked VT-d error signaling on 0000:00:05.0 (XEN) [ 18.448450] Found masked UR signaling on 0000:20:00.0 (XEN) [ 18.449116] Found masked UR signaling on 0000:20:01.0 (XEN) [ 18.449794] Found masked UR signaling on 0000:20:01.1 (XEN) [ 18.450551] Found masked UR signaling on 0000:20:03.0 (XEN) [ 18.451234] Found masked UR signaling on 0000:20:03.1 (XEN) [ 18.451926] Found masked UR signaling on 0000:20:03.2 (XEN) [ 18.452640] Found masked UR signaling on 0000:20:03.3 (XEN) [ 18.458452] Masked VT-d error signaling on 0000:20:05.0@fohdeesha said in Weird issue with PCIe passthrough and XCP-NG/Xenserver:
I was never succesfully able to get PCI passthrough to work, but for me it was related to a bug in Dell's IOMMU implementation. In the BIOS make sure anything mentioning IOMMU, vt-d, and SR-IOV is enabled. However your CPUs not officially supporting SR-IOV points to your CPU/BIOS not having a proper IOMMU management/implementation either
Yea, after many hours of trobbleshooting this issue, that's what it's starting to look like for me. I recently tried Proxmox on the same server, and even after adding the IOMMU stuff in the kernel boot paramaters like the official guides say to do, functionality doesn't change at all. I can't even complete the passthrough process in proxmox because I have to use hardware ids, which all the NICs have the same one, and I have to disable the driver, and all the NICs use the same driver. It's possible I could fix this by buying an Ivy Bridge Xeon or a 2nd network card off ebay, but at that point I may as well save up for a 2nd used server to run pfsense on bare medal like your supposed to do. I think, because vmware can do it, i'm just going to run ESXi on the server, then save up for a 2nd server to handle pfsense and switch to Xen on this one. However, I am intirested in investigating the issue further if it is an issue that can be corrected on the OS side. Anyways, thanks for the help!
-
pretty much everyone here is running PfSense fully virtualized, no passthrough. I'm able to saturate gigabit links with full firewalling etc. What's nice about not having it pegged down to one host with a pci passthrough is you can hot-migrate it between servers with no dropped traffic. Or maybe you wanted passthrough for the ALTQ-reliant traffic shaping features?
-
Hi,
I am having this issue, too, after passing through m usb 3 card to a vm.
When I now try to start the vm it will not and instead throw this error.
INTERNAL_ERROR(xenopsd internal error: (Failure
"Error from xenguesthelper: xenguest: xc_dom_build_image: [1] panic: xc_dom_boot.c:159: xc_dom_boot_domU_map: failed "))I am using XCP-NG on a ML350p (BIOS recently updated).
Any help would be appreciated!
Cheers
-
-
Sorry, I have moved away from XCP to Proxmox since then.
-
@alexanderk Hi, apologies but we were never able to trace this down to anything in the xen kernel itself, it seems to be something specific to the older gen 8 HP and how their BIOS/UEFI firmware handles VT-d / passthrough. If there's newer BIOS firmware available for your machine, you could try that, and double check vt-d, IOMMU etc is enabled in bios.
-
@fohdeesha really offtopic.. you are the one who made the tutorials for flashing it mode to various raid cards??
-
@alexanderk Heh yes, and a few other Brocade/Quanta/Dell guides
-
@fohdeesha wow!
thanks for your help! really happy that you belong to xcp-ng pro support team! -
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem. message: xenopsd internal error: (Failure "Error from xenguesthelper: xenguest: xc_dom_build_image: [1] panic: xc_dom_boot.c:159: xc_dom_boot_domU_map: failed ")hpe dl360p gen8. So its HPE problem?
