Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work
-
@tuxen that didn't work either.
probably not so interesting but tried Xcp-ng 8.2.1 also same error there
-
@tuxen Got hold of custom bios with G4 available now. But disabling 4g didn't help.. It was hidden but always enabled
But see that in Proxmox the pci slot does not change. it stays at slot 1 with Xcp-ng it changes to slot 8.
Proxmox user uses hex pci id for gpu passthrough does this matter?
-
It shouldn't. Any idea here @Teddy-Astie ?
-
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
@tuxen Got hold of custom bios with G4 available now. But disabling 4g didn't help.. It was hidden but always enabled
But see that in Proxmox the pci slot does not change. it stays at slot 1 with Xcp-ng it changes to slot 8.
Is this slot change in the guest or in XCP-ng itself ?
I would not be supprised to see it chaging in the guest (it's a part of what QEMU pci passthrough can do) but that would be weird for Dom0.Proxmox user uses hex pci id for gpu passthrough does this matter?
I don't think it's a guarantee that you see the same BDF in guest and host regarding in proxmox (more likely a coincidence). But that's a behavior that would change in XCP-ng with Q35/proper PCIe support.
Aside that, it "appears" to work in a Linux VM (albeit with no display), have you managed to make CUDA running there (to see if the device actually works) ? And if the BAR window message still exist.
-
@Teddy-Astie no only in guest.
Haven't tried anything more with pop os than trying to extend the screen.
can try to look at it with CUDA running -
@Teddy-Astie I have installed nvidia drivers and CUDA Toolkit. It seems like the drivers are running properly and I see that the pop os gpu card in popos but ran dmesg again and got an error
ation="profile_load" profile="unconfined" name="libreoffice-xpdfimport" pid=516 comm="apparmor_parser" [ 5.425026] Adding 4193784k swap on /dev/mapper/cryptswap. Priority:-2 extents:1 across:4193784k SS [ 5.427939] nvidia: module license 'NVIDIA' taints kernel. [ 5.427941] Disabling lock debugging due to kernel taint [ 5.427943] nvidia: module license taints kernel. [ 5.473953] nvidia-nvlink: Nvlink Core is being initialized, major device number 238 [ 5.475202] xen: --> pirq=24 -> irq=17 (gsi=17) [ 5.475401] nvidia 0000:00:08.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io+mem [ 5.476296] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 560.35.03 Fri Aug 16 21:39:15 UTC 2024 [ 5.502521] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 560.35.03 Fri Aug 16 21:21:48 UTC 2024 [ 5.505900] [drm] [nvidia-drm] [GPU ID 0x00000008] Loading driver [ 5.829042] zram: Added device: zram0 [ 6.103268] kauditd_printk_skb: 18 callbacks suppressed [ 6.103273] audit: type=1400 audit(1731496933.838:29): apparmor="DENIED" operation="capable" class="cap" profile="/usr/sbin/cupsd" pid=671 comm="cupsd" capability=12 capname="net_admin" [ 6.288396] snd_hda_intel 0000:00:09.0: azx_get_response timeout, switching to polling mode: last cmd=0x000f0000 [ 6.385067] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:08.0 on minor 1 [ 6.422276] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint. [ 6.452377] nvidia-uvm: Loaded the UVM driver, major device number 236. [ 6.844962] zram0: detected capacity change from 0 to 32735232 [ 7.215079] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:09.0/sound/card0/input6 [ 7.216764] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:09.0/sound/card0/input7 [ 7.217068] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:09.0/sound/card0/input8 [ 7.217363] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:09.0/sound/card0/input9 [ 7.332815] Adding 16367612k swap on /dev/zram0. Priority:1000 extents:1 across:16367612k SS [ 10.684565] rfkill: input handler disabled [ 20.569595] rfkill: input handler enabled [ 23.259676] rfkill: input handler disabled [ 23.572353] [drm:nv_drm_master_set [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000008] Failed to grab modeset ownership steff@pop-os:~$ lspci | grep VGA 00:02.0 VGA compatible controller: Device 1234:1111 (rev 02) 00:08.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1) -
@steff22 This was a known bug that had no significance.
((The warning message is expected. When a client (such as the modesetting driver) attempts to open our DRM device node while modesetting permission is already acquired by something else (like the NVIDIA X driver), it has to fail, but the kernel won’t let us return a failure after v5.9-rc1, so we print this message. It won’t impact functionality of the NVIDIA X driver that already has modesetting permission. Safe to ignore as long as long as you didn’t need the other client to actually get modesetting permission. If you want to suppress the error, you would need to find which client is attempting to open the NVIDIA DRM device node and prevent it from doing so.))
-
@steff22 I have some questions:
- Is the host being powered up with a monitor or a dummy plug (headless) already attached to the dGPU?
- Without rebooting the VM and right after the driver installation succeeds (showing that the device is working OK), what happens if you click the
[Detect]button at the display settings window? - Instead of a reboot, did you try a VM shutdown/start cycle for the 1st time after the driver installation?
Nonetheless, if the same dGPU card works normally on another XCP-ng host, a possible Xen passthrough incompatibility with that AM5 board should be considered. For example:
- CSM/UEFI GPU compat issues, as referenced by @Teddy-Astie
- Beta BIOS/IOMMU broken or lacking features (eg. ACS support for PCI/PCIe isolation)
Tux
-
@tuxen No. 1 yes there is an hdmi screen connected to the dGPU.
Is also an ipmi with dedicated vga connected.But is an error that makes the primary screen not work completely.
The bios disabled internal ipma when an Ext GPU card is connected even though int gpu is selected as primary gpu in the bios. So I only see xcp-ng startup on screen no xsconsole. Have tried without a screen connected extgpu same error then
no. 2 Have tried pressing Detect only to be told that there is no more screen. Have only tried reboot
At first I thought there was something wrong with the bios. But this works with Vmware esxi and proxmox.
-
@steff22 I think I remembered that there were also some others who struggled with the same problem with Asrock rack B650D4U with gpu passthrough, not sure if it was with AM5.
-
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
The bios disabled internal ipma when an Ext GPU card is connected even though int gpu is selected as primary gpu in the bios. So I only see xcp-ng startup on screen no xsconsole. Have tried without a screen connected extgpu same error then
I suggest to call the Asrock support and explain this behavior.
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
no. 2 Have tried pressing Detect only to be told that there is no more screen. Have only tried reboot
Could you try the shutdown/start after the driver installation?
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
At first I thought there was something wrong with the bios. But this works with Vmware esxi and proxmox.
Considering it worked with the same XCP-ng version, but on a different hardware, that's why I'm more inclined to a Xen incompatibility issue with the combo Nvidia + some AMD motherboards. If you search the forum, there's a mixed result about that.
-
@tuxen tried shutdown/start now same error.
Yes I have that opinion too when it comes to Nvidia the drivers are a bit picky.
Got it to work Asrock motherboard and an old Radeon HD 5450 so I think I'll sell the 1070ti card is starting to get old. Oh, I'd rather invest in a new Radeon GPU. Then I'll have to bet that this works on the Asrock motherboard.Does vGPU only work like that on enterprise GPUs like Nvidia Tesla?
-
Thank you for the time you spent on the troubleshooting process
-
what kind of magic have you put in the last 7 patches?
Now everything works no errors with letting windows install the drivers or adding drivers yourself Directly from nvidia
Double checked this by installing a clean Xcp-ng from iso without update. Clean win 10 install install drivers then error 43 appeared install patch and started win 10 and everything works. also works with win 11
-
@steff22 Wow, great news! Kudos to the Xen & XCP-ng dev teams
-
@tuxen yes Xen & XCP-ng dev teams have been really good
Started to get a little nervous that the big investment in hardware was wasted. Never would have needed such a powerful cpu without gpu passthrough.So thank you very much Xen & XCP-ng dev teams
-
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
what kind of magic have you put in the last 7 patches?
Now everything works no errors with letting windows install the drivers or adding drivers yourself Directly from nvidia
Double checked this by installing a clean Xcp.ng from iso without update clean win 10 install install drivers then error 43 appeared install patch and started win 10 and everything works. also works with win 11
Thank you very much for your feedback. Frankly, IDK personally about the detailed content of the patch and why it fixed your issue, but I will pass the word to the team so we can guess/pinpoint the issue. Also keeping @Teddy-Astie in the loop
-
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
what kind of magic have you put in the last 7 patches?
You've got a very recent AMD processor, so it's probably this fix https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=86001b3970fea4536048607ea6e12541736c48e1 from upstream.
-
@andyhhp Was something like I was hoping for. But thank you very much for the fix
But still doesn't work on pop os 22.04. Does the guest tools version matter? It seems to be using V6.6.80-0
-
Before this update fix I ordered a new Radeon rx 7800 xt Gpu in case there was something wrong with Nvidia Gpu.
But now when I try the new Radeon rx 7800 xt it doesn't work and Xcp-ng has become unstable and hangs.
09:40 xcp-ng-AsRoCk ~]# xl dmesg __ __ _ _ _ _____ ____ \ \/ /___ _ __ | || | / |___ | ___| \ // _ \ '_ \ | || |_ | | / /|___ \ / \ __/ | | | |__ _|| | / /_ ___) | /_/\_\___|_| |_| |_|(_)_|/_/(_)____/ (XEN) [00000038f3c0ea92] Xen version 4.17.5-4 (mockbuild@[unknown]) (gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)) debug=n Wed Nov 13 11:42:55 CET 2024 (XEN) [00000038f3c0fbb4] Latest ChangeSet: 430ce6cd9365, pq f09cf5633773 (XEN) [00000038f3c10eaf] build-id: 0c6d91001e3befae6cbe6dbb635b6313c99e706c (XEN) [00000038f3c11490] Bootloader: GRUB 2.06 (XEN) [00000038f3c11a71] Command line: dom0_mem=2624M,max:2624M watchdog ucode=scan dom0_max_vcpus=1-8 crashkernel=256M,below=4G console=vga vga=mode-0x0311 (XEN) [00000038f3c12302] Xen image load base address: 0x86e00000 (XEN) [00000038f3c1270a] Video information: (XEN) [00000038f3c130f3] VGA is graphics mode 1024x768, 32 bpp (XEN) [00000038f3c134a5] Disc information: (XEN) [00000038f3c13c09] Found 0 MBR signatures (XEN) [00000038f3c13fe6] Found 3 EDD information structures (XEN) [00000038f3c15e78] CPU Vendor: AMD, Family 26 (0x1a), Model 68 (0x44), Stepping 0 (raw 00b40f40) (XEN) [00000038f3c16d96] Enabling Supervisor Shadow Stacks (XEN) [00000038f3c1b5a5] EFI RAM map: (XEN) [00000038f3c1bf63] [0000000000000000, 000000000009ffff] (usable) (XEN) [00000038f3c1c671] [00000000000a0000, 00000000000fffff] (reserved) (XEN) [00000038f3c1cd29] [0000000000100000, 0000000009afefff] (usable) (XEN) [00000038f3c1d208] [0000000009aff000, 0000000009ffffff] (reserved) (XEN) [00000038f3c1d691] [000000000a000000, 000000000a1fffff] (usable) (XEN) [00000038f3c1daef] [000000000a200000, 000000000a21dfff] (ACPI NVS) (XEN) [00000038f3c1dfa3] [000000000a21e000, 000000000affffff] (usable) (XEN) [00000038f3c1e401] [000000000b000000, 000000000b020fff] (reserved) (XEN) [00000038f3c1e8e0] [000000000b021000, 000000008857efff] (usable) (XEN) [00000038f3c1edea] [000000008857f000, 000000008e57efff] (reserved) (XEN) [00000038f3c1f2c9] [000000008e57f000, 000000008e67efff] (ACPI data) (XEN) [00000038f3c1f854] [000000008e67f000, 000000009067efff] (ACPI NVS) (XEN) [00000038f3c1fcdd] [000000009067f000, 00000000987fefff] (reserved) (XEN) [00000038f3c20166] [00000000987ff000, 0000000099ff6fff] (usable) (XEN) [00000038f3c205c4] [0000000099ff7000, 0000000099ffbfff] (reserved) (XEN) [00000038f3c20a22] [0000000099ffc000, 0000000099ffffff] (usable) (XEN) [00000038f3c20eab] [000000009a000000, 000000009bffffff] (reserved) (XEN) [00000038f3c21309] [000000009d7f3000, 000000009fffffff] (reserved) (XEN) [00000038f3c21767] [00000000e0000000, 00000000efffffff] (reserved) (XEN) [00000038f3c21c1b] [00000000fd000000, 00000000ffffffff] (reserved) (XEN) [00000038f3c2217b] [0000000100000000, 000000083dd7ffff] (usable) (XEN) [00000038f3c2265a] [000000083edc0000, 00000008801fffff] (reserved) (XEN) [00000038f3c22bba] [000000fd00000000, 000000ffffffffff] (reserved) (XEN) [00000038f68c20e8] Kdump: 256MB (262144kB) at 0x77000000 (XEN) [00000038f694ab30] ACPI: RSDP 90665014, 0024 (r2 ALASKA) (XEN) [00000038f694d228] ACPI: XSDT 90664728, 00E4 (r1 ALASKA A M I 1072009 AMI 1000013) (XEN) [00000038f694f33f] ACPI: FACP 8E673000, 0114 (r6 ALASKA A M I 1072009 AMI 10013) (XEN) [00000038f6951150] ACPI: DSDT 8E5F6000, 7329C (r208 ALASKA A M I 1072009 INTL 20230331) (XEN) [00000038f6952043] ACPI: FACS 90662000, 0040 (XEN) [00000038f69529d6] ACPI: SSDT 8E676000, 816C (r2 AMD Splinter 2 MSFT 5000000) (XEN) [00000038f695381d] ACPI: SPMI 8E675000, 0041 (r5 ALASKA A M I 0 AMI. 0) (XEN) [00000038f69543df] ACPI: SSDT 8E674000, 03F1 (r2 ALASKA CPUSSDT 1072009 AMI 1072009) (XEN) [00000038f6954f4b] ACPI: FIDT 8E66D000, 009C (r1 ALASKA A M I 1072009 AMI 10013) (XEN) [00000038f6955ce6] ACPI: MCFG 8E66C000, 003C (r1 ALASKA A M I 1072009 MSFT 10013) (XEN) [00000038f69567a6] ACPI: HPET 8E66B000, 0038 (r1 ALASKA A M I 1072009 AMI 5) (XEN) [00000038f695723b] ACPI: FPDT 8E66A000, 0044 (r1 ALASKA A M I 1072009 AMI 1000013) (XEN) [00000038f6957cd0] ACPI: VFCT 8E5EB000, AE84 (r1 ALASKA A M I 1 AMD 33504F47) (XEN) [00000038f6958811] ACPI: BGRT 8E672000, 0038 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) [00000038f6959250] ACPI: SPCR 8E671000, 0050 (r2 ALASKA A M I 1072009 AMI 50023) (XEN) [00000038f695a143] ACPI: SSDT 8E5E1000, 9BAE (r2 AMD AMD CPU 1 AMD 1) (XEN) [00000038f695abad] ACPI: TPM2 8E670000, 004C (r4 ALASKA A M I 1 AMI 0) (XEN) [00000038f695b5c1] ACPI: SSDT 8E66F000, 06D4 (r2 AMD CPMWLRC 1 INTL 20230331) (XEN) [00000038f695c056] ACPI: SSDT 8E5DF000, 169E (r2 AMD CPMDFIG2 1 INTL 20230331) (XEN) [00000038f695cac0] ACPI: SSDT 8E5DC000, 2AA6 (r2 AMD CDFAAIG2 1 INTL 20230331) (XEN) [00000038f695d4d4] ACPI: SSDT 8E5D2000, 9A9E (r2 AMD CPMCMN 1 INTL 20230331) (XEN) [00000038f695e2f0] ACPI: SSDT 8E5CF000, 26F3 (r2 AMD AOD 1 INTL 20230331) (XEN) [00000038f695ed04] ACPI: WSMT 8E66E000, 0028 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) [00000038f695f718] ACPI: APIC 8E5CE000, 015E (r6 ALASKA A M I 1072009 AMI 10013) (XEN) [00000038f6960182] ACPI: IVRS 8E5CD000, 00C8 (r2 AMD AmdTable 1 AMD 1) (XEN) [00000038f6960bec] ACPI: SSDT 8E5CC000, 0500 (r2 AMD MEMTOOL0 2 INTL 20230331) (XEN) [00000038f6961656] ACPI: SSDT 8E5CB000, 047C (r2 AMD AMDWOV 1 INTL 20230331) (XEN) [00000038f6962447] ACPI: SSDT 8E5CA000, 0460 (r2 AMD AmdTable 1 INTL 20230331) (XEN) [00000038f699a05c] System RAM: 31861MB (32626028kB) (XEN) [00000038f86a80e8] No NUMA configuration found (XEN) [00000038f86a89a4] Faking a node at 0000000000000000-000000083dd80000 (XEN) [00000038fcfe4673] Domain heap initialised (XEN) [00000038ff20f14f] vesafb: framebuffer at 0x00000000fa000000, mapped to 0xffff82c000201000, using 3072k, total 3072k (XEN) [00000038ff20ff15] vesafb: mode is 1024x768x32, linelength=4096, font 8x14 (XEN) [00000038ff21108d] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0 (XEN) [0000003900d0a80b] SMBIOS 3.7 present. (XEN) [0000003900d8b403] XSM Framework v1.0.1 initialized (XEN) [0000003900e3f7b3] Initialising XSM SILO mode (XEN) [0000003900f3da8f] Using APIC driver default (XEN) [0000003900ff02b3] ACPI: PM-Timer IO Port: 0x808 (32 bits) (XEN) [00000039010a25cd] ACPI: v5 SLEEP INFO: control[0:0], status[0:0] (XEN) [000000390117a7f1] ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0] (XEN) [00000039012704c3] ACPI: 32/64X FACS address mismatch in FADT - 90662000/0000000000000000, using 32 (XEN) [000000390139cbdc] ACPI: wakeup_vec[9066200c], vec_size[20] (XEN) [00000039015a7e3f] Overriding APIC driver with bigsmp (XEN) [000000390167c5ef] ACPI: IOAPIC (id[0x20] address[0xfec00000] gsi_base[0]) (XEN) [000000390175654d] IOAPIC[0]: apic_id 32, version 33, address 0xfec00000, GSI 0-23 (XEN) [0000003901858043] ACPI: IOAPIC (id[0x21] address[0xfec01000] gsi_base[24]) (XEN) [0000003901948ffc] IOAPIC[1]: apic_id 33, version 33, address 0xfec01000, GSI 24-55 (XEN) [0000003901a7b0c7] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) (XEN) [0000003901b582b4] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) (XEN) [0000003901c81463] ACPI: HPET id: 0x10228201 base: 0xfed00000 (XEN) [0000003901d41739] PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff (XEN) [0000003901e47203] PCI: MCFG area at e0000000 reserved in E820 (XEN) [0000003901f1714e] PCI: Using MCFG for segment 0000 bus 00-ff (XEN) [0000003901fe8746] ACPI: BGRT: invalidating v1 image at 0x82521018 (XEN) [00000039020bf08e] Using ACPI (MADT) for SMP configuration information (XEN) [00000039021a2ea7] SMP: Allowing 32 CPUs (0 hotplug CPUs) (XEN) [000000390226e73b] IRQ limits: 56 GSI, 6600 MSI/MSI-X (XEN) [00000039024b2527] BSP microcode revision: 0x0b404022 (XEN) [000000390255b26d] xstate: size: 0x988 and states: 0x2e7 (XEN) [000000390266a86b] CPU0: AMD Fam1ah machine check reporting enabled (XEN) [0000003902736053] Speculative mitigation facilities: (XEN) [00000039027ee98d] Hardware hints: STIBP_ALWAYS IBRS_FAST IBRS_SAME_MODE BTC_NO IBPB_RET IBPB_BRTYPE (XEN) [0000003902921bfb] Hardware features: IBPB IBRS STIBP SSBD PSFD L1D_FLUSH SBPB (XEN) [0000003902a1e753] Compiled-in support: INDIRECT_THUNK HARDEN_ARRAY HARDEN_BRANCH HARDEN_GUEST_ACCESS HARDEN_LOCK (XEN) [0000003902b73ed8] Xen settings: BTI-Thunk: JMP, SPEC_CTRL: IBRS+ STIBP+ SSBD- PSFD-, Other: IBPB-ctxt BRANCH_HARDEN (XEN) [0000003902ccf00f] Support for HVM VMs: MSR_SPEC_CTRL MSR_VIRT_SPEC_CTRL RSB IBPB-entry (XEN) [0000003902de2d45] Support for PV VMs: None (XEN) [0000003902e88bd9] XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID) (XEN) [0000003902f9799c] PV L1TF shadowing: Dom0 disabled, DomU disabled (XEN) [0000003903076490] Using scheduler: SMP Credit Scheduler (credit) (XEN) [000000390fe0dbff] Platform timer is 14.318MHz HPET (XEN) [ 0.110124] Detected 4291.856 MHz processor. (XEN) [ 0.112783] EFI memory map: (XEN) [ 0.112881] 0000000000000-0000000003fff type=2 attr=000000000000000f (XEN) [ 0.113099] 0000000004000-000000008efff type=7 attr=000000000000000f (XEN) [ 0.113316] 000000008f000-000000009efff type=2 attr=000000000000000f (XEN) [ 0.113534] 000000009f000-000000009ffff type=4 attr=000000000000000f (XEN) [ 0.113752] 0000000100000-00000039e2fff type=2 attr=000000000000000f (XEN) [ 0.113970] 00000039e3000-0000009afefff type=7 attr=000000000000000f (XEN) [ 0.114187] 0000009aff000-0000009ffffff type=0 attr=000000000000000f (XEN) [ 0.114407] 000000a000000-000000a1fffff type=7 attr=000000000000000f (XEN) [ 0.114624] 000000a200000-000000a21dfff type=10 attr=000000000000000f (XEN) [ 0.114851] Setting RUNTIME attribute for 000000a200000-000000a21dfff (XEN) [ 0.115075] 000000a21e000-000000a29dfff type=4 attr=000000000000000f (XEN) [ 0.115304] 000000a29e000-000000affffff type=7 attr=000000000000000f (XEN) [ 0.115522] 000000b000000-000000b020fff type=0 attr=000000000000000f (XEN) [ 0.115739] 000000b021000-00000603dffff type=7 attr=000000000000000f (XEN) [ 0.121062] 00000603e0000-0000080b2cfff type=1 attr=000000000000000f (XEN) [ 0.126460] 0000080b2d000-0000080b2efff type=4 attr=000000000000000f (XEN) [ 0.131856] 0000080b2f000-0000080b6dfff type=7 attr=000000000000000f (XEN) [ 0.137260] 0000080b6e000-0000080b6efff type=4 attr=000000000000000f (XEN) [ 0.142676] 0000080b6f000-0000080b70fff type=7 attr=000000000000000f (XEN) [ 0.148364] 0000080b71000-0000080b80fff type=4 attr=000000000000000f (XEN) [ 0.153818] 0000080b81000-0000080b83fff type=7 attr=000000000000000f (XEN) [ 0.159282] 0000080b84000-0000080ba0fff type=4 attr=000000000000000f (XEN) [ 0.164692] 0000080ba1000-0000080ba5fff type=7 attr=000000000000000f (XEN) [ 0.170186] 0000080ba6000-0000080ba6fff type=4 attr=000000000000000f (XEN) [ 0.175609] 0000080ba7000-0000080badfff type=7 attr=000000000000000f (XEN) [ 0.181021] 0000080bae000-0000080baffff type=4 attr=000000000000000f (XEN) [ 0.186349] 0000080bb0000-0000080bb1fff type=7 attr=000000000000000f (XEN) [ 0.191718] 0000080bb2000-0000080bbefff type=4 attr=000000000000000f (XEN) [ 0.197112] 0000080bbf000-0000080db7fff type=7 attr=000000000000000f (XEN) [ 0.202448] 0000080db8000-0000080dd4fff type=4 attr=000000000000000f (XEN) [ 0.207792] 0000080dd5000-0000080e0efff type=7 attr=000000000000000f (XEN) [ 0.213177] 0000080e0f000-0000080e0ffff type=4 attr=000000000000000f (XEN) [ 0.218527] 0000080e10000-0000080e10fff type=7 attr=000000000000000f (XEN) [ 0.223898] 0000080e11000-0000080e11fff type=4 attr=000000000000000f (XEN) [ 0.229178] 0000080e12000-0000080e17fff type=7 attr=000000000000000f (XEN) [ 0.234557] 0000080e18000-0000080e18fff type=4 attr=000000000000000f (XEN) [ 0.239955] 0000080e19000-0000080e1afff type=7 attr=000000000000000f (XEN) [ 0.245278] 0000080e1b000-0000080e1dfff type=4 attr=000000000000000f (XEN) [ 0.250609] 0000080e1e000-0000080e27fff type=7 attr=000000000000000f (XEN) [ 0.256015] 0000080e28000-0000080e29fff type=4 attr=000000000000000f (XEN) [ 0.261386] 0000080e2a000-0000080e31fff type=7 attr=000000000000000f (XEN) [ 0.266747] 0000080e32000-0000080f45fff type=1 attr=000000000000000f (XEN) [ 0.272089] 0000080f46000-0000080f48fff type=7 attr=000000000000000f (XEN) [ 0.277471] 0000080f49000-0000080f49fff type=4 attr=000000000000000f (XEN) [ 0.282720] 0000080f4a000-0000080f4efff type=7 attr=000000000000000f (XEN) [ 0.287897] 0000080f4f000-000008106bfff type=4 attr=000000000000000f (XEN) [ 0.293071] 000008106c000-000008106cfff type=7 attr=000000000000000f (XEN) [ 0.298236] 000008106d000-000008107cfff type=4 attr=000000000000000f (XEN) [ 0.303326] 000008107d000-000008107dfff type=7 attr=000000000000000f (XEN) [ 0.308434] 000008107e000-0000081089fff type=4 attr=000000000000000f (XEN) [ 0.313561] 000008108a000-000008108dfff type=7 attr=000000000000000f (XEN) [ 0.318699] 000008108e000-000008108efff type=2 attr=000000000000000f (XEN) [ 0.323797] 000008108f000-00000822abfff type=4 attr=000000000000000f (XEN) [ 0.328938] 00000822ac000-00000822b7fff type=7 attr=000000000000000f (XEN) [ 0.334168] 00000822b8000-00000822d7fff type=4 attr=000000000000000f (XEN) [ 0.339345] 00000822d8000-00000822defff type=7 attr=000000000000000f (XEN) [ 0.344472] 00000822df000-00000822ecfff type=4 attr=000000000000000f (XEN) [ 0.349607] 00000822ed000-00000822edfff type=7 attr=000000000000000f (XEN) [ 0.354808] 00000822ee000-00000822f0fff type=4 attr=000000000000000f (XEN) [ 0.359956] 00000822f1000-00000822f1fff type=7 attr=000000000000000f (XEN) [ 0.365087] 00000822f2000-00000822f2fff type=4 attr=000000000000000f (XEN) [ 0.370248] 00000822f3000-00000822f4fff type=7 attr=000000000000000f (XEN) [ 0.375416] 00000822f5000-0000082302fff type=4 attr=000000000000000f (XEN) [ 0.380540] 0000082303000-0000082303fff type=7 attr=000000000000000f (XEN) [ 0.385675] 0000082304000-00000826effff type=4 attr=000000000000000f (XEN) [ 0.390850] 00000826f0000-00000826f0fff type=7 attr=000000000000000f (XEN) [ 0.396058] 00000826f1000-00000826f1fff type=4 attr=000000000000000f (XEN) [ 0.401211] 00000826f2000-00000826f6fff type=7 attr=000000000000000f (XEN) [ 0.406336] 00000826f7000-0000082705fff type=4 attr=000000000000000f (XEN) [ 0.411517] 0000082706000-0000082709fff type=7 attr=000000000000000f (XEN) [ 0.416646] 000008270a000-0000082715fff type=4 attr=000000000000000f (XEN) [ 0.421771] 0000082716000-0000082716fff type=7 attr=000000000000000f (XEN) [ 0.426898] 0000082717000-0000082717fff type=4 attr=000000000000000f (XEN) [ 0.432073] 0000082718000-000008271bfff type=7 attr=000000000000000f (XEN) [ 0.437223] 000008271c000-0000082726fff type=4 attr=000000000000000f (XEN) [ 0.442350] 0000082727000-0000082727fff type=7 attr=000000000000000f (XEN) [ 0.447518] 0000082728000-00000829d3fff type=4 attr=000000000000000f (XEN) [ 0.452657] 00000829d4000-00000829d5fff type=7 attr=000000000000000f (XEN) [ 0.457808] 00000829d6000-000008657efff type=4 attr=000000000000000f (XEN) [ 0.462933] 000008657f000-0000086ffffff type=7 attr=000000000000000f (XEN) [ 0.468153] 0000087000000-00000879fffff type=2 attr=000000000000000f (XEN) [ 0.473293] 0000087a00000-0000087a60fff type=7 attr=000000000000000f (XEN) [ 0.478419] 0000087a61000-000008857efff type=3 attr=000000000000000f (XEN) [ 0.483564] 000008857f000-000008e57efff type=0 attr=000000000000000f (XEN) [ 0.488773] 000008e57f000-000008e67efff type=9 attr=000000000000000f (XEN) [ 0.493899] 000008e67f000-000009067efff type=10 attr=000000000000000f (XEN) [ 0.499120] Setting RUNTIME attribute for 000008e67f000-000009067efff (XEN) [ 0.504370] 000009067f000-000009867efff type=6 attr=800000000000000f (XEN) [ 0.509722] 000009867f000-00000987fefff type=5 attr=800000000000000f (XEN) [ 0.515177] 00000987ff000-0000098dfffff type=4 attr=000000000000000f (XEN) [ 0.520508] 0000098e00000-0000098f0cfff type=7 attr=000000000000000f (XEN) [ 0.525650] 0000098f0d000-000009900cfff type=4 attr=000000000000000f (XEN) [ 0.530858] 000009900d000-0000099031fff type=3 attr=000000000000000f (XEN) [ 0.536163] 0000099032000-0000099066fff type=4 attr=000000000000000f (XEN) [ 0.541553] 0000099067000-0000099098fff type=3 attr=000000000000000f (XEN) [ 0.546824] 0000099099000-00000990b1fff type=4 attr=000000000000000f (XEN) [ 0.552024] 00000990b2000-00000990c9fff type=3 attr=000000000000000f (XEN) [ 0.557305] 00000990ca000-0000099e5cfff type=4 attr=000000000000000f (XEN) [ 0.562632] 0000099e5d000-0000099e60fff type=3 attr=000000000000000f (XEN) [ 0.567916] 0000099e61000-0000099e74fff type=4 attr=000000000000000f (XEN) [ 0.573234] 0000099e75000-0000099e76fff type=3 attr=000000000000000f (XEN) [ 0.578622] 0000099e77000-0000099e87fff type=4 attr=000000000000000f (XEN) [ 0.583992] 0000099e88000-0000099eacfff type=3 attr=000000000000000f (XEN) [ 0.589439] 0000099ead000-0000099eaefff type=4 attr=000000000000000f (XEN) [ 0.594698] 0000099eaf000-0000099eaffff type=7 attr=000000000000000f (XEN) [ 0.600041] 0000099eb0000-0000099ed3fff type=4 attr=000000000000000f (XEN) [ 0.605343] 0000099ed4000-0000099ed4fff type=7 attr=000000000000000f (XEN) [ 0.610676] 0000099ed5000-0000099ed5fff type=3 attr=000000000000000f (XEN) [ 0.615920] 0000099ed6000-0000099fe7fff type=4 attr=000000000000000f (XEN) [ 0.621323] 0000099fe8000-0000099feefff type=3 attr=000000000000000f (XEN) [ 0.626683] 0000099fef000-0000099ff6fff type=4 attr=000000000000000f (XEN) [ 0.632047] 0000099ff7000-0000099ffbfff type=6 attr=800000000000000f (XEN) [ 0.637343] 0000099ffc000-0000099ffffff type=4 attr=000000000000000f (XEN) [ 0.642527] 0000100000000-000083dd7ffff type=7 attr=000000000000000f (XEN) [ 0.647838] 00000000a0000-00000000fffff type=0 attr=000000000000000f (XEN) [ 0.653159] 000009a000000-000009bffffff type=0 attr=000000000000000f (XEN) [ 0.658442] 000009d7f3000-000009fffffff type=0 attr=000000000000000f (XEN) [ 0.663903] 00000e0000000-00000efffffff type=11 attr=800000000000100d (XEN) [ 0.669306] 00000fd000000-00000fedfffff type=11 attr=800000000000100d (XEN) [ 0.674614] 00000fee00000-00000fee00fff type=11 attr=8000000000000001 (XEN) [ 0.679860] 00000fee01000-00000ffffffff type=11 attr=800000000000100d (XEN) [ 0.685349] 000083edc0000-000085fffffff type=0 attr=000000000000000f (XEN) [ 0.690723] 0000860000000-00008801fffff type=11 attr=800000000000100d (XEN) [ 0.696002] 000fd00000000-000ffffffffff type=0 attr=0000000000000001 (XEN) [ 0.701240] alt table ffff82d0406a36b8 -> ffff82d0406b2cc2 (XEN) [ 0.716324] AMD-Vi: IOMMU Extended Features: (XEN) [ 0.721540] - Peripheral Page Service Request (XEN) [ 0.726851] - NX bit (XEN) [ 0.732090] - Guest APIC Physical Processor Interrupt (XEN) [ 0.737254] - Invalidate All Command (XEN) [ 0.742331] - Guest APIC (XEN) [ 0.747400] - Performance Counters (XEN) [ 0.752763] - Host Address Translation Size: 0x2 (XEN) [ 0.758029] - Guest Address Translation Size: 0 (XEN) [ 0.763019] - Guest CR3 Root Table Level: 0x1 (XEN) [ 0.768155] - Maximum PASID: 0xf (XEN) [ 0.773215] - SMI Filter Register: 0x1 (XEN) [ 0.778313] - SMI Filter Register Count: 0x1 (XEN) [ 0.783096] - Guest Virtual APIC Modes: 0x1 (XEN) [ 0.787987] - Dual PPR Log: 0x2 (XEN) [ 0.792751] - Dual Event Log: 0x2 (XEN) [ 0.797451] - Secure ATS (XEN) [ 0.802045] - User / Supervisor Page Protection (XEN) [ 0.806599] - Device Table Segmentation: 0x3 (XEN) [ 0.811207] - PPR Log Overflow Early Warning (XEN) [ 0.815780] - PPR Automatic Response (XEN) [ 0.820453] - Memory Access Routing and Control: 0x1 (XEN) [ 0.825073] - Block StopMark Message (XEN) [ 0.829495] - Performance Optimization (XEN) [ 0.833916] - MSI Capability MMIO Access (XEN) [ 0.838472] - Guest I/O Protection (XEN) [ 0.842869] - Enhanced PPR Handling (XEN) [ 0.847365] - Invalidate IOTLB Type (XEN) [ 0.851582] - VM Table Size: 0x2 (XEN) [ 0.855807] - Guest Access Bit Update Disable (XEN) [ 0.868536] AMD-Vi: IOMMU 0 Enabled. (XEN) [ 0.872824] I/O virtualisation enabled (XEN) [ 0.876896] - Dom0 mode: Relaxed (XEN) [ 0.881009] Interrupt remapping enabled (XEN) [ 0.885044] Enabled directed EOI with ioapic_ack_old on! (XEN) [ 0.889092] Enabling APIC mode. Using 2 I/O APICs (XEN) [ 0.894056] ENABLING IO-APIC IRQs (XEN) [ 0.898035] -> Using old ACK method (XEN) [ 0.902257] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) [ 1.265651] Defaulting to alternative key handling; send 'A' to switch to normal mode. (XEN) [ 1.269685] Allocated console ring of 128 KiB. (XEN) [ 1.273661] HVM: ASIDs enabled. (XEN) [ 1.277575] SVM: Supported advanced features: (XEN) [ 1.281493] - Nested Page Tables (NPT) (XEN) [ 1.285369] - Last Branch Record (LBR) Virtualisation (XEN) [ 1.289264] - Next-RIP Saved on #VMEXIT (XEN) [ 1.293106] - VMCB Clean Bits (XEN) [ 1.296889] - DecodeAssists (XEN) [ 1.300620] - Virtual VMLOAD/VMSAVE (XEN) [ 1.304329] - Virtual GIF (XEN) [ 1.307975] - Pause-Intercept Filter (XEN) [ 1.311606] - Pause-Intercept Filter Threshold (XEN) [ 1.315234] - TSC Rate MSR (XEN) [ 1.318788] - NPT Supervisor Shadow Stack (XEN) [ 1.322359] - MSR_SPEC_CTRL virtualisation (XEN) [ 1.325930] HVM: SVM enabled (XEN) [ 1.329463] HVM: Hardware Assisted Paging (HAP) detected (XEN) [ 1.333067] HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) [ 1.336829] alt table ffff82d0406a36b8 -> ffff82d0406b2cc2 (XEN) [ 1.971969] Brought up 32 CPUs (XEN) [ 1.975587] Testing NMI watchdog on all CPUs:ok (XEN) [ 1.998076] Scheduling granularity: cpu, 1 CPU per sched-resource (XEN) [ 2.001828] mcheck_poll: Machine check polling timer started. (XEN) [ 2.025601] NX (Execute Disable) protection active (XEN) [ 2.029288] d0 has maximum 1304 PIRQs (XEN) [ 2.033063] csched_alloc_domdata: setting dom 0 as the privileged domain (XEN) [ 2.036864] *** Building a PV Dom0 *** (XEN) [ 2.159016] Xen kernel: 64-bit, lsb (XEN) [ 2.162750] Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x302c000 (XEN) [ 2.166867] PHYSICAL MEMORY ARRANGEMENT: (XEN) [ 2.170718] Dom0 alloc.: 0000000820000000->0000000824000000 (642482 pages to be allocated) (XEN) [ 2.174954] Init. ramdisk: 000000083a9b2000->000000083dbff800 (XEN) [ 2.179078] VIRTUAL MEMORY ARRANGEMENT: (XEN) [ 2.183059] Loaded kernel: ffffffff81000000->ffffffff8302c000 (XEN) [ 2.187120] Phys-Mach map: 0000008000000000->0000008000520000 (XEN) [ 2.191244] Start info: ffffffff8302c000->ffffffff8302c4b8 (XEN) [ 2.195323] Page tables: ffffffff8302d000->ffffffff8304a000 (XEN) [ 2.199479] Boot stack: ffffffff8304a000->ffffffff8304b000 (XEN) [ 2.203621] TOTAL: ffffffff80000000->ffffffff83400000 (XEN) [ 2.207816] ENTRY ADDRESS: ffffffff8242b180 (XEN) [ 2.212274] Dom0 has maximum 8 VCPUs (XEN) [ 2.227159] Initial low memory virq threshold set at 0x4000 pages. (XEN) [ 2.231412] Scrubbing Free RAM in background (XEN) [ 2.235639] Std. Loglevel: Errors, warnings and info (XEN) [ 2.240056] Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) [ 2.244590] Xen is relinquishing VGA console. (XEN) [ 2.259091] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) (XEN) [ 2.259238] Freed 2048kB init memory [09:53 xcp-ng-AsRoCk ~]#Pci devices that I have Passthrough enabled disappear from Xcp-ng after it hangs after reboot. and several pci devices with other pci numbers appear in xl pci-assignable-list
09:38 xcp-ng-AsRoCk ~]# lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14d8 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 14d9 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da 00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db 00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e5 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e6 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e7 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57502 NetXtreme-E 10Gb/25Gb/40Gb/50Gb Ethernet (rev 12) 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57502 NetXtreme-E 10Gb/25Gb/40Gb/50Gb Ethernet (rev 12) 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port (rev 01) 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 03:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 07:00.0 PCI bridge: ASRock Incorporation Device 1150 (rev 06) 08:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52) 0b:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller (rev 01) 0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01) 0d:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. FURY Renegade NVMe SSD with heatsink (rev 01) 0e:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 13c0 (rev c1) 0e:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller 0e:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP 0e:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b6 0e:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b7 0e:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 62) 0e:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller 0f:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8 [09:39 xcp-ng-AsRoCk ~]# xl pci-assignable-list 0000:03:00.0 0000:08:00.0 0000:07:00.0 0000:04:00.0 [09:40 xcp-ng-AsRoCk ~]#
