I liked as well. Easy to find the topics and good layout

@olivierlambert congrats to the team and also to this great community!

Kudos to XCP-ng team!

@erfant probably not because the nvme driver is loaded and there're no nvme errors in the logs.

@olivierlambert thank you and your team for this great project and community! It's a nice place to share knowledge and learn new stuff. I learn a lot here!

@erfant after seeing your uploaded dmesg, the steps 2 & 3 boot options can be put aside for while because the error isn't the same as the other topics.

The log is showing MxGPU driver probe/initialization errors. After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

[  119.418930]        gim error:(gim_probe:123) gim_probe(08:00.0)
[  121.145663]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003044131 sec
[  121.145719]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0
[  121.145984]        gim error:(init_register_init_state:4643) Failed to INIT PF for initial register 'init-state'

Edited for clarification.

@Appollonius said in Strange issue with booting XCP-NG:

Its only when I install the GPU and dont connect it to a monitor that it will not boot properly.

Maybe because, when there's a GPU installed but no monitor attached, the motherboard POST fails at EDID probe? As stated, some boards/BIOS require an explicit configuration in order to boot without a monitor/keyboard/mouse plugged, eg.:

https://www.supermicro.com/support/faqs/faq.cfm?faq=11902

The incorrect clock results mean that Xen isn't in charge of frequency scaling management. Set the CPU Power Management to Performance Per Watt (OS) and run the previous xenpm, this time with a watch for real-time:

# watch 'xenpm start 1 | grep -i "avg freq"'

Start a VM boot storm (or a stress test inside one or more VMs) in order to generate some CPU load.

Could the fcoe driver causing the issue?

dmesg:

[   42.363389] bnx2fc: QLogic FCoE Driver bnx2fc v2.12.5 (November 16, 2018)
[   42.371336] bnx2fc: FCoE initialized for eth1.
[   42.371641] bnx2fc: [04]: FCOE_INIT passed
[   42.387017] bnx2fc: FCoE initialized for eth0.
[   42.387305] bnx2fc: [04]: FCOE_INIT passed

lsmod:

fcoe                   32768  0 
libfcoe                77824  2 fcoe,bnx2fc
libfc                 147456  3 fcoe,bnx2fc,libfcoe
scsi_transport_fc      69632  3 fcoe,libfc,bnx2fc

@erfant probably not because the nvme driver is loaded and there're no nvme errors in the logs.

@olivierlambert thank you and your team for this great project and community! It's a nice place to share knowledge and learn new stuff. I learn a lot here!

@erfant after seeing your uploaded dmesg, the steps 2 & 3 boot options can be put aside for while because the error isn't the same as the other topics.

The log is showing MxGPU driver probe/initialization errors. After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

[  119.418930]        gim error:(gim_probe:123) gim_probe(08:00.0)
[  121.145663]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003044131 sec
[  121.145719]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0
[  121.145984]        gim error:(init_register_init_state:4643) Failed to INIT PF for initial register 'init-state'

Edited for clarification.

@erfant could you upload lspci -k and dmesg ? Also, have you checked/tried:

1. SR-IOV enabled in BIOS;
2. On vmlinuz entry, boot with pci=realloc;
3. On vmlinuz entry, boot with pci=realloc pci=assign-busses

@Appollonius said in Strange issue with booting XCP-NG:

Its only when I install the GPU and dont connect it to a monitor that it will not boot properly.

Maybe because, when there's a GPU installed but no monitor attached, the motherboard POST fails at EDID probe? As stated, some boards/BIOS require an explicit configuration in order to boot without a monitor/keyboard/mouse plugged, eg.:

https://www.supermicro.com/support/faqs/faq.cfm?faq=11902

@Appollonius check the BIOS for display emulation or ACPI configuration. Some boards may refuse to boot If the BIOS isn't properly configured to accept the headless mode.

It's from the terminal/CLI. Alternatively, you can verify/change the boot options in /boot/grub/grub.cfg (for dom0 boot, see module2 /boot/vmlinuz entries).

Found this Citrix KB adding one more pci option, take a look:
https://support.citrix.com/article/CTX250121

@tbluml did you try the pci=realloc workaround, as stated in the RHEL link?

# /opt/xensource/libexec/xen-cmdline --set-dom0 pci=realloc

Edit: reboot the host after applying the change.

I liked as well. Easy to find the topics and good layout

@olivierlambert congrats to the team and also to this great community!