Posts made by tuxen

This got my attention:

Jan 15 19:17:40 xcp-ng-xen12-lon2 xapi: [error||623653 INET :::80||import] Caught exception in import handler: VDI_IO_ERROR: [ Device I/O errors ]
Jan 15 19:17:40 xcp-ng-xen12-lon2 xapi: [error||623653 INET :::80||backtrace] VDI.import D:378e6880299b failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
Jan 15 19:17:40 xcp-ng-xen12-lon2 xapi: [error||623653 INET :::80||backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")

This Unix.EPIPE error on the remote target means that the pipe stream is being closed before VDI.Import receives all the data. The outcome is a VDI I/O error due to a broken, partial sent/received VDI.

Since a remote-over-the-internet link can be more prone to latency/intermittency issues, it might be needed to adjust the remote NFS soft timeout/retries or mounting the target with hard option.

I would also check if the remote target is running out-of-space during the backup process.

Could the fcoe driver causing the issue?

dmesg:

[   42.363389] bnx2fc: QLogic FCoE Driver bnx2fc v2.12.5 (November 16, 2018)
[   42.371336] bnx2fc: FCoE initialized for eth1.
[   42.371641] bnx2fc: [04]: FCOE_INIT passed
[   42.387017] bnx2fc: FCoE initialized for eth0.
[   42.387305] bnx2fc: [04]: FCOE_INIT passed

lsmod:

fcoe                   32768  0 
libfcoe                77824  2 fcoe,bnx2fc
libfc                 147456  3 fcoe,bnx2fc,libfcoe
scsi_transport_fc      69632  3 fcoe,libfc,bnx2fc

@erfant probably not because the nvme driver is loaded and there're no nvme errors in the logs.

@olivierlambert thank you and your team for this great project and community! It's a nice place to share knowledge and learn new stuff. I learn a lot here!

@erfant after seeing your uploaded dmesg, the steps 2 & 3 boot options can be put aside for while because the error isn't the same as the other topics.

The log is showing MxGPU driver probe/initialization errors. After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

[  119.418930]        gim error:(gim_probe:123) gim_probe(08:00.0)
[  121.145663]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003044131 sec
[  121.145719]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0
[  121.145984]        gim error:(init_register_init_state:4643) Failed to INIT PF for initial register 'init-state'

Edited for clarification.

@erfant could you upload lspci -k and dmesg ? Also, have you checked/tried:

1. SR-IOV enabled in BIOS;
2. On vmlinuz entry, boot with pci=realloc;
3. On vmlinuz entry, boot with pci=realloc pci=assign-busses

@Appollonius said in Strange issue with booting XCP-NG:

Its only when I install the GPU and dont connect it to a monitor that it will not boot properly.

Maybe because, when there's a GPU installed but no monitor attached, the motherboard POST fails at EDID probe? As stated, some boards/BIOS require an explicit configuration in order to boot without a monitor/keyboard/mouse plugged, eg.:

https://www.supermicro.com/support/faqs/faq.cfm?faq=11902

@Appollonius check the BIOS for display emulation or ACPI configuration. Some boards may refuse to boot If the BIOS isn't properly configured to accept the headless mode.

It's from the terminal/CLI. Alternatively, you can verify/change the boot options in /boot/grub/grub.cfg (for dom0 boot, see module2 /boot/vmlinuz entries).

Found this Citrix KB adding one more pci option, take a look:
https://support.citrix.com/article/CTX250121

@tbluml did you try the pci=realloc workaround, as stated in the RHEL link?

# /opt/xensource/libexec/xen-cmdline --set-dom0 pci=realloc

Edit: reboot the host after applying the change.

I liked as well. Easy to find the topics and good layout

@olivierlambert congrats to the team and also to this great community!

@cyrus104 said in XOA guest tools error:

Thoughts on some ways to fix this?

The problem was semantics... The command you really wanted was xe vm-cd-eject. Issuing a xe vm-cd-remove not only removes the .ISO but also the DVD-drive. In this case, you must recreate the device. It's possible to do so through the CLI, but it can be a bit tricky because it's needed to identify a free device position... IMHO the easier way is via XCP-ng Center:

Image from https://support.citrix.com/article/CTX132411:

@stormi Yes, after trying to save the changes, an error banner appeared with the message "Your account has been flagged". It was indeed a policy for new accounts, according to the github support team. I'll try again, thanks!

@stormi I've just created a github account but it was flagged right after trying to edit the wiki

Is there some sort of permission?

FYI, I couldn't edit the wiki without an account. The edit button only appears after signed in.

@enes-selcuk

This is one of our production servers with the CPU Power Management config I posted earlier. It's a Dell R730 - Xeon E5-2630v3 @ 2.40 Ghz. As you can see, some cores are boosting to 3.0-3.1GHz. The maximum for this CPU model is 3.2 GHz:

  Avg freq	2737140	KHz
  Avg freq	2737140	KHz
  Avg freq	2689120	KHz
  Avg freq	2665110	KHz
  Avg freq	2737140	KHz
  Avg freq	3097290	KHz
  Avg freq	3073280	KHz
  Avg freq	2713130	KHz
  Avg freq	2665110	KHz
  Avg freq	2641100	KHz
  Avg freq	2761150	KHz
  Avg freq	2665110	KHz
  Avg freq	2689120	KHz
  Avg freq	2617090	KHz
  Avg freq	2881200	KHz
  Avg freq	2785160	KHz
  Avg freq	2641100	KHz
  Avg freq	2713130	KHz
  Avg freq	2689120	KHz
  Avg freq	2689120	KHz
  Avg freq	2689120	KHz
  Avg freq	2785160	KHz
  Avg freq	2833180	KHz
  Avg freq	2761150	KHz
  Avg freq	2761150	KHz
  Avg freq	2905210	KHz
  Avg freq	2713130	KHz
  Avg freq	3025260	KHz
  Avg freq	2761150	KHz
  Avg freq	2713130	KHz
  Avg freq	2833180	KHz
  Avg freq	2617090	KHz

The incorrect clock results mean that Xen isn't in charge of frequency scaling management. Set the CPU Power Management to Performance Per Watt (OS) and run the previous xenpm, this time with a watch for real-time:

# watch 'xenpm start 1 | grep -i "avg freq"'

Start a VM boot storm (or a stress test inside one or more VMs) in order to generate some CPU load.

@enes-selcuk

What's the output of:

# xenpm start 1 | grep -i "avg freq"

@Silencer80

No, my systems are Intel-based but might work also for AMD processors. The only caveat is whether AMD fixed the C-STATES/MWAIT bug causing spontaneous reboots on the 1st gen architecture. Since ROME is a new arch, maybe it's not affected. The only way to be sure is enabling C-STATES and keep an eye on it.

That Citrix KB is my go-to reference about Xen power management. Very good article.

Kudos to XCP-ng team!

@Silencer80

Before installing Xen, the first thing I change is the BIOS power profile settings. In my experience, it works pretty well with Performance and OS Control. For production, my choice is the latter because it gives to Xen the core frequency scaling management (boost) and saves energy when there's a low demand after hours.

@nmym said in AMD Eypc 7402P ?:

When I set this to ondemand, my network speed drops. It's somehow ironic as I have the demand for this speed, but the process does not demand, resulting in slower network transfer

For benchmark purposes, try reducing the dom0 #vcpus (4 is good value IMHO for the tests). Since it's a high core count processor, a 8+ dom0 cores is good for hosting 100+ VMs but it might distort benchmark results because the packet processing will be spread across all dom0 cores. With the CPU governor set to ondemand the more cores you have the less will be the need to boost. This behavior doesn't occur in performance due to the cores being at the base clock minimum frequency.