Found it. Error is related to https://github.com/xapi-project/xen-api/issues/4141

Somehow, the PCIe bus address of the server's ASPEED VGA adapter changed.

I got the new bus address with lspci and edited state.db accordingly. Works now. However, this seems to be a very strange bug.

stormi created this issue in xapi-project/xen-api open XAPI failing to initialize after GPU removal? #4141

You are welcome. Weird issue indeed 🤔

Good news 🙂 Also, now you are in HVM, which is a LOT better than PV for a lot of reasons (security, performance etc.)

@Marcsteven said in Troubleshooting System Interrupts issue:

To resolve troubleshooting system interrupts errors, many methods can be executed. You can maybe disable sound effects. To do that, you can go to the taskbar and right-click the speaker icon. Then double-click your Default Device (speaker) > open Properties. Now tap the option of Enhancements and check the box which says ‘Disable all sound effects.’ Now press Ok to save the settings, and we will hope this will resolve the method. If it doesn't work then maybe you can update your PC's Bios by launching the Windows search > write CMD > run Command Prompt and then type these commands one after another > hit before you enter after each command systeminfo | findstr /I /c:bios wmic bios get manufacturer, smbiosbiosversion .

Appreciate the ideas here, but most of it pertains to desktop machines not virtual guest instances (upgrading the PC BIOS is not really applicable for this unless it were occurring on ALL guests, for example).

@tekwendell xen carefully manages CPU power management to match VM load and vCPU count, I would not manually try to adjust things with xenpm in the meantime as it's likely you'll make things worse (don't try to outsmart xen power management unless you have a VERY specific use case). Xen is designed for paralleled workloads (more than a single VM), so there's many tunables for VMs that are set with this in mind (like CPU affinity). So by default I'm sure the CPU affinity for your single windows VM is still set somewhere in the "middle", so it's not going to be allowed to schedule the full CPU time versus what dom0 is also using.

I'm not an expert in AMD/Epyc power management, but I believe it's pretty typical that CPU power/clock management boosts based on overall CPU load, and running a benchmark on only a single VM using something like 8 cores on a 64 core processor is not going to demand a lot CPU time, so I'm not surprised to see it's not boosting very far. Spin up 6 more of those VMs and benchmark them all at the same time, I wouldn't be surprised if you see it start boosting higher

475 cpu-z versus 501 bare metal is very good and indicates pretty clearly there's no issue here, you're getting 94% bare metal performance on windows under a large virtualization stack (historically the OS with the most overhead to virtualize). I would be very happy about this

If you really want to dig further, ensure your bios power management is set to "OS-controlled", this will hand more control over turbo and c-states to the xen power manager and is what is recommended on AMD processors, and then you can use some commands listed here to check actual turbo status. But again, note that I won't be surprised if you can't get a 64-core processor to enter its highest turbo states when only stressing 1/10th of its cores: https://support.citrix.com/article/CTX200390/power-settings-in-citrix-hypervisor-cstates-turbo-and-cpu-frequency-scaling

Okay sqo both disks are on VM_space_H1. Do a xe sr-param-list uuid=<SR UUID>, get the PBDs and for each do a xe pbd-param-list uuid=<PBD UUID> and paste the result 🙂

Ah nice!

Ideally, we'd like to avoid to use the complicated XAPI RRD + XML system. See https://xcp-ng.org/blog/2022/05/25/fetching-metrics-in-xen-why-the-method-matters/

Thanks a lot to bring this, I'll promote it!

PV mode is dead, HVM is faster anyway (with PV drivers).

Note this is not posted in the right category, since we talk mainly about XCP-ng and not Xen in a general distro 🙂

@gsrfan01 Not sure that was ever officially supported, though I did implement a direct connection to a VM on XenServer with better results than with the standard process. You're right, it may or may not work with much newer versions of XCP-NG/XS/CH.

@ganeshb Hi, think twice before going this path, with the xcp and the ubuntu hosting the containers seems to be a very good solution.
You can make good use of xcp features as backups snapshots, if something goes wrong, you can easily restore everything. You shouldn't be accessing dom0 if you aren't 100% sure of what u doing.

I havent used in prod but the coreos + the plugin looks the most native xen docker compatibility since xenserver 7.0 (I think), which would be more towards enterprises solutions.
CoreOS is RedHat based and a bit different than Debian based.

Between Large and small prod environments I'd say you are going fine.
If you want just for a home/small lab, you can try freenas which already comes with a web dashboard.

I hope it helps

@stevewest15 I would also increase your dom0 memory, 16 GB is already pretty much all being used and for migrating VMs, you need memory. Some of the resources are used for dealing with storage, whoch requires such resources. If the migration is slow, run iostat and see what resources are being used; there may be a storage bottleneck, as well.

@ember1205 it's not a matter of not being "well optimized" or xen arbitrarily removing support for old CPUs (we don't do that), it's a matter of the CPUs he's using being so old they're missing entire instruction sets that are required for proper virtualization. The Z800 systems you mention use xeons a full generation newer than the processors he's using, and were the first intel xeon generation to have proper full IOMMU/VT-d/etc instruction set support, so of course I'm sure they worked just great. The Z800 line is the same xeon generation as the 11th gen dells I mentioned, which is why I stated that is the oldest I would comfortably use - any generations older than these (like the poweredges zxcp is using) are missing core instruction sets xen relies upon.

@jaayb Amazing, TYSM!

@Darkbeldin Aaah, thanks for the clarification. Now I know....learned something today.

By the way, I've been so busy at work that I forgot my host has an iLO which I'd previously configured. I used it to gain console access and successfully rest the root credentials. I'm in and I have successfully reconnected XO to the host using the root credentials.

@AlexanderK

The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem. message: xenopsd internal error: (Failure "Error from xenguesthelper: xenguest: xc_dom_build_image: [1] panic: xc_dom_boot.c:159: xc_dom_boot_domU_map: failed ")

hpe dl360p gen8. So its HPE problem?

xen-pciback.hide is precisely removing the devices from the Dom0.

However, xen-cmdline --delete-dom0 will actually REMOVE the parameters you entered in the grub configuration file.

In other words, xen-cmdline is a helper tool to edit Grub configuration file for Xen and Linux arguments (on boot).

As it's written, xen-cmdline --set-dom0 will actually set parameters for the dom0, and xen-cmdline --delete-dom0 will remove them.

@fohdeesha lol :-). ok.... good to now :). Next optimization by decoupling harddisks and fans.... and always be quite on the datacenter :).

I don't really remeber about the solution, but assume I deleted the old logfiles too. I did never observe the same failure again.

I'm also running xcp-ng on a Supermicro X11 with a XEON Silver 4210 CPU.

That would be interesting to get some pointers (like serial debug and so on). I wonder if there's already some resources for Xen Project somewhere 🤔