@peder No, but let me ping our XAPI developers @gthvn1 and @andriy.sultanov

I'd start with checking the usual logs (notably xensource.log) when the issue happens.

See https://docs.xcp-ng.org/troubleshooting/log-files/

Yes, I know, getting logs about an issue getting logs. Of course, I mean checking them via CLI on XCP-ng hosts :).

@MJGZ51 said in XCP-ng 8.3 PCI Passthrough issue:

For anyone experiencing this issue, it happens when you enable Passthrough via CLI (hide devices from DOM).

After a clean install of xcp-ng 8.3, Passthrough would work via XOA GUI. However, as soon as I enabled PCI Passthrough for a device via CLI, the errors appeared.

I've moved to just passing the devices via GUI and not CLI. Everything is now working.

What's the involved CLI command?

@archw It's written in the blog post

@bvitnik It's better to consider it as working by chance in some scenarios, but I think Teddy can give you details

The default templates are all re-created from JSON files on the filesystem everytime the guest-templates-json* RPMs are updated in dom0.

@psylo xensource.log and daemon.log will usually give more information about failed operations. It could also be an issue in XOA's warm migration code in specific cases.

There's no built-in feature to backup changes made to dom0 outside changes made via XAPI, so configuration as code is probably the best option here.

@gb.123 I doesn't seem directly related to this thread, which is dedicated to testing update candidates. Maybe open a new thread?

@psylo We are working on updating XAPI in XCP-ng 8.3. When that's done, a simple yum update should be enough to recover. But for now @ricardowz's or @reiichi001's way is safer.

When we released XCP-ng 8.3, migration from XenServer 8 did work. However, XS8 has kept on evolving, rolling-release style, so it's possible that the current installation ISOs for XCP-ng 8.3 are not recent enough to migrate a current day XenServer 8.

However, the problems I would expect are not the ones you describe. I would expect the upgrade process to go smoothtly, and then the XAPI service fail to start because our version of XAPI sees a database that is too recent.

If no choices are displayed to upgrade an existing XenServer 8 with the 8.3 ISO, the reason will be provided in the logs in /tmp/install-log (ALT+RIGHT then view /tmp/install-log, IIRC). It can be several things: a first boot process not complete on XS side, booting in BIOS mode when the system is currently UEFI, ... But there's also the possibility that the platform number we use to detect an upgradeable product has changed.

@Mefosheez Marking a new post as a question will not be necessary. @anthonyper is the one working on what we think is the issue (a regression on the EDK2 package). I haven't heard of a fix yet, but it's in progress.

XenServer 8 is actually XenServer 8.3 (or 8.4 but let's not complicate things, for our matter here 8.3 is good enough).

So you can migrate it to XCP-ng 8.3 but not to XCP-ng 8.2.1.

I think this part of the doc describes your issue: https://docs.xcp-ng.org/releases/release-8-3/#a-uefi-vm-started-once-on-xcp-ng-83-cant-start-if-moved-back-to-xcp-ng-821

I'm sure we have forum threads about I210 already, but the forum's search engine is as useless as usual, trying to be smarter and not respecting search words when they're not in its dictionary

The Linux driver for I210 is igb and I think it usually works.

Which version of XCP-ng are you trying?

We have an intel-igb-alt driver RPM that you can install (see https://docs.xcp-ng.org/installation/hardware/#alternate-drivers). On 8.2, it provides a newer version of the driver. On XCP-ng 8.3, on the contrary, you get an older version, but that can still be an interesting test. See https://github.com/xcp-ng/xcp/wiki/Drivers for the available driver versions.

Pinging @bleader for the network part, with help from @gduperrey to navigate among the existing drivers. @Andrew might have pieces of insight, too.

@Mefosheez I forwarded internally. Feel free to ask again after a week if no answer comes.

What does mdadm --detail /dev/md127 output on either hosts?

@kagbasi-ngc Does your VM have a vTPM? If so, could you try without?

@kagbasi-ngc @Mefosheez We're trying to find developer time to diagnose the cause, but it's not the best time of the year, so I can't promise anything on delays.

@kagbasi-ngc Looks like we missed something, as the log indicates it loaded OVMF-release.fd:

Dec 13 13:54:59 VMH01 xenguest-2-build[8345]: Loaded OVMF from /usr/share/edk2/OVMF-release.fd

We'll come back to you with a better test procedure.

For now I just have a quick and dirty one, consisting in overwriting the release file with the debug one:

cp /usr/share/edk2/OVMF-debug.fd /usr/share/edk2/OVMF-release.fd

You can revert this change with yum reinstall edk2.