I don't even know how this is possible but for some reason the xen-cmdline file was empty. I've never touched that file. But I did have another xcpng pool that I checked its xen-cmdline and copied it to my other pool and restarted the XAPI service and BAM! passthrough is working.

I don't even know how this is possible but for some reason the xen-cmdline file was empty. I've never touched that file. But I did have another xcpng pool that I checked its xen-cmdline and copied it to my other pool and restarted the XAPI service and BAM! passthrough is working.

I have been trying to pass through my GPU on XCPNG 8.3 for a while now. When I have my nvidia GPU installed the XAPI crashes and constantly tries to start over and over. When I don't have the GPU connected the XAPI service runs fine. I do have IOMMU and virtualization enabled.

XAPI Logs:

ct  1 20:09:21 xcp-ng-01 message-switch[2733]: main: [ info|message-switch] Session xapi:15696 cleaning up
Oct  1 20:09:24 xcp-ng-01 forkexecd: [error||0 ||forkexecd] execve failed: Unix.Unix_error(Unix.ENOEXEC, "execve", "/opt/xensource/libexec/xen-cmdline")
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.616Z||2731|About to bind to /var//run/nonpersistent/forkexecd//fd_8db15b9d-6f4d-2fe6-823c-ff710db962b0\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.616Z||2731|bound, listening\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.620Z||17064|Child here!\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.625Z||17065|Grandchild here!\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.625Z||17065|Started: state.cmdargs = [/opt/xensource/libexec/xen-cmdline;--get-dom0;xen-pciback.hide]\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.625Z||17065|Started: state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin]\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.626Z||17065|Selecting in handle_comms_no_fd_sock2\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Done\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|fd sock\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Selecting in handle_comms_with_fd_sock2\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Done\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|fd sock2\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Received fd named: 4207af65-3d99-3f64-4464-86cf6c96672e - duping to 1 (from 8)\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Selecting in handle_comms_with_fd_sock2\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Done\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|fd sock2\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Received fd named: 4a8a3659-34d4-1763-11a6-9296cc393b61 - duping to 2 (from 8)\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.627Z||17065|Selecting in handle_comms_with_fd_sock2\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.628Z||17065|Done\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.628Z||17065|comms sock\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.628Z||17065|Exec\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.628Z||17065|Finished...\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.628Z||17065|Args after replacement = [/opt/xensource/libexec/xen-cmdline;--get-dom0;xen-pciback.hide]\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.628Z||17065|I've received the following fds: [2;1]\x0A\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [ warn||0 ||forkexecd] 20251002T01:09:24.653Z||17066|Caught unexpected exception: Unix.Unix_error(Unix.ENOEXEC, "execve", "/opt/xensource/libexec/xen-cmdline")\x0A
Oct  1 20:09:24 xcp-ng-01 forkexecd: [error||0 ||forkexecd] 17066 (/opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide) exited with code 126
Oct  1 20:09:24 xcp-ng-01 systemd[1]: xapi.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct  1 20:09:24 xcp-ng-01 xapi-init[17069]: Stopping xapi: [  OK  ]
Oct  1 20:09:24 xcp-ng-01 systemd[1]: Unit xapi.service entered failed state.
Oct  1 20:09:24 xcp-ng-01 systemd[1]: xapi.service failed.
Oct  1 20:09:25 xcp-ng-01 systemd[1]: xapi.service holdoff time over, scheduling restart.
Oct  1 20:09:25 xcp-ng-01 systemd[1]: Cannot add dependency job for unit lvm2-activation.service, ignoring: Unit is masked.
Oct  1 20:09:25 xcp-ng-01 systemd[1]: Cannot add dependency job for unit lvm2-activation-early.service, ignoring: Unit is masked.

@andriy.sultanov After running that command I didn't have anything returned.

@andriy.sultanov Here are the logs. To your second question. I thought I was able to pass through devices from XO and then the host reboots. Maybe I misunderstood how this worked. Are you saying I need to do this in the console and then XO?

Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] PCI.get_dom0_access_status R:1f40280dd69a failed with exception Server_error(INTERNAL_ERROR, [ 126 ])
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] Raised Server_error(INTERNAL_ERROR, [ 126 ])
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 1/15 xapi Raised at file ocaml/xapi-aux/helper_process.ml, line 17
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 2/15 xapi Called from file ocaml/xapi/pciops.ml, line 73
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 3/15 xapi Called from file ocaml/xapi/pciops.ml, line 95
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 4/15 xapi Called from file ocaml/xapi/pciops.ml, line 99
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 5/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 6/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 7/15 xapi Called from file ocaml/xapi/xapi_pci_helpers.ml, line 213
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 8/15 xapi Called from file ocaml/xapi/message_forwarding.ml, line 141
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 9/15 xapi Called from file ocaml/xapi/rbac.ml, line 188
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 10/15 xapi Called from file ocaml/xapi/rbac.ml, line 197
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 11/15 xapi Called from file ocaml/xapi/server_helpers.ml, line 77
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 12/15 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 13/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 14/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace] 15/15 xapi Called from file ocaml/libs/log/debug.ml, line 250
Jul  7 16:42:08 xcp-ng-01 xapi: [error||62070 :::80|dispatch:PCI.get_dom0_access_status D:7ee7c7e624f9|backtrace]

@TeddyAstie I ran /opt/xensource/libexec/xen-cmdline --get-dom0 on the host and it didn't return anything.

EDIT: I also want to mention that I upgraded from 8.2 to 8.3 is that makes any difference. But I only started to testing passing through devices after I upgraded.

@olivierlambert I have SR-IOV enabled in the BIOS.

@olivierlambert I was actually using the official XOA not from source, but I do have source instance installed with the commit of c8f9d and I have having the same issue.

• XCPNG: 8.3
• node: 20.18.3
• npm: 10.8.3
• xen-orchestra-upload-ova: 0.1.6
• xen-orchestra-web: 0.17.1
• xo-server: 5.177.4
• xo-server-telemetry: 0.7.0
• xo-server-xoa: 0.30.1
• xo-web-free: 5.174.3
• xoa-cli: 0.40.3
• xoa-updater: 0.50.10

When I go to pass through a device I get the following error on the screen.

I was curious if there are going to be any improvements to the hub and templets in XO? Maybe have a community repo and Enterprise repo for prebuilt base images for OS's. Or maybe have prebuilt applications? Or maybe have a way where we can setup a template scripts to run, kind of like linode's StackScript's to build applications from the hub.

Just a few ideas to give the hub a bit more action and I think it would be a powerful tool for homelab and enterprise.

PS - I think everyone at Vates is doing a great job expanding XCPNG/XOA.

@uxes Are you able to supply any logs?