@olivierlambert This is correct, your logic is spot on. The pFsense VM was soft-migrated with continuous replication. However the XO VM was made afresh from scratch and it was not starting. Anyway, things work as expected now, thanks!

@jeffmetal You are connecting to Dom0. It does not use (or show) all cores/threads, only the 16 configured for it. So, there is no issue as that is correct.

RTFM about Dom0.

@chon It's Jun 3, 2023. After three years, I'm going to finally do the migration to a new Dell server hardware. I'll update this thread for those who are interested. thanks.

@dthenot THANK YOU !!

Check your kernel boot params in Grub, it's probably linked to an existing (or missing) console kernel parameter.

Hello, I'm honestly don't know how Citrix vGPU stuff works, but couple of thoughts on this topic:

If I understand correctly, you say Nvidia use VFIO Linux framework to enable mediated devices which can be exported to guest. The VFIO framework isn't supported by XEN, as VFIO need the presense of IOMMU device managed by IOMMU Linux kernel driver. And XEN doesn't proide/virtualize IOMMU access to dom0 (XEN manages IOMMU by itself, but doesn't offer such access to guests)

Bascally to export SR-IOV virtual function to guest with XEN you don't have to use VFIO, you can just assign the virtual function PCI "bdf" id to guest and normally the guest should see this device.

From what I understand Nvidia user-mode toolstack (scripts & binaries) doesn't JUST create SR-IOV virtual functions, but want to access VFIO/MDEV framework, so all this thing fails.

So may be, you can check if you there's some options with Nvidia tools to just create SR-IOV functions, OR try to run VFIO in "no-iommu" mode (no IOMMU presence in Linux kernel required)

BTW, we working on some project where we are intending to use VFIO with dom0, and so we're implementing the IOMMU driver in dom0 kernel, so it would be interesting to know in the future, if this can help with your case.

Hope this help

@AndreS I mentionned it already but I can repeat again. My system is already installed in 3 partitions: /boot, /boot/efi and the last one is encrypted, inside 2 LVM volumes, one for swap and and the other one as root.

I can't try just xen without LVM and dmcrypt, because my system is already installed.

And no, I am not trying to add 3 in one, without Xen, my system work already with LVM and dmcrypt

For what we heard it works very well and you got up to 20% uplift in the previous generation.

Thanks, passing the compliment to our teams! 👍

Dear,

There is indeed enough memory on the recieving node. We are having nodes with 1TB of memory, and currently they are loaded with around 500GB each. Ill try to reproduce it with a cloned production server. So i can reproduce it a few times with and without network and to a empty receiving node. Ill keep you updated.

@Andrew That's what I thought, but basically wanted confirmation. Are there any settings to look at that may be able to be tuned in XCP to improve responsiveness?

I'm having exactly the same problem with a "Tiger Jet Network Inc. / ICP DAS Tiger3XX Modem/ISDN interface" card.
logs show this:

May 5 16:13:17 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] xapi_cache: creating cache for ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:17 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops_cache: creating empty cache for ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:17 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xapi_network] register_vif vif=OpaqueRef:8b603d12-373e-4086-a03b-e63e45afb21e network=OpaqueRef:de3a7251-596a-4052-a90a-0caa23d93947 May 5 16:13:17 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops: VM.start ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:17 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops: Queueing VM.unpause ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:24 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xapi_network] Caught Xenops_interface.Xenopsd_error([S(Internal_error);S(Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")))]): detaching networks May 5 16:13:24 xcp-ng xapi: [error||412529 |Async.VM.start R:f406c721ea1c|xenops] Caught exception starting VM: Xenops_interface.Xenopsd_error([S(Internal_error);S(Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")))]) May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] VM ae56aad1-bab7-e6eb-8599-8eea93291e54 set_resident_on May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] Signalling xenapi event thread to re-register, and xenopsd events to sync May 5 16:13:24 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops: UPDATES.refresh_vm ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] Client.UPDATES.inject_barrier 6 May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] triggering xapi event thread to re-register via session.logout May 5 16:13:24 xcp-ng xapi: [error||412529 |Async.VM.start R:f406c721ea1c|xenops] Re-raising as INTERNAL_ERROR [ xenopsd internal error: Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")) ] May 5 16:13:24 xcp-ng xapi: [error||412529 ||backtrace] Async.VM.start R:f406c721ea1c failed with exception Server_error(INTERNAL_ERROR, [ xenopsd internal error: Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")) ]) May 5 16:13:24 xcp-ng xapi: [error||412529 ||backtrace] Raised Server_error(INTERNAL_ERROR, [ xenopsd internal error: Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")) ]) May 5 16:13:24 xcp-ng xapi: [error||412529 ||backtrace] 1/1 xapi Raised at file (Thread 412529 has no backtrace table. Was with_backtraces called?, line 0

Outside of the initial topic scope, but if you have missing Terraform features, please report them on the repo so we can know and decide what to do 🙂

So no luck 😞

lspci | grep USB 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05) 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05) 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) xl pci-assignable-list 0000:00:11.4 0000:04:00.0 vm.start { "id": "da3eae3e-a67a-f0e2-f113-dd67f65baef1", "bypassMacAddressesCheck": false, "force": false } { "code": "INTERNAL_ERROR", "params": [ "xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\"))" ], "call": { "method": "VM.start", "params": [ "OpaqueRef:63acb07f-292f-4062-820f-98ea4934b653", false, false ] }, "message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\")))", "name": "XapiError", "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\"))) at Function.wrap (/home/node/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12) at /home/node/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:35:27 at AsyncResource.runInAsyncScope (async_hooks.js:197:9) at cb (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42) at tryCatcher (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromise0 (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10) at Promise._settlePromises (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18) at _drainQueueStep (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12) at _drainQueue (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues [as _onImmediate] (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14) at processImmediate (internal/timers.js:464:21) at process.topLevelDomainCallback (domain.js:147:15) at process.callbackTrampoline (internal/async_hooks.js:129:24)" }

Any ideas anyone before I give up and send the card back......

EDIT

OK, computer says NO!!!! - This indicates that your device is using RMRR (opens new window). Intel IOMMU does not allow DMA to these devices (opens new window)and therefore PCI passthrough is not supported

EDIT 2

Could we patch the kernel too?

https://github.com/kiler129/relax-intel-rmrr/blob/master/README.md#other-distros

@olivierlambert Thanks. I suspected that, but thought it was me being dumb and haven't found anything on it 🙂

Thanks

@Andrew Thank you very much for your reply. This helped a lot, and I was able to get serial access to VMs configured to use the serial port.

Hi,

No, it's not possible: XAPI and xl are very very different. However, you can tell us what do you need and we could help to tell if it's needed or possible in XAPI 🙂

Thank you very much @phipra : it's important to know what happened and this is really helpful for others experiencing similar issues!