Thanks, passing the compliment to our teams!

Dear, There is indeed enough memory on the recieving node. We are having nodes with 1TB of memory, and currently they are loaded with around 500GB each. Ill try to reproduce it with a cloned production server. So i can reproduce it a few times with and without network and to a empty receiving node. Ill keep you updated.

@Andrew That's what I thought, but basically wanted confirmation. Are there any settings to look at that may be able to be tuned in XCP to improve responsiveness?

I'm having exactly the same problem with a "Tiger Jet Network Inc. / ICP DAS Tiger3XX Modem/ISDN interface" card. logs show this: May 5 16:13:17 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] xapi_cache: creating cache for ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:17 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops_cache: creating empty cache for ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:17 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xapi_network] register_vif vif=OpaqueRef:8b603d12-373e-4086-a03b-e63e45afb21e network=OpaqueRef:de3a7251-596a-4052-a90a-0caa23d93947 May 5 16:13:17 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops: VM.start ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:17 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops: Queueing VM.unpause ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:24 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xapi_network] Caught Xenops_interface.Xenopsd_error([S(Internal_error);S(Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")))]): detaching networks May 5 16:13:24 xcp-ng xapi: [error||412529 |Async.VM.start R:f406c721ea1c|xenops] Caught exception starting VM: Xenops_interface.Xenopsd_error([S(Internal_error);S(Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")))]) May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] VM ae56aad1-bab7-e6eb-8599-8eea93291e54 set_resident_on May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] Signalling xenapi event thread to re-register, and xenopsd events to sync May 5 16:13:24 xcp-ng xapi: [ info||412529 |Async.VM.start R:f406c721ea1c|xenops] xenops: UPDATES.refresh_vm ae56aad1-bab7-e6eb-8599-8eea93291e54 May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] Client.UPDATES.inject_barrier 6 May 5 16:13:24 xcp-ng xapi: [debug||412529 |Async.VM.start R:f406c721ea1c|xenops] triggering xapi event thread to re-register via session.logout May 5 16:13:24 xcp-ng xapi: [error||412529 |Async.VM.start R:f406c721ea1c|xenops] Re-raising as INTERNAL_ERROR [ xenopsd internal error: Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")) ] May 5 16:13:24 xcp-ng xapi: [error||412529 ||backtrace] Async.VM.start R:f406c721ea1c failed with exception Server_error(INTERNAL_ERROR, [ xenopsd internal error: Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")) ]) May 5 16:13:24 xcp-ng xapi: [error||412529 ||backtrace] Raised Server_error(INTERNAL_ERROR, [ xenopsd internal error: Cannot_add(0000:06:00.0, Xenctrlext.Unix_error(35, "3: No such process")) ]) May 5 16:13:24 xcp-ng xapi: [error||412529 ||backtrace] 1/1 xapi Raised at file (Thread 412529 has no backtrace table. Was with_backtraces called?, line 0

Outside of the initial topic scope, but if you have missing Terraform features, please report them on the repo so we can know and decide what to do

So no luck lspci | grep USB 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05) 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05) 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) xl pci-assignable-list 0000:00:11.4 0000:04:00.0 vm.start { "id": "da3eae3e-a67a-f0e2-f113-dd67f65baef1", "bypassMacAddressesCheck": false, "force": false } { "code": "INTERNAL_ERROR", "params": [ "xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\"))" ], "call": { "method": "VM.start", "params": [ "OpaqueRef:63acb07f-292f-4062-820f-98ea4934b653", false, false ] }, "message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\")))", "name": "XapiError", "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\"))) at Function.wrap (/home/node/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12) at /home/node/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:35:27 at AsyncResource.runInAsyncScope (async_hooks.js:197:9) at cb (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42) at tryCatcher (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromise0 (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10) at Promise._settlePromises (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18) at _drainQueueStep (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12) at _drainQueue (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues [as _onImmediate] (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14) at processImmediate (internal/timers.js:464:21) at process.topLevelDomainCallback (domain.js:147:15) at process.callbackTrampoline (internal/async_hooks.js:129:24)" } Any ideas anyone before I give up and send the card back...... EDIT OK, computer says NO!!!! - This indicates that your device is using RMRR (opens new window). Intel IOMMU does not allow DMA to these devices (opens new window)and therefore PCI passthrough is not supported EDIT 2 Could we patch the kernel too? https://github.com/kiler129/relax-intel-rmrr/blob/master/README.md#other-distros

@olivierlambert Thanks. I suspected that, but thought it was me being dumb and haven't found anything on it Thanks

@Andrew Thank you very much for your reply. This helped a lot, and I was able to get serial access to VMs configured to use the serial port.

Hi, No, it's not possible: XAPI and xl are very very different. However, you can tell us what do you need and we could help to tell if it's needed or possible in XAPI

Thank you very much @phipra : it's important to know what happened and this is really helpful for others experiencing similar issues!

@dj423 Second occurrence of kernel BUG error: [237621.042604] BUG: unable to handle page fault for address: ffff9bd9bc6c9840 [237621.049589] #PF: supervisor read access in kernel mode [237621.055044] #PF: error_code(0x0000) - not-present page I am seeing some strange errors on the host (X1), for storage devices that no longer exist on the host. e.g.: [331863.198883] print_req_error: I/O error, dev tde, sector 74842904 [332612.663953] block tdd: sector-size: 512/512 capacity: 178257920 [332612.711329] block tde: sector-size: 512/512 capacity: 251658240 [333679.019750] print_req_error: 86 callbacks suppressed [333679.019750] print_req_error: I/O error, dev tdd, sector 74924032 The storage devs on host: tdc 254:2 0 85G 0 disk tda 254:0 0 120G 0 disk tdf 254:5 0 32G 0 disk Any idea why I would see I/O errors for storage devices that are no longer on the host? I only see this on one VM on this host, all others are fine.

@olivierlambert we couldn't find a reason either. We will run more tests in the following weeks and report if we find anything of value.

@olivierlambert Those were the only messages on the log, nothing else captured on any other log file like the crit.log file or dmesg

@olivierlambert Had the same issue with a similar server. Replaced both RAM and disk but the host kept going into read-only mode. The host would run just fine when installing any other OS/hypervisor. The server model is Supermicro SYS-6028TP-HC0R

@fred974 thath is the intendended behaviour, since, it's starts the destination VM , and the source and destination have the same ip

@Andrew said in Xen-guest-tools installation problems: You can rename libxenstore.so.4.0 to libxenstore.so.4 Thanks for the info... I figured the utilities have not yet been updated, and you have confirmed.

+1 to @Andrew setup