It sounds really slow to install on this hardware (even if it's not crazy fast, it shouldn't be that long). I would take a look at usual logs (like dmesg) to see if there's a problem on the hardware. I would also check to have latest BIOS and firmware installed on that machine.

Ok, I was able to fix this. I migrated it to the pool master on the cli. I don't think this was necessary in hindsight but I just wanted to be sure I knew where it was running. I then brought up xo-lite in Chrome on the pool master. At that point I could view the VM in the embedded console. It turned out that the root partition had errors on it and was stuck in Initramfs waiting for fsck to be run. Once I repaired the errors with fsck, the VM booted and I'm back in business.

@gsrfan01 As near as I can tell the apt version does not install the xe-linux-distribution.service and run it, and therefore the various daemons won't load. You can read more about my plight here: https://xcp-ng.org/forum/topic/7494/ubuntu-22-04lts-guest-utiilities?_=1688263726663

@stormi said in vTPM support requirements: The TPM device is emulated in the controller domain, so it's all software. No need for hardware TPM. thanks, That's great to hear. Makes getting windows 11 machines working much easier.

Thank you so much @olivierlambert, That helps me alot with clarification.

That script has a shebang pointing to node (first line), so no need to define it separately, just execute the script directly.

@ronan-a My pool is mostly working now, though I'm still seeing some quirks. I believe the problems I've seen thus far is due to the hardware instability of one of my hosts, as noted before. While frustrating, it has given me the opportunity to work through some host and SR recovery processes. That's the silver lining. The main issue that I'm still seeing is when I attempt to load a specific stopped VM on a specific host (xcp-ng3). The VM will fail to start and throw this error: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=VDI a9d53abc-f492-4826-b659-744ee87d4d94 not detached cleanly], ) This VM will start without issue on the other two hosts. XO is not showing any other issues with this VM and is not showing any orphaned VDIs. I am also seeing these errors in the /var/log/linstor-controller directory: Host xcp-ng1: ERROR REPORT 648FDBE6-00000-000000 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:39:06 Node: xcp-ng1 ============================================================ Reported error: =============== Category: Exception Class name: SocketException Class canonical name: java.net.SocketException Generated at: Method 'bind0', Source file 'Net.java, Unknown line number Error message: Protocol family unavailable Call backtrace: Method Native Class:Line number bind0 Y sun.nio.ch.Net:unknown bind N sun.nio.ch.Net:461 bind N sun.nio.ch.Net:453 bind N sun.nio.ch.ServerSocketChannelImpl:222 bind N sun.nio.ch.ServerSocketAdaptor:85 bindToChannelAndAddress N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:107 bind N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:64 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:215 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:195 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:186 start N org.glassfish.grizzly.http.server.NetworkListener:711 start N org.glassfish.grizzly.http.server.HttpServer:256 start N com.linbit.linstor.api.rest.v1.config.GrizzlyHttpService:314 initialize N com.linbit.linstor.systemstarter.GrizzlyInitializer:88 startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87 start N com.linbit.linstor.core.Controller:365 main N com.linbit.linstor.core.Controller:613 END OF ERROR REPORT. Host xcp-ng2 (each truncated for post limit): ERROR REPORT 648E4932-00000-000012 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng1' ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Execute single-stage API UpdateFreeCapacity |_ checkpoint ⇢ Fallback error handling wrapper Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000013 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng3' ============================================================ Reported error: =============== Category: RuntimeException Class name: ErrorCallbackNotImplemented Class canonical name: reactor.core.Exceptions.ErrorCallbackNotImplemented Generated at: <UNKNOWN> Error message: com.linbit.linstor.transaction.TransactionException: Failed to start transaction Call backtrace: Method Native Class:Line number ERROR REPORT 648E4932-00000-000014 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: RestClient(127.0.0.1; 'PythonLinstor/1.15.1 (API1.0.4): Client 1.15.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Modification of node 'xcp-ng1' failed due to an unknown exception. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Auto-Quorum and -Tiebreaker after node create |_ checkpoint ⇢ Reconnect node(s) Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000015 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng2' ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Execute single-stage API ApplyPropsFromStlt |_ checkpoint ⇢ Fallback error handling wrapper Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000016 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: RestClient(127.0.0.1; 'PythonLinstor/1.15.1 (API1.0.4): Client 1.15.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Modification of node 'xcp-ng1' failed due to an unknown exception. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Reconnect node(s) Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000017 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: ErrorCallbackNotImplemented Class canonical name: reactor.core.Exceptions.ErrorCallbackNotImplemented Generated at: <UNKNOWN> Error message: com.linbit.linstor.transaction.TransactionException: Failed to start transaction Error context: Exception thrown by connection observer when outbound connection established Call backtrace: Method Native Class:Line number ERROR REPORT 648E4932-00000-000018 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng1' ============================================================ Reported error: =============== Category: RuntimeException Class name: IllegalStateException Class canonical name: java.lang.IllegalStateException Generated at: Method 'assertOpen', Source file 'BaseGenericObjectPool.java', Line #759 Error message: Pool not open Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Execute single-stage API NotifyDevMgrRunCompleted |_ checkpoint ⇢ Fallback error handling wrapper Stack trace: Call backtrace: Method Native Class:Line number assertOpen N org.apache.commons.pool2.impl.BaseGenericObjectPool:759 ERROR REPORT 648E4932-00000-000019 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Restore node Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000020 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Fetch thin capacity info Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000021 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Fetch thin capacity info Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000023 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: RestClient(127.0.0.1; 'PythonLinstor/1.15.1 (API1.0.4): Client 1.15.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Modification of node 'xcp-ng1' failed due to an unknown exception. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Reconnect node(s) Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000024 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Fetch thin capacity info Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 Host xcp-ng3: ERROR REPORT 64905A1B-00000-000000 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 09:37:36 Node: xcp-ng3 ============================================================ Reported error: =============== Category: Exception Class name: SocketException Class canonical name: java.net.SocketException Generated at: Method 'bind0', Source file 'Net.java, Unknown line number Error message: Protocol family unavailable Call backtrace: Method Native Class:Line number bind0 Y sun.nio.ch.Net:unknown bind N sun.nio.ch.Net:461 bind N sun.nio.ch.Net:453 bind N sun.nio.ch.ServerSocketChannelImpl:222 bind N sun.nio.ch.ServerSocketAdaptor:85 bindToChannelAndAddress N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:107 bind N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:64 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:215 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:195 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:186 start N org.glassfish.grizzly.http.server.NetworkListener:711 start N org.glassfish.grizzly.http.server.HttpServer:256 start N com.linbit.linstor.api.rest.v1.config.GrizzlyHttpService:314 initialize N com.linbit.linstor.systemstarter.GrizzlyInitializer:88 startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87 start N com.linbit.linstor.core.Controller:365 main N com.linbit.linstor.core.Controller:613 END OF ERROR REPORT. Sorry, that was a lot of logs, but I don't know exactly what you'd be interested in seeing. It's possible that all of the logs from host xcp-ng2 are from the last time host xcp-ng1 melted down. I'm still trying to determine exactly when that happened overnight. Let me know if you need to see anything else! Thank you!

Use it from the sources then

the only thing referencing xcp-ng-02 when i xe vm-list uuid ( RO) : 0002a023-44d2-4185-a949-a8583d070a76 name-label ( RW): Control domain on host: xcp-ng-02 power-state ( RO): running

That might change in the future We are working on better chipset emulation that should help on many aspects

@bikepope Thank you. Useful information to understand the market landscape.

Ah great news! Thanks a lot for the feedback: it's really important to come back to tell when it works, as you said, it helps a lot the whole community!

@olivierlambert This is correct, your logic is spot on. The pFsense VM was soft-migrated with continuous replication. However the XO VM was made afresh from scratch and it was not starting. Anyway, things work as expected now, thanks!

@jeffmetal You are connecting to Dom0. It does not use (or show) all cores/threads, only the 16 configured for it. So, there is no issue as that is correct. RTFM about Dom0.

@chon It's Jun 3, 2023. After three years, I'm going to finally do the migration to a new Dell server hardware. I'll update this thread for those who are interested. thanks.

@dthenot THANK YOU !!

Hello, I'm honestly don't know how Citrix vGPU stuff works, but couple of thoughts on this topic: If I understand correctly, you say Nvidia use VFIO Linux framework to enable mediated devices which can be exported to guest. The VFIO framework isn't supported by XEN, as VFIO need the presense of IOMMU device managed by IOMMU Linux kernel driver. And XEN doesn't proide/virtualize IOMMU access to dom0 (XEN manages IOMMU by itself, but doesn't offer such access to guests) Bascally to export SR-IOV virtual function to guest with XEN you don't have to use VFIO, you can just assign the virtual function PCI "bdf" id to guest and normally the guest should see this device. From what I understand Nvidia user-mode toolstack (scripts & binaries) doesn't JUST create SR-IOV virtual functions, but want to access VFIO/MDEV framework, so all this thing fails. So may be, you can check if you there's some options with Nvidia tools to just create SR-IOV functions, OR try to run VFIO in "no-iommu" mode (no IOMMU presence in Linux kernel required) BTW, we working on some project where we are intending to use VFIO with dom0, and so we're implementing the IOMMU driver in dom0 kernel, so it would be interesting to know in the future, if this can help with your case. Hope this help

@AndreS I mentionned it already but I can repeat again. My system is already installed in 3 partitions: /boot, /boot/efi and the last one is encrypted, inside 2 LVM volumes, one for swap and and the other one as root. I can't try just xen without LVM and dmcrypt, because my system is already installed. And no, I am not trying to add 3 in one, without Xen, my system work already with LVM and dmcrypt

For what we heard it works very well and you got up to 20% uplift in the previous generation.