I second what @olivierlambert says here, really should NOT disable them, these mitigations are in place for a reason and should be left in place regardless of how trusted or untrusted of an environment it is. If this is a production system I would also note that you really shouldn't disable them, could be considered negligence in the event of a security incident.

Thanks for your feedback, this might help the community

Good luck!

@ivanobom I have made a request for packages to be added to the Synergy repository of Alma Linux. Give them the time to add the packages to the repository then try the Alma Linux distribution.

I didn't find a solution, it looked like a change of the host that affected network. Rerunning from USB with latest beta kept showing an upgrade. Network reset did not fix it and the machine was laggy, like when I tried to join from XO ( via settings ). I just did a complete reinstall, disabled HA on the NFS SR, joined pool and it worked and re-enabled HA. Seems ok now, thanks for the help. By the way, is there a way to join a node to the main pool in XO ? Thanks!

I'm talking in the BIOS or in Xen, I would stat to be sure that the CPU is boosting

Ahh great news! CPU issues are really tricky

@KFC-Netearth The Rocky Linux bugtracker indeed mentions it's mostly fixed, but there are still some kernel errors present: https://bugs.rockylinux.org/view.php?id=3565#c4293

Probably (ideally) before the end of 2024, but it's really hard to make any promises (it might be -at that point- an ever more recent version)

@Andrew At the moment I can not find why or what could be running at 21:30 but have a few thoughts into it, Including possible outside hack attempts, as I am on a Fixed IPv4 & IPv6 and the modem is bridged mode to one NIC. But on the VM I have running Pfsense2.6, Plex (Debian10), (Emails, dns, web) Freebsd12, Unifi (debain10), Asterisk (debian10), XenOrchestra7.4CE (Debian10). Not running but can be spun up, Winxp64, XenOrchestra(Paid Version) I have a replacement Nic in the mail and hope this will fix the problem, but wondering if I need to disable the onboard ones to stop the crash even if I force all to use the new ones, but hope this replacement card fixes the problems.

It sounds really slow to install on this hardware (even if it's not crazy fast, it shouldn't be that long). I would take a look at usual logs (like dmesg) to see if there's a problem on the hardware. I would also check to have latest BIOS and firmware installed on that machine.

Ok, I was able to fix this. I migrated it to the pool master on the cli. I don't think this was necessary in hindsight but I just wanted to be sure I knew where it was running. I then brought up xo-lite in Chrome on the pool master. At that point I could view the VM in the embedded console. It turned out that the root partition had errors on it and was stuck in Initramfs waiting for fsck to be run. Once I repaired the errors with fsck, the VM booted and I'm back in business.

@gsrfan01 As near as I can tell the apt version does not install the xe-linux-distribution.service and run it, and therefore the various daemons won't load. You can read more about my plight here: https://xcp-ng.org/forum/topic/7494/ubuntu-22-04lts-guest-utiilities?_=1688263726663

@stormi said in vTPM support requirements: The TPM device is emulated in the controller domain, so it's all software. No need for hardware TPM. thanks, That's great to hear. Makes getting windows 11 machines working much easier.

Thank you so much @olivierlambert, That helps me alot with clarification.

That script has a shebang pointing to node (first line), so no need to define it separately, just execute the script directly.

@ronan-a My pool is mostly working now, though I'm still seeing some quirks. I believe the problems I've seen thus far is due to the hardware instability of one of my hosts, as noted before. While frustrating, it has given me the opportunity to work through some host and SR recovery processes. That's the silver lining. The main issue that I'm still seeing is when I attempt to load a specific stopped VM on a specific host (xcp-ng3). The VM will fail to start and throw this error: SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=VDI a9d53abc-f492-4826-b659-744ee87d4d94 not detached cleanly], ) This VM will start without issue on the other two hosts. XO is not showing any other issues with this VM and is not showing any orphaned VDIs. I am also seeing these errors in the /var/log/linstor-controller directory: Host xcp-ng1: ERROR REPORT 648FDBE6-00000-000000 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:39:06 Node: xcp-ng1 ============================================================ Reported error: =============== Category: Exception Class name: SocketException Class canonical name: java.net.SocketException Generated at: Method 'bind0', Source file 'Net.java, Unknown line number Error message: Protocol family unavailable Call backtrace: Method Native Class:Line number bind0 Y sun.nio.ch.Net:unknown bind N sun.nio.ch.Net:461 bind N sun.nio.ch.Net:453 bind N sun.nio.ch.ServerSocketChannelImpl:222 bind N sun.nio.ch.ServerSocketAdaptor:85 bindToChannelAndAddress N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:107 bind N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:64 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:215 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:195 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:186 start N org.glassfish.grizzly.http.server.NetworkListener:711 start N org.glassfish.grizzly.http.server.HttpServer:256 start N com.linbit.linstor.api.rest.v1.config.GrizzlyHttpService:314 initialize N com.linbit.linstor.systemstarter.GrizzlyInitializer:88 startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87 start N com.linbit.linstor.core.Controller:365 main N com.linbit.linstor.core.Controller:613 END OF ERROR REPORT. Host xcp-ng2 (each truncated for post limit): ERROR REPORT 648E4932-00000-000012 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng1' ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Execute single-stage API UpdateFreeCapacity |_ checkpoint ⇢ Fallback error handling wrapper Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000013 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng3' ============================================================ Reported error: =============== Category: RuntimeException Class name: ErrorCallbackNotImplemented Class canonical name: reactor.core.Exceptions.ErrorCallbackNotImplemented Generated at: <UNKNOWN> Error message: com.linbit.linstor.transaction.TransactionException: Failed to start transaction Call backtrace: Method Native Class:Line number ERROR REPORT 648E4932-00000-000014 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: RestClient(127.0.0.1; 'PythonLinstor/1.15.1 (API1.0.4): Client 1.15.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Modification of node 'xcp-ng1' failed due to an unknown exception. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Auto-Quorum and -Tiebreaker after node create |_ checkpoint ⇢ Reconnect node(s) Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000015 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng2' ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Execute single-stage API ApplyPropsFromStlt |_ checkpoint ⇢ Fallback error handling wrapper Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000016 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: RestClient(127.0.0.1; 'PythonLinstor/1.15.1 (API1.0.4): Client 1.15.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Modification of node 'xcp-ng1' failed due to an unknown exception. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Reconnect node(s) Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000017 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: ErrorCallbackNotImplemented Class canonical name: reactor.core.Exceptions.ErrorCallbackNotImplemented Generated at: <UNKNOWN> Error message: com.linbit.linstor.transaction.TransactionException: Failed to start transaction Error context: Exception thrown by connection observer when outbound connection established Call backtrace: Method Native Class:Line number ERROR REPORT 648E4932-00000-000018 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: Node: 'xcp-ng1' ============================================================ Reported error: =============== Category: RuntimeException Class name: IllegalStateException Class canonical name: java.lang.IllegalStateException Generated at: Method 'assertOpen', Source file 'BaseGenericObjectPool.java', Line #759 Error message: Pool not open Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Execute single-stage API NotifyDevMgrRunCompleted |_ checkpoint ⇢ Fallback error handling wrapper Stack trace: Call backtrace: Method Native Class:Line number assertOpen N org.apache.commons.pool2.impl.BaseGenericObjectPool:759 ERROR REPORT 648E4932-00000-000019 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Restore node Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000020 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Fetch thin capacity info Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000021 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Fetch thin capacity info Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000023 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 Peer: RestClient(127.0.0.1; 'PythonLinstor/1.15.1 (API1.0.4): Client 1.15.1') ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Error context: Modification of node 'xcp-ng1' failed due to an unknown exception. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Reconnect node(s) Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 ERROR REPORT 648E4932-00000-000024 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 00:38:51 Node: xcp-ng2 ============================================================ Reported error: =============== Category: RuntimeException Class name: TransactionException Class canonical name: com.linbit.linstor.transaction.TransactionException Generated at: Method 'startTransaction', Source file 'ControllerSQLTransactionMgrGenerator.java', Line #32 Error message: Failed to start transaction Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ⇢ Fetch thin capacity info Stack trace: Call backtrace: Method Native Class:Line number startTransaction N com.linbit.linstor.transaction.manager.ControllerSQLTransactionMgrGenerator:32 Host xcp-ng3: ERROR REPORT 64905A1B-00000-000000 ============================================================ Application: LINBIT® LINSTOR Module: Controller Version: 1.21.1 Build ID: a677db312062add13e9b230b8b902d43a69caf13 Build time: 2023-03-22T14:05:41+00:00 Error time: 2023-06-19 09:37:36 Node: xcp-ng3 ============================================================ Reported error: =============== Category: Exception Class name: SocketException Class canonical name: java.net.SocketException Generated at: Method 'bind0', Source file 'Net.java, Unknown line number Error message: Protocol family unavailable Call backtrace: Method Native Class:Line number bind0 Y sun.nio.ch.Net:unknown bind N sun.nio.ch.Net:461 bind N sun.nio.ch.Net:453 bind N sun.nio.ch.ServerSocketChannelImpl:222 bind N sun.nio.ch.ServerSocketAdaptor:85 bindToChannelAndAddress N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:107 bind N org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler:64 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:215 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:195 bind N org.glassfish.grizzly.nio.transport.TCPNIOTransport:186 start N org.glassfish.grizzly.http.server.NetworkListener:711 start N org.glassfish.grizzly.http.server.HttpServer:256 start N com.linbit.linstor.api.rest.v1.config.GrizzlyHttpService:314 initialize N com.linbit.linstor.systemstarter.GrizzlyInitializer:88 startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87 start N com.linbit.linstor.core.Controller:365 main N com.linbit.linstor.core.Controller:613 END OF ERROR REPORT. Sorry, that was a lot of logs, but I don't know exactly what you'd be interested in seeing. It's possible that all of the logs from host xcp-ng2 are from the last time host xcp-ng1 melted down. I'm still trying to determine exactly when that happened overnight. Let me know if you need to see anything else! Thank you!

Use it from the sources then