Hi, as promised, here are the log excerpts from the incident, in case they help the @Team-Storage folks (or anyone hitting the same thing). All times are CET (host time). VM dd25471a-fbd3-2b0c-df4c-b8182546f6a8, SR 1a97f391-…, migration task R:4fa24c2fefb3 (beta → gamma). 1. The failed attach (SMlog on gamma) The SR lock is acquired at 23:00:04 and the exception comes 6s later: Nov 10 23:00:04 gamma SM: [11963] lock: acquired /var/lock/sm/1a97f391-c733-8520-62c3-2f3847235a36/sr Nov 10 23:00:10 gamma SM: [11963] Raising exception [47, The SR is not available [opterr=Error: ERRO:NoneType]] Nov 10 23:00:10 gamma SM: [11963] ***** generic exception: vdi_attach: EXCEPTION <class 'xs_errors.SROSError'>, The SR is not available [opterr=Error: ERRO:NoneType] File "/opt/xensource/sm/SRCommand.py", line 157, in _run_locked target = sr.vdi(self.vdi_uuid) File "/opt/xensource/sm/LinstorSR", line 549, in wrap return load(self, *args, **kwargs) File "/opt/xensource/sm/LinstorSR", line 479, in load raise xs_errors.XenError('SRUnavailable', opterr=str(e)) One second later the same SR answers normally, which is why I think the failure was transient: Nov 10 23:00:11 gamma SM: [12120] Failed to join node(s): {'gamma', 'alpha'} Nov 10 23:00:11 gamma SM: [12120] Synchronize metadata... Nov 10 23:00:11 gamma SM: [12120] VDI c90eeb92-... loaded! (path=/dev/drbd/by-res/xcp-persistent-ha-statefile/0, hidden=0) 2. LINSTOR side (controller on beta) Three error reports between 22:59 and 23:00, all identical: ERROR REPORT 6AF3924C-00000-000001 (also -000002 and -000003) Version: 1.33.1 Node: beta Class name: com.linbit.linstor.netcom.PeerClosingConnectionException Generated at: Method 'connectionClosing', Source file 'TcpConnectorPeer.java', Line #627 *__checkpoint ⇢ Fetch thin capacity info 3. How it surfaces in xapi gamma xapi: [error|VDI.attach2] sm_exec failed with exception Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_47);[S();S(The SR is not available [opterr=Error: ERRO:NoneType]);S()]]]) gamma xapi: [error|Async.VM.pool_migrate R:4fa24c2fefb3|Storage_smapiv1_wrapper] dp:vbd/1/xvda sr:1a97f391-... vdi:60342e65-... op:attach(RW) error:Storage_error (...SR_BACKEND_FAILURE_47...) 4. The cleanup on the source also fails (beta) 23:00:10 beta xenopsd-xc: [error|71|Async.VM.pool_migrate R:4fa24c2fefb3|xenops_server] VM.migrate: Failed during Synchronisation point 4. msg: Xenops_interface.Xenopsd_error([S(Storage_backend_error);[S(SR_BACKEND_FAILURE_47);...]]) 23:00:10 beta xenopsd-xc: [error|71|...|task_server] Task 101 failed; Xenopsd_error([S(Does_not_exist);[S(VM);S(dd25471a-.../config)]]) 23:00:10 beta xenopsd-xc: [error|71|...|xenops_server] Triggering cleanup actions failed: Xenopsd_error([S(Does_not_exist);[S(VM);S(dd25471a-.../config)]]) So after the migration error, the cleanup fails too because the VM config is no longer in xenopsd on the source. That matches what I saw afterwards: XAPI says running on beta, but xenopsd doesn't know the VM. 5. What is left on the destination (gamma) The temporary extra/ entry is created at 22:58:09 and renamed to the final uuid at 23:00:03, i.e. before the attach fails at 23:00:10. Nothing removes it afterwards: 22:58:09 gamma xenopsd-xc: [debug|Async.VM.pool_migrate R:4fa24c2fefb3|xenops_utils] TypedTable: Writing extra/dd25471a-fbd3-2b0c-df4c-000000000001 23:00:03 gamma xenopsd-xc: [debug|13|Async.VM.pool_migrate R:4fa24c2fefb3|xenops_utils] TypedTable: Renaming extra/dd25471a-fbd3-2b0c-df4c-000000000001 -> extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8 6. What the following RPU runs report in XO 2nd run (00:03, VM still in that state): END failure Evacuate INTERNAL_ERROR(Object with type VM and id dd25471a-fbd3-2b0c-df4c-b8182546f6a8/config does not exist in xenopsd) END failure Rolling pool update INTERNAL_ERROR(Object with type VM and id dd25471a-.../config does not exist in xenopsd) 3rd run (00:11–00:14, after I had recovered the VM on beta only): END failure Evacuate INTERNAL_ERROR(xenopsd internal error: VM.migrate: Failed during Synchronisation point 4. msg: Xenops_interface.Xenopsd_error([S(Already_exists);[S(extra);S(extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8)]])) And on gamma for that run: 00:13:42 gamma xenopsd-xc: [debug|94|Async.VM.pool_migrate R:005a41ca1c30|xenops_utils] Key extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8 already exists 00:13:42 gamma xenopsd-xc: [error|94|...|backtrace] Raised Xenopsd_error([S(Already_exists);[S(extra);S(extra/dd25471a-fbd3-2b0c-df4c-b8182546f6a8)]]) 00:13:42 gamma xenopsd-xc: [debug|94|...|xenops_server] Caught Xenopsd_error([S(Already_exists);...]): cleaning up VM state Also at 00:14:03, opening the VM console from XO fails on beta with the same Does_not_exist … /config error. What the logs show A single transient LINSTOR controller error (ERRO:NoneType) appears to trigger the failure vdi_attach with no retry. That failure at sync point 4 leaves state on both hosts: on the source the cleanup fails with Does_not_exist …/config, and on the destination the extra/<uuid> entry stays. Subsequent RPU runs then trips on that leftover (first Does_not_exist, then Already_exists), so it can't recover by itself and requires manual recovery on both hosts. I still have the full xensource.log, SMlog and ErrorReport files for that window if someone needs more. Thanks!

@SeriouZ said: Thanks for your help! I have the SMlogs from both hosts ready, but they’re too large to upload here, even compressed Whats the best way to share them with you or Team-Storage? @seriouz Have you tried using zstd at a high or ultra compression level, to see if you can get it smaller? You may need to use a Vates recommended or provided cloud temporary file storage service, outside of attaching to your forum post or PM!

@ph7 Thanks for sharing info. Maybe it will help someone

@Mitchel-APD Thank you! That might acctually help me quite a bit... I have been having trouble getting my ansible-instances to recover from the VM re-initiating the network connection! Perhaps this could be helpfull. I will absolutely test it

@acebmxer that is exactley what i was saying before. if you disable the remote before nas shutdown you are good on the XO side and will work indefinitely, but if you don't execute umount -l /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70 you will have the mount stuck on the os side. You will be able to reboot XO vm eventually but you will have to wait for a long timeout during XO shutdown. if you want to simulate my issue just disable the remote 5 minutes after shutting down the synology, get the pid from service xo-server status then ps -T pid after a while you'll have 4 "rpc_wait_bit_killable" and XO will stop responding

Your reading of the code matches what I see on master, as far as I can tell. The object.enabled === true check in _getHosts() seems to have come in with PR #10209, released in XO 6.7.1 in August, to stop the balancer migrating VMs onto disabled hosts. If I follow the code right, a host that XO shuts down goes through disableHost first, so the power-on step in performance-plan.js no longer sees it. That looks like a side effect of the fix to me. The docs still say the performance plan powers halted hosts back on (https://docs.xen-orchestra.com/manage-your-infrastructure/load_balancing#plan-modes), so I think one of the two is wrong. I haven't reproduced it and there may be a reason to filter halted hosts that I can't see from the code, so @Team-XO-Backend is better placed to say if your powerState === 'Halted' exception is the right fix.

This is a question for @dinhngtu

error via reverse proxy. Uncaught Error: Body is not a JSON, original message is : <html> <title>500 Internal Server Error</title> <style>html{color-scheme:light dark;}</style> </head> <body> <center><h1>500 Internal Server Error</h1></center> <hr><center>Angie</center> </body> </html> pr index.js:2564 me index.js:4029 i index.js:4029 promise callback*me index.js:4029 i index.js:4029 promise callback*me index.js:4029 i index.js:4029 promise callback*me index.js:4029 i index.js:4029 be index.js:4029 be index.js:4029 dr index.js:2531 importDisks index.js:2575 Lodash 2 importDisks index.js:2574 e index.js:113 P index.js:22 i index.js:22 R index.js:22 R index.js:22 import index.js:121 t factory.js:307 o factory.js:331 e action-button.js:93 m action-button.js:10 i action-button.js:10 _execute action-button.js:10 _execute action-button.js:10 _execute action-button.js:10 v action-button.js:140 componentDidMount action-button.js:147 React 14 componentWillUnmount factory.js:125 f factory.js:31 _dispatch factory.js:158 t factory.js:322 promise callback*t factory.js:313 o factory.js:331 value index.js:206 promise callback*value index.js:174 React 5

@IgorGlock What do you mean by "okay"? Backing up the VM with Commvault or...?

@tjkreidl said: @john.c Note that a two-host pool running HA can run into the split-brain issue if one of the hosts fails; it's highly recommended to have at least three hosts in an HA pool (or look into HA-Lizard if running a pool with two hosts and HA). Right but also considering the in development Twinstor for when it goes stable and in production!

@vmpr Popping it behind an instance of BunkerWeb (https://www.bunkerweb.io/) or similar, with Xen Orchestra on a new IP address would help. As BunkerWeb is what’s known as a Web Application Firewall. It will help to protect your instance of Xen Orchestra, especially if you change the IP address of the appliance, and only give that friend the one that goes through BunkerWeb. As always don’t expose Xen Orchestra to the Internet, but if it’s absolutely necessary always use something like BunkerWeb (with BunkerNET) and only provide the BunkerWeb based IP address! Also run a check on all of your hosts and may be best to roll them a fresh (in case there’s an Advanced Persistent Threat). As well as a fresh instance of Xen Orchestra, and maybe encourage your friend while also doing yourself to start using hardware security keys, preferably ones that support biometrics and PINs. Alternatively other secure enclaves with cryptographic credentials held there. Additionally opt into the new encrypted datastore for Xen Orchestra on the new fresh Xen Orchestra and XCP-ng. Also make sure it’s completely up to date! Also if the hosts support it ensure that Intel TME or similar is enabled on the fresh installs of XCP-ng. Finally rotate all passwords associated with that system, including your Vates account password, especially if it’s linked to the Xen Orchestra instance hit.

@jr-m4 I did not took time to test Podman yet. But we expect this to work. You'd have to expose a socket, like the docker daemon does. Probably systemctl enable --now podman.socket would be enough. More in this page https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/using-the-container-tools-api and this one https://docs.podman.io/en/latest/markdown/podman-system-service.1.html

@gudge25 check these https://docs.xcp-ng.org/vms/#enabling-viridian-extensions and https://lazyadmin.nl/it/september-2026-update-break-rds-how-to-fix/

Thanks @ataxyanetwork I was about to answer the same...! (but was too busy this morning )

Hi @poddingue ..yes, I have quite a few 'requests' I need to post in there...I just have yet to do so Thanks!

For the partitioning part, the docs say NVIDIA vGPU isn't supported on XCP-ng, since it depends on a proprietary XenServer component that can't be redistributed: https://docs.xcp-ng.org/compute#vgpu. So as far as I can tell, the "20, 40 or 80 GB per VM" split won't work through vGPU. Passing a whole GPU to one VM is the documented path, on the same page. I haven't seen anyone report on H100 or H200 cards here, though, so I can't tell you if that class of card brings its own surprises. There's also a blog post doing LLM inference that way on a much smaller Tesla T4: https://xcp-ng.org/blog/2025/07/28/your-own-gpu-powered-llms-with-xcp-ng/. There's an open design discussion upstream in XAPI about GPU partitioning, MIG included (https://github.com/xapi-project/xen-api/pull/7260), but that's the XenServer side, and I don't know if or when it would reach XCP-ng. As GPU partitioning matters for your project, it might be worth opening a request on https://feedback.vates.tech/ with what you described here (H200s, LLM inference, several tenants per card, fixed VRAM per VM). From what I've seen, a concrete use case like yours carries more weight there than a plain "please support vGPU", and other people needing the same thing can vote on it.