<![CDATA[Xen orchestra VM Console broken?]]>Updated to a newer Xen orchestra recently, didn't record what version. But noticed today, that the VM console for a LOT of VMs is broken, and doesn't show up. XCP-ng Center still works fine.
Tried restarting the toolstack on one host, made no difference. Use the XenOrchestraInstallerUpdater to go to update, today, and it didn't fix anything Went back to the earliest version I had, that I can roll back to, and the consoles seem to work again. The working version is: build 04913 5.89/ 5.94.

Any ideas?

]]>https://xcp-ng.org/forum/topic/5753/xen-orchestra-vm-console-brokenRSS for NodeSun, 15 Mar 2026 08:47:50 GMTMon, 11 Apr 2022 16:01:34 GMT60<![CDATA[Reply to Xen orchestra VM Console broken? on Mon, 02 May 2022 20:50:16 GMT]]>Updated to Master build, a66ae, xo-server 5.92.0, xo-web 5.96.0.
Consoles seem to work fine again.
thanks all!

]]>https://xcp-ng.org/forum/post/48897https://xcp-ng.org/forum/post/48897Mon, 02 May 2022 20:50:16 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Mon, 25 Apr 2022 12:51:58 GMT]]>@florent thanks, yeah this seems to fix it again.

]]>https://xcp-ng.org/forum/post/48683https://xcp-ng.org/forum/post/48683Mon, 25 Apr 2022 12:51:58 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Mon, 25 Apr 2022 12:41:00 GMT]]>@ronivay hi, can you test this branch fix_console_bis before we merge it?

thank you for your report

]]>https://xcp-ng.org/forum/post/48680https://xcp-ng.org/forum/post/48680Mon, 25 Apr 2022 12:41:00 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Mon, 25 Apr 2022 08:52:38 GMT]]>yeah it’s 7.6 i see this with

]]>https://xcp-ng.org/forum/post/48667https://xcp-ng.org/forum/post/48667Mon, 25 Apr 2022 08:52:38 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Mon, 25 Apr 2022 08:44:00 GMT]]>@ronivay hi , something should not have worked as intended, I'm working on it. Your server is a 7.6 ?

]]>https://xcp-ng.org/forum/post/48666https://xcp-ng.org/forum/post/48666Mon, 25 Apr 2022 08:44:00 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Sat, 23 Apr 2022 10:59:37 GMT]]>@florent I see that the issue is still open, but looks like PR https://github.com/vatesfr/xen-orchestra/pull/6191 was merged so i assume it's not WIP anymore? Just wanted to let you know that some of the changes in that PR before merge made the initial fix non functional. Tested with installation from latest master branch commit.

Errors in browser have changed slightly:

Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54
    Error util.js:44
    _fail rfb.js:578
    f rfb.js:233
    onclose websock.js:305

and

Tried changing state of a disconnected RFB object util.js:44:54
    Error util.js:44
    _updateConnectionState rfb.js:473
    disconnect rfb.js:284
    _clean react-novnc.js:66
    componentWillUnmount react-novnc.js:115
    React 139

XO doesn't log anything related to fallback method, just a normal console proxy INFO message. Host still gives the same Failure "ws_proxy: not implemented" error and seems that the fallback to TCP doesn't happen and console doesn't open. Still works for PV guests as it did before.

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server/proxy-console): fallback on TCP if WS not available #6191

]]>https://xcp-ng.org/forum/post/48636https://xcp-ng.org/forum/post/48636Sat, 23 Apr 2022 10:59:37 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 15:56:40 GMT]]>@florent
I ran the install upgrade script from @ronivay with the fix_console_proxy_xcp_7x branch. Seems to be working.

]]>https://xcp-ng.org/forum/post/48379https://xcp-ng.org/forum/post/48379Thu, 14 Apr 2022 15:56:40 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 15:29:03 GMT]]>@bberndt are you able to test the fix ?

]]>https://xcp-ng.org/forum/post/48378https://xcp-ng.org/forum/post/48378Thu, 14 Apr 2022 15:29:03 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 15:24:30 GMT]]>@stormi at least it failed on 7.1CU2 installed this morning.
I implemented a fallback not checking the version, but catching the specific error message and going back to the old method

]]>https://xcp-ng.org/forum/post/48377https://xcp-ng.org/forum/post/48377Thu, 14 Apr 2022 15:24:30 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 15:24:15 GMT]]>I don't know if this helps:
most of my VMs are Windows. There are a few Linux, Centos 7+, Rocky, and Ubuntu 18.
the production XO I use, is behind an Apache reverse proxy, using https. The temporary one, is just XO, using http on 8080. Looks like all of these are PVHVM, (so, HVM?) it claims. Both of these XO's are VMs on the same host, and the other hosts are on the same subnet, but some are on a different VLAN. There are also a number that are in a co-location facility, connected via a point to point fiber link, on a different VLAN, but all behind our edge router.
Console working: XCP 8.1, XS 7.1
Console not working: XCP 7.6 - I can't say for sure that these have any updates applied, though.

]]>https://xcp-ng.org/forum/post/48376https://xcp-ng.org/forum/post/48376Thu, 14 Apr 2022 15:24:15 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 15:21:12 GMT]]>@olivierlambert Yes, that was the case of one of the tests as far as I can tell.

]]>https://xcp-ng.org/forum/post/48375https://xcp-ng.org/forum/post/48375Thu, 14 Apr 2022 15:21:12 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 15:19:11 GMT]]>You need both HVM guest AND XS < 7.2

]]>https://xcp-ng.org/forum/post/48373https://xcp-ng.org/forum/post/48373Thu, 14 Apr 2022 15:19:11 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 14:59:13 GMT]]>@olivierlambert said in Xen orchestra VM Console broken?:

@ronivay Do you mean it works for PV guests but not HVM ones?

edit: that might be the reason why we can't reproduce since we probably test on PV guest 🤔

When I tested yesterday, this was on HVM guests.

]]>https://xcp-ng.org/forum/post/48371https://xcp-ng.org/forum/post/48371Thu, 14 Apr 2022 14:59:13 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 09:36:16 GMT]]>@ronivay yeah

I'm making a little code cleanup before merging in master (removing console , adding changelog)
We'll probably make another patch release with this fix.

I'm glad you saw it and give us such a precise feedback.

]]>https://xcp-ng.org/forum/post/48357https://xcp-ng.org/forum/post/48357Thu, 14 Apr 2022 09:36:16 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 09:27:10 GMT]]>Thanks. This seems to fix it for me at least. XO logs UNEXPECTED RESPONSE and FALLBACK for those HVM VM's and console opens up fine.

]]>https://xcp-ng.org/forum/post/48356https://xcp-ng.org/forum/post/48356Thu, 14 Apr 2022 09:27:10 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 09:22:27 GMT]]>@ronivay @bberndt Could you try this branch fix_console_proxy_xcp_7x containing a quick fix (fallback for older hosts to the old method)?

from this PR: https://github.com/vatesfr/xen-orchestra/pull/6191

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server/proxy-console): fallback on TCP if WS not available #6191

]]>https://xcp-ng.org/forum/post/48355https://xcp-ng.org/forum/post/48355Thu, 14 Apr 2022 09:22:27 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 07:59:09 GMT]]>@ronivay that's a great information. I can reproduce the comportment with xenserver 7.1 CU 2
Thank you

]]>https://xcp-ng.org/forum/post/48351https://xcp-ng.org/forum/post/48351Thu, 14 Apr 2022 07:59:09 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 07:38:04 GMT]]>Yeah good point, all those old ones are indeed PV guests and all the rest are HVM.

]]>https://xcp-ng.org/forum/post/48350https://xcp-ng.org/forum/post/48350Thu, 14 Apr 2022 07:38:04 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 07:25:53 GMT]]>@ronivay Do you mean it works for PV guests but not HVM ones?

edit: that might be the reason why we can't reproduce since we probably test on PV guest 🤔

]]>https://xcp-ng.org/forum/post/48349https://xcp-ng.org/forum/post/48349Thu, 14 Apr 2022 07:25:53 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 07:22:48 GMT]]>Interestingly console for some old VM's with old OS are working, like centos 6, debian 8 but anything above those (centos7, debian 10, debian 11) aren't. I'm getting same errors in browser as listed by @bberndt earlier. This can be found from host xensource.log

Apr 14 09:55:45 xs xapi: [debug|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] VM OpaqueRef:be204b50-6028-41f7-9744-e94ad7329046 console port: Some unix:/var/run/xen/vnc-290
Apr 14 09:55:45 xs xapi: [error|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] No implementation for web-sockets console proxy to a Unix domain socket
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Connection to VM console R:7839b84e856e failed with exception (Failure "ws_proxy: not implemented")
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Raised (Failure "ws_proxy: not implemented")
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] 1/1 xapi @ xs Raised at file (Thread 33053260 has no backtrace table. Was with_backtraces called?, line 0
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace]

So far i've tested with XO built with XenOrchestraInstallerUpdater (obviously ;)) and also built one manually, same symptoms. I've tried XO with and without reverse proxy, with http, with https, with one in same private network as XCP-ng host, with one connecting to it over VPN. No special firewalling or filtering in between and no HTTP proxy.

But as the error from host says, it sounds like it’s simply missing support for this type of console connection.

E: and just for the record. wsproxy package is installed:

# rpm -q wsproxy
wsproxy-1.6.0-4.el7.centos.x86_64
]]>https://xcp-ng.org/forum/post/48348https://xcp-ng.org/forum/post/48348Thu, 14 Apr 2022 07:22:48 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 07:02:48 GMT]]>@ronivay I did a lot of tests yesterday (master/latest/old and new commit) , with a wide range of xcp and didn't reproduce it

Do you use any http proxy between XO an XCP ? Do you use some kind of network filtering ? do you use xo in HTTP or HTTPS mode ?

]]>https://xcp-ng.org/forum/post/48347https://xcp-ng.org/forum/post/48347Thu, 14 Apr 2022 07:02:48 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 06:37:25 GMT]]>Okay we tested with XS 7.1 and XCP-ng 7.5, without any luck 😕

Can you describe a bit more your test protocol?

]]>https://xcp-ng.org/forum/post/48344https://xcp-ng.org/forum/post/48344Thu, 14 Apr 2022 06:37:25 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Thu, 14 Apr 2022 06:31:40 GMT]]>I can also reproduce this with XCP-ng 7.6. I’ll try to do some additional tests. Let me know if there’s anything i can assist with.

]]>https://xcp-ng.org/forum/post/48343https://xcp-ng.org/forum/post/48343Thu, 14 Apr 2022 06:31:40 GMT<![CDATA[Reply to Xen orchestra VM Console broken? on Wed, 13 Apr 2022 22:57:48 GMT]]>@olivierlambert
hrm, well, I bumped my temp install of XO back to the master branch, and consoles broke. -Appears- to only broke with XCP 7.6. 7.1, sites worked.

]]>https://xcp-ng.org/forum/post/48340https://xcp-ng.org/forum/post/48340Wed, 13 Apr 2022 22:57:48 GMT