XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Xen orchestra VM Console broken?

    Scheduled Pinned Locked Moved Xen Orchestra
    36 Posts 6 Posters 7.7k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B Offline
      bberndt
      last edited by

      Updated to a newer Xen orchestra recently, didn't record what version. But noticed today, that the VM console for a LOT of VMs is broken, and doesn't show up. XCP-ng Center still works fine.
      Tried restarting the toolstack on one host, made no difference. Use the XenOrchestraInstallerUpdater to go to update, today, and it didn't fix anything Went back to the earliest version I had, that I can roll back to, and the consoles seem to work again. The working version is: build 04913 5.89/ 5.94.

      Any ideas?

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates πŸͺ Co-Founder CEO
        last edited by

        Hi,

        Always start with https://xen-orchestra.com/docs/community.html

        1 Reply Last reply Reply Quote 0
        • DanpD Offline
          Danp Pro Support Team
          last edited by

          No issues here with latest sources connecting to VM consoles.

          1 Reply Last reply Reply Quote 0
          • B Offline
            bberndt
            last edited by

            Is there any reason this might happen if my Xen Server is Citrix 7.1, or XCP 7.6? This is the current suspicion, after a suggestion from the writer of the install script.

            1 Reply Last reply Reply Quote 0
            • B Offline
              bberndt
              last edited by olivierlambert

              Made another VM with Rocky Linux 8, and a copy of the existing to duplicate my set up. Added some hosts to it, and sure enough same problem. Tried to pull an update to XenOrchestraInstallerUpdater, but its already up to date, and the author says its auto-updates.
              So far, not finding a common thread, between them showing up and not showing up. Happens in Firefox, and Chrome, Doesn't appear tob e related to Xen Guest Tools version, getting device drivers from MS or no, or version of XenServer.
              As long as im on this old version of XO, it seems to work.

              I do see these when i bring up a VM console that doesn't show, and DON'T see them when one does show.

              Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54
                  Error util.js:44
                  _fail rfb.js:578
                  f rfb.js:233
                  onclose websock.js:305
                  open websock.js:307
                  _connect rfb.js:384
                  _updateConnectionState rfb.js:546
                  connect rfb.js:279
                  p react-novnc.js:106
                  componentDidMount react-novnc.js:111
                  React 12
                  listen EventListener.js:29
                  React 29
              
              Tried changing state of a disconnected RFB object util.js:44:54
                  Error util.js:44
                  _updateConnectionState rfb.js:473
                  disconnect rfb.js:284
                  _clean react-novnc.js:66
                  p react-novnc.js:72
                  (Async: setTimeout handler)
                  _onUpdateState react-novnc.js:44
                  _updateConnectionState rfb.js:519
                  _fail rfb.js:588
                  f rfb.js:233
                  onclose websock.js:305
                  open websock.js:307
                  _connect rfb.js:384
                  _updateConnectionState rfb.js:546
                  connect rfb.js:279
                  p react-novnc.js:106
                  componentDidMount react-novnc.js:111
                  React 12
                  listen EventListener.js:29
                  React 29
                      trapBubbledEvent
                      listenTo
                      z
                      _updateDOMProperties
                      mountComponent
                      mountComponent
                      mountChildren
                      _createInitialChildren
                      mountComponent
                      mountComponent
                      mountChildren
                      _createInitialChildren
                      mountComponent
                      mountComponent
                      performInitialMount
                      mountComponent
                      mountComponent
                      performInitialMount
                      mountComponent
                      mountComponent
                      performInitialMount
                      mountComponent
                      mountComponent
                      performInitialMount
                      mountComponent
                      mountComponent
                      performInitialMount
                      mountComponent
                      mountComponent
              
              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates πŸͺ Co-Founder CEO
                last edited by

                Can you check since which commit you have the issue? (eg just before the commit that @ronivay pointed to you)

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates πŸͺ Co-Founder CEO
                  last edited by olivierlambert

                  Try to get on commit b9ff3db9b0e023aef36961d0127a266ae2b621a6 (just before c99120bd24b3b73128a0944bb54d4f1dd6ed61e8), rebuild and try again πŸ™‚

                  1 Reply Last reply Reply Quote 1
                  • B Offline
                    bberndt
                    last edited by

                    after googling some git commands; im not very versed in git at all, so not sure how to get that. πŸ˜•
                    this is the one that last worked for me, and the one im currently using. I save the last 3 in @ronivay script.
                    v5.16.0-8953-g04913cabb

                    this is the one that I pulled and it seems to break:
                    v5.16.0-9058-gf6e1b9571

                    I see another comment on the issue tracker from @ronivay. I'll try to get to his suggestion a little later.
                    and just to link these two together for someone in the way future:
                    https://github.com/ronivay/XenOrchestraInstallerUpdater/issues/116

                    bberndt created this issue in ronivay/XenOrchestraInstallerUpdater

                    closed VM console not working in XO. #116

                    1 Reply Last reply Reply Quote 0
                    • B Offline
                      bberndt
                      last edited by

                      @olivierlambert , I received exact instructions from @ronivay on how to do that. Using commit b9ff3db9b in my new temporary install of XO, everything works fine.
                      Moving to c99120b seems to break the console on anything other than XCP 8.2.
                      And I've mentioned before I think: all of our hardware does support XCP 8.2, but almost none of the RAID card monitor software works there. 80% of them are using 3Ware/ LSI 9750's. I can't recall the exact problem, but the executable for the web-admin doesn't run.

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates πŸͺ Co-Founder CEO
                        last edited by olivierlambert

                        1. We'll restore backward compatibility and fix that ( @florent will fix that commit). Thanks for reporting it πŸ™‚
                        2. Your previous XCP-ng installs aren't patched anymore, so you are at risk. You should really migrate to it, regardless the rest (and maybe seek for a solution for your RAID monitoring, but installing stuff in dom0 is not recommended in general). At least this advice worth for production use, not home lab.
                        1 Reply Last reply Reply Quote 1
                        • olivierlambertO Offline
                          olivierlambert Vates πŸͺ Co-Founder CEO
                          last edited by

                          Issue created: https://github.com/vatesfr/xen-orchestra/issues/6187

                          olivierlambert created this issue in vatesfr/xen-orchestra

                          closed Consoles are broken on older XCP/XS < 8.0 #6187

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates πŸͺ Co-Founder CEO
                            last edited by

                            We can't reproduce the problem here πŸ˜•

                            B ronivayR 2 Replies Last reply Reply Quote 0
                            • B Offline
                              bberndt @olivierlambert
                              last edited by bberndt

                              @olivierlambert
                              hrm, well, I bumped my temp install of XO back to the master branch, and consoles broke. -Appears- to only broke with XCP 7.6. 7.1, sites worked.

                              1 Reply Last reply Reply Quote 0
                              • ronivayR Offline
                                ronivay Top contributor @olivierlambert
                                last edited by

                                I can also reproduce this with XCP-ng 7.6. I’ll try to do some additional tests. Let me know if there’s anything i can assist with.

                                florentF 1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates πŸͺ Co-Founder CEO
                                  last edited by

                                  Okay we tested with XS 7.1 and XCP-ng 7.5, without any luck πŸ˜•

                                  Can you describe a bit more your test protocol?

                                  ronivayR 1 Reply Last reply Reply Quote 0
                                  • florentF Offline
                                    florent Vates πŸͺ XO Team @ronivay
                                    last edited by

                                    @ronivay I did a lot of tests yesterday (master/latest/old and new commit) , with a wide range of xcp and didn't reproduce it

                                    Do you use any http proxy between XO an XCP ? Do you use some kind of network filtering ? do you use xo in HTTP or HTTPS mode ?

                                    1 Reply Last reply Reply Quote 0
                                    • ronivayR Offline
                                      ronivay Top contributor @olivierlambert
                                      last edited by ronivay

                                      Interestingly console for some old VM's with old OS are working, like centos 6, debian 8 but anything above those (centos7, debian 10, debian 11) aren't. I'm getting same errors in browser as listed by @bberndt earlier. This can be found from host xensource.log

                                      Apr 14 09:55:45 xs xapi: [debug|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] VM OpaqueRef:be204b50-6028-41f7-9744-e94ad7329046 console port: Some unix:/var/run/xen/vnc-290
                                      Apr 14 09:55:45 xs xapi: [error|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] No implementation for web-sockets console proxy to a Unix domain socket
                                      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Connection to VM console R:7839b84e856e failed with exception (Failure "ws_proxy: not implemented")
                                      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Raised (Failure "ws_proxy: not implemented")
                                      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] 1/1 xapi @ xs Raised at file (Thread 33053260 has no backtrace table. Was with_backtraces called?, line 0
                                      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace]
                                      

                                      So far i've tested with XO built with XenOrchestraInstallerUpdater (obviously ;)) and also built one manually, same symptoms. I've tried XO with and without reverse proxy, with http, with https, with one in same private network as XCP-ng host, with one connecting to it over VPN. No special firewalling or filtering in between and no HTTP proxy.

                                      But as the error from host says, it sounds like it’s simply missing support for this type of console connection.

                                      E: and just for the record. wsproxy package is installed:

                                      # rpm -q wsproxy
                                      wsproxy-1.6.0-4.el7.centos.x86_64
                                      
                                      florentF 1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates πŸͺ Co-Founder CEO
                                        last edited by olivierlambert

                                        @ronivay Do you mean it works for PV guests but not HVM ones?

                                        edit: that might be the reason why we can't reproduce since we probably test on PV guest πŸ€”

                                        ronivayR stormiS 2 Replies Last reply Reply Quote 0
                                        • ronivayR Offline
                                          ronivay Top contributor @olivierlambert
                                          last edited by

                                          Yeah good point, all those old ones are indeed PV guests and all the rest are HVM.

                                          florentF 1 Reply Last reply Reply Quote 0
                                          • florentF Offline
                                            florent Vates πŸͺ XO Team @ronivay
                                            last edited by

                                            @ronivay that's a great information. I can reproduce the comportment with xenserver 7.1 CU 2
                                            Thank you

                                            1 Reply Last reply Reply Quote 2
                                            • First post
                                              Last post