XCP-ng

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    1. Home
    2. ronivay
    3. Posts
    R
    • Profile
    • Following 0
    • Followers 0
    • Topics 1
    • Posts 32
    • Best 9
    • Controversial 0
    • Groups 0

    Posts made by ronivay

    • RE: Xen orchestra VM Console broken?

      @florent thanks, yeah this seems to fix it again.

      posted in Xen Orchestra
      R
      ronivay
    • RE: Xen orchestra VM Console broken?

      yeah it’s 7.6 i see this with

      posted in Xen Orchestra
      R
      ronivay
    • RE: Xen orchestra VM Console broken?

      @florent I see that the issue is still open, but looks like PR https://github.com/vatesfr/xen-orchestra/pull/6191 was merged so i assume it's not WIP anymore? Just wanted to let you know that some of the changes in that PR before merge made the initial fix non functional. Tested with installation from latest master branch commit.

      Errors in browser have changed slightly:

      Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54
          Error util.js:44
          _fail rfb.js:578
          f rfb.js:233
          onclose websock.js:305
      

      and

      Tried changing state of a disconnected RFB object util.js:44:54
          Error util.js:44
          _updateConnectionState rfb.js:473
          disconnect rfb.js:284
          _clean react-novnc.js:66
          componentWillUnmount react-novnc.js:115
          React 139
      

      XO doesn't log anything related to fallback method, just a normal console proxy INFO message. Host still gives the same Failure "ws_proxy: not implemented" error and seems that the fallback to TCP doesn't happen and console doesn't open. Still works for PV guests as it did before.

      fbeauchamp created this issue in vatesfr/xen-orchestra

      closed fix(xo-server/proxy-console): fallback on TCP if WS not available #6191

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      Yep. Same domain, asks A and AAAA at the same time, both being individual queries obviously.

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      Did some further testing if amount of DNS queries would correlate to the amount of actual connections made to the host. This doesn't seem to be the case which is even more interesting 🙂 Some results below.

      Ran an incremental from delta backup which took in total of 9 minutes:

      • Amount of DNS queries: close to 7k
      • Amount of HTTPS connects logged to host IP-address: 478.
      • Amount of HTTPS connects/disconnects logged in total to host IP-address: 955

      Connection counts were about the same with installation from dns.lookup branch provided by @julien-f above, without the amount of DNS queries obviously.

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      @julien-f this changed the situation from thousands of queries in minutes to no noticeable spike in query graphs during backup job, so huge improvement.

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      I think the main point to focus on here is that XO is doing totally unnecessary DNS queries with excessive frequency. I don't see this as implementing a non-core feature but a fix in the logic how application figures out where to connect and how often. How exactly and what options there are is outside of my knowledge 🙂

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      I'll put this to test and see tomorrow what the DNS query stats look like.

      Just my two cents but i feel like one shouldn't "fix" a flaw or bad behaviour in application by relying on external dependency to deal with it, especially if it's fixable. Sure using something like nscd in XOA would kinda fix the issue in it but wouldn't possible perf issue etc still exist in node? I'm not competent to review the code so can't say anything about the actual implementation in feature branch.

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      This is not really an issue for me, just an observation which could possibly have all sorts of undesired effects to many XO users.

      I know multiple ways of getting around it, couple of them already mentioned here. Local caching with nscd is one more of those. Point is, this should rather be fixed in XO than worked around.

      posted in Xen Orchestra
      R
      ronivay
    • RE: DNS queries during backup job

      Could very well be. With multiple hosts and simultaneous backups, the amount of requests can easily be overwhelming if it multiplies in such scenario. I've actually created an issue about stalled backup due to failed DNS resolution back in 2019 (https://github.com/vatesfr/xen-orchestra/issues/4122). I didn't dig this deep into it then and kinda assumed it was something specific to my environment and switched to using just IP-address. Switched back to domain name at some point. Personally haven't seen failed backups due to same reason recently so could be it retries nowadays at least to some extent.

      /etc/hosts sure is also an option but as a static config it takes away the benefit you get from using DNS.

      ronivay created this issue in vatesfr/xen-orchestra

      closed Failed name resolution stalling backup #4122

      posted in Xen Orchestra
      R
      ronivay
    • DNS queries during backup job

      I got following setup:

      • XO from sources (latest master commit) connected to one XCP-ng host using DNS name
      • Delta backup job with ~20VM's using local remote

      I've recently kept an eye on DNS request done by XO. On average when idle it does around ~150-200 DNS requests per hour to get IP-address for host that is configured using DNS name. This is fine but when backup job starts, this amount goes through the roof:

      My latest backup job ran from 04:00AM to 04:17AM. During this time, XO did 9320 identical requests, all asking for A/AAAA record for configured host. This averages ~550 requests per minute and ~9 requests per second. Obviously the number is not constant during backup job and i even saw it hit my DNS resolver rate limit of 1000 r/minute.

      To me this sounds like something that could and should be improved. Although DNS requests are lightweight and results often cached by resolvers, they're still traffic in the network and obsolete queries should be avoided. It shouldn't be necessary to ask same information so many times during such short time frame.

      Yeah sure i could just configure the host using IP-address and avoid this but it doesn't fix the underlying issue.

      posted in Xen Orchestra
      R
      ronivay
    • RE: Xen orchestra VM Console broken?

      Thanks. This seems to fix it for me at least. XO logs UNEXPECTED RESPONSE and FALLBACK for those HVM VM's and console opens up fine.

      posted in Xen Orchestra
      R
      ronivay
    • RE: Xen orchestra VM Console broken?

      Yeah good point, all those old ones are indeed PV guests and all the rest are HVM.

      posted in Xen Orchestra
      R
      ronivay
    • RE: Xen orchestra VM Console broken?

      Interestingly console for some old VM's with old OS are working, like centos 6, debian 8 but anything above those (centos7, debian 10, debian 11) aren't. I'm getting same errors in browser as listed by @bberndt earlier. This can be found from host xensource.log

      Apr 14 09:55:45 xs xapi: [debug|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] VM OpaqueRef:be204b50-6028-41f7-9744-e94ad7329046 console port: Some unix:/var/run/xen/vnc-290
      Apr 14 09:55:45 xs xapi: [error|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] No implementation for web-sockets console proxy to a Unix domain socket
      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Connection to VM console R:7839b84e856e failed with exception (Failure "ws_proxy: not implemented")
      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Raised (Failure "ws_proxy: not implemented")
      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] 1/1 xapi @ xs Raised at file (Thread 33053260 has no backtrace table. Was with_backtraces called?, line 0
      Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace]
      

      So far i've tested with XO built with XenOrchestraInstallerUpdater (obviously ;)) and also built one manually, same symptoms. I've tried XO with and without reverse proxy, with http, with https, with one in same private network as XCP-ng host, with one connecting to it over VPN. No special firewalling or filtering in between and no HTTP proxy.

      But as the error from host says, it sounds like it’s simply missing support for this type of console connection.

      E: and just for the record. wsproxy package is installed:

      # rpm -q wsproxy
      wsproxy-1.6.0-4.el7.centos.x86_64
      
      posted in Xen Orchestra
      R
      ronivay
    • RE: Xen orchestra VM Console broken?

      I can also reproduce this with XCP-ng 7.6. I’ll try to do some additional tests. Let me know if there’s anything i can assist with.

      posted in Xen Orchestra
      R
      ronivay
    • RE: Unable to deploy a proxy

      This is expected. See for example: https://xcp-ng.org/forum/topic/3951/is-xo-proxy-available-in-community-version/5?_=1649088214690

      It is possible to build proxy manually and even attach it to XO (with somewhat limited functionality) but there is no documented procedure yet at least.

      posted in Xen Orchestra
      R
      ronivay
    • RE: Update and passwd

      It's quite normal that any new package that adds a service user modifies passwd file or some patch could've added/modified such user. Check the passwd backup file located in /etc/passwd- if its modification time matches the time you ran updates. If yes, then you can just diff passwd and passwd- to see what was actually changed.

      posted in Compute
      R
      ronivay
    • RE: Backup report template

      Wouldn't that support something like doesn't match pattern? I got similar monitoring, although i have a custom script that does the check but essentially i just look for status being successand if not, then alert.

      posted in Xen Orchestra
      R
      ronivay
    • RE: list backup times in .... time format?

      Just out of curiosity, how would you present something like "On monday, wednesday, friday, sunday, every other month at 4:30AM" or "Every day, every six hours, 15minutes past in january, march, june, september, december" in date format on that tiny space except showing the exact next scheduled time which doesn't tell anything really. Imo the cron time in that view is the best for quick overview. For those who want more details or don't understand cron time can do schedules without knowing anything about it from the edit view and also see the future scheduled execution times in date format.

      Sure i get the idea for tasks that run every day at specific time, that'd be easy but as the examples above show, there can be quite complex cron schedules that would be difficult to show.

      posted in Xen Orchestra
      R
      ronivay
    • RE: [Solved] create XO account from ssh

      Yeah you're right, it indeed should be there already. I must have done something to it in my own lab 😄

      posted in Xen Orchestra
      R
      ronivay