Posts made by ronivay

@florent thanks, yeah this seems to fix it again.

yeah it’s 7.6 i see this with

@florent I see that the issue is still open, but looks like PR https://github.com/vatesfr/xen-orchestra/pull/6191 was merged so i assume it's not WIP anymore? Just wanted to let you know that some of the changes in that PR before merge made the initial fix non functional. Tested with installation from latest master branch commit.

Errors in browser have changed slightly:

Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54
    Error util.js:44
    _fail rfb.js:578
    f rfb.js:233
    onclose websock.js:305

and

Tried changing state of a disconnected RFB object util.js:44:54
    Error util.js:44
    _updateConnectionState rfb.js:473
    disconnect rfb.js:284
    _clean react-novnc.js:66
    componentWillUnmount react-novnc.js:115
    React 139

XO doesn't log anything related to fallback method, just a normal console proxy INFO message. Host still gives the same Failure "ws_proxy: not implemented" error and seems that the fallback to TCP doesn't happen and console doesn't open. Still works for PV guests as it did before.

fbeauchamp created this issue in vatesfr/xen-orchestra

closed fix(xo-server/proxy-console): fallback on TCP if WS not available #6191

Yep. Same domain, asks A and AAAA at the same time, both being individual queries obviously.

Did some further testing if amount of DNS queries would correlate to the amount of actual connections made to the host. This doesn't seem to be the case which is even more interesting Some results below.

Ran an incremental from delta backup which took in total of 9 minutes:

• Amount of DNS queries: close to 7k
• Amount of HTTPS connects logged to host IP-address: 478.
• Amount of HTTPS connects/disconnects logged in total to host IP-address: 955

Connection counts were about the same with installation from dns.lookup branch provided by @julien-f above, without the amount of DNS queries obviously.

@julien-f this changed the situation from thousands of queries in minutes to no noticeable spike in query graphs during backup job, so huge improvement.

I think the main point to focus on here is that XO is doing totally unnecessary DNS queries with excessive frequency. I don't see this as implementing a non-core feature but a fix in the logic how application figures out where to connect and how often. How exactly and what options there are is outside of my knowledge

I'll put this to test and see tomorrow what the DNS query stats look like.

Just my two cents but i feel like one shouldn't "fix" a flaw or bad behaviour in application by relying on external dependency to deal with it, especially if it's fixable. Sure using something like nscd in XOA would kinda fix the issue in it but wouldn't possible perf issue etc still exist in node? I'm not competent to review the code so can't say anything about the actual implementation in feature branch.

This is not really an issue for me, just an observation which could possibly have all sorts of undesired effects to many XO users.

I know multiple ways of getting around it, couple of them already mentioned here. Local caching with nscd is one more of those. Point is, this should rather be fixed in XO than worked around.

Could very well be. With multiple hosts and simultaneous backups, the amount of requests can easily be overwhelming if it multiplies in such scenario. I've actually created an issue about stalled backup due to failed DNS resolution back in 2019 (https://github.com/vatesfr/xen-orchestra/issues/4122). I didn't dig this deep into it then and kinda assumed it was something specific to my environment and switched to using just IP-address. Switched back to domain name at some point. Personally haven't seen failed backups due to same reason recently so could be it retries nowadays at least to some extent.

/etc/hosts sure is also an option but as a static config it takes away the benefit you get from using DNS.

ronivay created this issue in vatesfr/xen-orchestra

closed Failed name resolution stalling backup #4122

I got following setup:

• XO from sources (latest master commit) connected to one XCP-ng host using DNS name
• Delta backup job with ~20VM's using local remote

I've recently kept an eye on DNS request done by XO. On average when idle it does around ~150-200 DNS requests per hour to get IP-address for host that is configured using DNS name. This is fine but when backup job starts, this amount goes through the roof:

My latest backup job ran from 04:00AM to 04:17AM. During this time, XO did 9320 identical requests, all asking for A/AAAA record for configured host. This averages ~550 requests per minute and ~9 requests per second. Obviously the number is not constant during backup job and i even saw it hit my DNS resolver rate limit of 1000 r/minute.

To me this sounds like something that could and should be improved. Although DNS requests are lightweight and results often cached by resolvers, they're still traffic in the network and obsolete queries should be avoided. It shouldn't be necessary to ask same information so many times during such short time frame.

Yeah sure i could just configure the host using IP-address and avoid this but it doesn't fix the underlying issue.

Thanks. This seems to fix it for me at least. XO logs UNEXPECTED RESPONSE and FALLBACK for those HVM VM's and console opens up fine.

Yeah good point, all those old ones are indeed PV guests and all the rest are HVM.

Interestingly console for some old VM's with old OS are working, like centos 6, debian 8 but anything above those (centos7, debian 10, debian 11) aren't. I'm getting same errors in browser as listed by @bberndt earlier. This can be found from host xensource.log

Apr 14 09:55:45 xs xapi: [debug|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] VM OpaqueRef:be204b50-6028-41f7-9744-e94ad7329046 console port: Some unix:/var/run/xen/vnc-290
Apr 14 09:55:45 xs xapi: [error|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] No implementation for web-sockets console proxy to a Unix domain socket
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Connection to VM console R:7839b84e856e failed with exception (Failure "ws_proxy: not implemented")
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Raised (Failure "ws_proxy: not implemented")
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] 1/1 xapi @ xs Raised at file (Thread 33053260 has no backtrace table. Was with_backtraces called?, line 0
Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace]

So far i've tested with XO built with XenOrchestraInstallerUpdater (obviously ;)) and also built one manually, same symptoms. I've tried XO with and without reverse proxy, with http, with https, with one in same private network as XCP-ng host, with one connecting to it over VPN. No special firewalling or filtering in between and no HTTP proxy.

But as the error from host says, it sounds like it’s simply missing support for this type of console connection.

E: and just for the record. wsproxy package is installed:

# rpm -q wsproxy
wsproxy-1.6.0-4.el7.centos.x86_64

I can also reproduce this with XCP-ng 7.6. I’ll try to do some additional tests. Let me know if there’s anything i can assist with.

This is expected. See for example: https://xcp-ng.org/forum/topic/3951/is-xo-proxy-available-in-community-version/5?_=1649088214690

It is possible to build proxy manually and even attach it to XO (with somewhat limited functionality) but there is no documented procedure yet at least.

It's quite normal that any new package that adds a service user modifies passwd file or some patch could've added/modified such user. Check the passwd backup file located in /etc/passwd- if its modification time matches the time you ran updates. If yes, then you can just diff passwd and passwd- to see what was actually changed.

Wouldn't that support something like doesn't match pattern? I got similar monitoring, although i have a custom script that does the check but essentially i just look for status being successand if not, then alert.

Just out of curiosity, how would you present something like "On monday, wednesday, friday, sunday, every other month at 4:30AM" or "Every day, every six hours, 15minutes past in january, march, june, september, december" in date format on that tiny space except showing the exact next scheduled time which doesn't tell anything really. Imo the cron time in that view is the best for quick overview. For those who want more details or don't understand cron time can do schedules without knowing anything about it from the edit view and also see the future scheduled execution times in date format.

Sure i get the idea for tasks that run every day at specific time, that'd be easy but as the examples above show, there can be quite complex cron schedules that would be difficult to show.

Yeah you're right, it indeed should be there already. I must have done something to it in my own lab