@florent thanks, yeah this seems to fix it again.
Posts made by ronivay
-
RE: Xen orchestra VM Console broken?
-
RE: Xen orchestra VM Console broken?
@florent I see that the issue is still open, but looks like PR https://github.com/vatesfr/xen-orchestra/pull/6191 was merged so i assume it's not WIP anymore? Just wanted to let you know that some of the changes in that PR before merge made the initial fix non functional. Tested with installation from latest master branch commit.
Errors in browser have changed slightly:
Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54 Error util.js:44 _fail rfb.js:578 f rfb.js:233 onclose websock.js:305
and
Tried changing state of a disconnected RFB object util.js:44:54 Error util.js:44 _updateConnectionState rfb.js:473 disconnect rfb.js:284 _clean react-novnc.js:66 componentWillUnmount react-novnc.js:115 React 139
XO doesn't log anything related to fallback method, just a normal
console proxy
INFO message. Host still gives the sameFailure "ws_proxy: not implemented"
error and seems that the fallback to TCP doesn't happen and console doesn't open. Still works for PV guests as it did before. -
RE: DNS queries during backup job
Yep. Same domain, asks A and AAAA at the same time, both being individual queries obviously.
-
RE: DNS queries during backup job
Did some further testing if amount of DNS queries would correlate to the amount of actual connections made to the host. This doesn't seem to be the case which is even more interesting
Some results below.
Ran an incremental from delta backup which took in total of 9 minutes:
- Amount of DNS queries: close to 7k
- Amount of HTTPS connects logged to host IP-address: 478.
- Amount of HTTPS connects/disconnects logged in total to host IP-address: 955
Connection counts were about the same with installation from
dns.lookup
branch provided by @julien-f above, without the amount of DNS queries obviously. -
RE: DNS queries during backup job
@julien-f this changed the situation from thousands of queries in minutes to no noticeable spike in query graphs during backup job, so huge improvement.
-
RE: DNS queries during backup job
I think the main point to focus on here is that XO is doing totally unnecessary DNS queries with excessive frequency. I don't see this as implementing a non-core feature but a fix in the logic how application figures out where to connect and how often. How exactly and what options there are is outside of my knowledge
-
RE: DNS queries during backup job
I'll put this to test and see tomorrow what the DNS query stats look like.
Just my two cents but i feel like one shouldn't "fix" a flaw or bad behaviour in application by relying on external dependency to deal with it, especially if it's fixable. Sure using something like
nscd
in XOA would kinda fix the issue in it but wouldn't possible perf issue etc still exist in node? I'm not competent to review the code so can't say anything about the actual implementation in feature branch. -
RE: DNS queries during backup job
This is not really an issue for me, just an observation which could possibly have all sorts of undesired effects to many XO users.
I know multiple ways of getting around it, couple of them already mentioned here. Local caching with
nscd
is one more of those. Point is, this should rather be fixed in XO than worked around. -
RE: DNS queries during backup job
Could very well be. With multiple hosts and simultaneous backups, the amount of requests can easily be overwhelming if it multiplies in such scenario. I've actually created an issue about stalled backup due to failed DNS resolution back in 2019 (https://github.com/vatesfr/xen-orchestra/issues/4122). I didn't dig this deep into it then and kinda assumed it was something specific to my environment and switched to using just IP-address. Switched back to domain name at some point. Personally haven't seen failed backups due to same reason recently so could be it retries nowadays at least to some extent.
/etc/hosts
sure is also an option but as a static config it takes away the benefit you get from using DNS. -
DNS queries during backup job
I got following setup:
- XO from sources (latest master commit) connected to one XCP-ng host using DNS name
- Delta backup job with ~20VM's using local remote
I've recently kept an eye on DNS request done by XO. On average when idle it does around ~150-200 DNS requests per hour to get IP-address for host that is configured using DNS name. This is fine but when backup job starts, this amount goes through the roof:
My latest backup job ran from 04:00AM to 04:17AM. During this time, XO did 9320 identical requests, all asking for A/AAAA record for configured host. This averages ~550 requests per minute and ~9 requests per second. Obviously the number is not constant during backup job and i even saw it hit my DNS resolver rate limit of 1000 r/minute.
To me this sounds like something that could and should be improved. Although DNS requests are lightweight and results often cached by resolvers, they're still traffic in the network and obsolete queries should be avoided. It shouldn't be necessary to ask same information so many times during such short time frame.
Yeah sure i could just configure the host using IP-address and avoid this but it doesn't fix the underlying issue.
-
RE: Xen orchestra VM Console broken?
Thanks. This seems to fix it for me at least. XO logs
UNEXPECTED RESPONSE
andFALLBACK
for those HVM VM's and console opens up fine. -
RE: Xen orchestra VM Console broken?
Yeah good point, all those old ones are indeed PV guests and all the rest are HVM.
-
RE: Xen orchestra VM Console broken?
Interestingly console for some old VM's with old OS are working, like centos 6, debian 8 but anything above those (centos7, debian 10, debian 11) aren't. I'm getting same errors in browser as listed by @bberndt earlier. This can be found from host
xensource.log
Apr 14 09:55:45 xs xapi: [debug|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] VM OpaqueRef:be204b50-6028-41f7-9744-e94ad7329046 console port: Some unix:/var/run/xen/vnc-290 Apr 14 09:55:45 xs xapi: [error|xs|33053260 INET :::80|Connection to VM console R:7839b84e856e|console] No implementation for web-sockets console proxy to a Unix domain socket Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Connection to VM console R:7839b84e856e failed with exception (Failure "ws_proxy: not implemented") Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] Raised (Failure "ws_proxy: not implemented") Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace] 1/1 xapi @ xs Raised at file (Thread 33053260 has no backtrace table. Was with_backtraces called?, line 0 Apr 14 09:55:46 xs xapi: [error|xs|33053260 INET :::80||backtrace]
So far i've tested with XO built with XenOrchestraInstallerUpdater (obviously ;)) and also built one manually, same symptoms. I've tried XO with and without reverse proxy, with http, with https, with one in same private network as XCP-ng host, with one connecting to it over VPN. No special firewalling or filtering in between and no HTTP proxy.
But as the error from host says, it sounds like it’s simply missing support for this type of console connection.
E: and just for the record. wsproxy package is installed:
# rpm -q wsproxy wsproxy-1.6.0-4.el7.centos.x86_64
-
RE: Xen orchestra VM Console broken?
I can also reproduce this with XCP-ng 7.6. I’ll try to do some additional tests. Let me know if there’s anything i can assist with.
-
RE: Unable to deploy a proxy
This is expected. See for example: https://xcp-ng.org/forum/topic/3951/is-xo-proxy-available-in-community-version/5?_=1649088214690
It is possible to build proxy manually and even attach it to XO (with somewhat limited functionality) but there is no documented procedure yet at least.
-
RE: Update and passwd
It's quite normal that any new package that adds a service user modifies passwd file or some patch could've added/modified such user. Check the passwd backup file located in
/etc/passwd-
if its modification time matches the time you ran updates. If yes, then you can just diffpasswd
andpasswd-
to see what was actually changed. -
RE: Backup report template
Wouldn't that support something like doesn't match pattern? I got similar monitoring, although i have a custom script that does the check but essentially i just look for status being
success
and if not, then alert. -
RE: list backup times in .... time format?
Just out of curiosity, how would you present something like "On monday, wednesday, friday, sunday, every other month at 4:30AM" or "Every day, every six hours, 15minutes past in january, march, june, september, december" in date format on that tiny space except showing the exact next scheduled time which doesn't tell anything really. Imo the cron time in that view is the best for quick overview. For those who want more details or don't understand cron time can do schedules without knowing anything about it from the edit view and also see the future scheduled execution times in date format.
Sure i get the idea for tasks that run every day at specific time, that'd be easy but as the examples above show, there can be quite complex cron schedules that would be difficult to show.
-
RE: [Solved] create XO account from ssh
Yeah you're right, it indeed should be there already. I must have done something to it in my own lab