@florent thanks, yeah this seems to fix it again.

Did some further testing if amount of DNS queries would correlate to the amount of actual connections made to the host. This doesn't seem to be the case which is even more interesting Some results below.

Ran an incremental from delta backup which took in total of 9 minutes:

• Amount of DNS queries: close to 7k
• Amount of HTTPS connects logged to host IP-address: 478.
• Amount of HTTPS connects/disconnects logged in total to host IP-address: 955

Connection counts were about the same with installation from dns.lookup branch provided by @julien-f above, without the amount of DNS queries obviously.

It's quite normal that any new package that adds a service user modifies passwd file or some patch could've added/modified such user. Check the passwd backup file located in /etc/passwd- if its modification time matches the time you ran updates. If yes, then you can just diff passwd and passwd- to see what was actually changed.

I can reproduce this (ref: https://github.com/ronivay/XenOrchestraInstallerUpdater/issues/133) and it seems to be caused by commit: https://github.com/vatesfr/xen-orchestra/commit/49890a09b7a3c4bcbe7f462577c6d0a7da8b7006

If you start from fresh (no redis database specifically) and install from that commit, or latest master, the default user creation doesn't work properly and above mentioned error is present in xo-server logs. It's actually all user creations (their passwords to be exact), not just default user. Doesn't affect existing users so if updating from previous version, everything seemingly works fine until new user is created and login with it fails.

If installing from commit: https://github.com/vatesfr/xen-orchestra/commit/dfce56cee8023ce0dc9292e0c1dda5e6937910e5 with fresh redis database, all works as it should. Also if broken admin@admin.net user was already created with installation from newer commit and redis database wasn't cleared, resetting user password with recover-account-cli.mjs in installation from this commit brings it back to life again.

rjlan created this issue in ronivay/XenOrchestraInstallerUpdater

open invalid credentials on login to fresh install #133

0 julien-f committed to vatesfr/xen-orchestra

chore: update dev deps

0 julien-f committed to vatesfr/xen-orchestra

feat(async-each): add basic JsDoc typing

@julien-f this changed the situation from thousands of queries in minutes to no noticeable spike in query graphs during backup job, so huge improvement.

Could very well be. With multiple hosts and simultaneous backups, the amount of requests can easily be overwhelming if it multiplies in such scenario. I've actually created an issue about stalled backup due to failed DNS resolution back in 2019 (https://github.com/vatesfr/xen-orchestra/issues/4122). I didn't dig this deep into it then and kinda assumed it was something specific to my environment and switched to using just IP-address. Switched back to domain name at some point. Personally haven't seen failed backups due to same reason recently so could be it retries nowadays at least to some extent.

/etc/hosts sure is also an option but as a static config it takes away the benefit you get from using DNS.

ronivay created this issue in vatesfr/xen-orchestra

closed Failed name resolution stalling backup #4122

Thanks. This seems to fix it for me at least. XO logs UNEXPECTED RESPONSE and FALLBACK for those HVM VM's and console opens up fine.

Locate the packages/xo-server directory to your from sources built xen orchestra instance. There should be a file in dist/recover-account-cli.mjs, copy this file as dist/recover-account-cli.mjs and then run it from the dist directory ./recover-account-cli.mjs admin@something.tld, it'll ask password and either reset the existing users password or create a new one with admin privileges if there isn't one with the email address.

Plugin configuration has good examples for those attributes. There really is no tutorial for your specific environment, depends how your AD is setup. If you have other applications setup with ldap, try to look for hints from their configuration. Note that it does not sync users, they’re being created on successful login after plugin is configured properly and activated.

That "VM" is a snapshot of that running XOA - xbackup1 machine. Open that VM in Xen Orchestra and check snapshots tab, you'll see that the name and uuid (copy button next to snapshot) matches the results from command line.

You can ignore snapshots in listing with: xe vm-list is-a-snapshot=false

I can reproduce this (ref: https://github.com/ronivay/XenOrchestraInstallerUpdater/issues/133) and it seems to be caused by commit: https://github.com/vatesfr/xen-orchestra/commit/49890a09b7a3c4bcbe7f462577c6d0a7da8b7006

If you start from fresh (no redis database specifically) and install from that commit, or latest master, the default user creation doesn't work properly and above mentioned error is present in xo-server logs. It's actually all user creations (their passwords to be exact), not just default user. Doesn't affect existing users so if updating from previous version, everything seemingly works fine until new user is created and login with it fails.

If installing from commit: https://github.com/vatesfr/xen-orchestra/commit/dfce56cee8023ce0dc9292e0c1dda5e6937910e5 with fresh redis database, all works as it should. Also if broken admin@admin.net user was already created with installation from newer commit and redis database wasn't cleared, resetting user password with recover-account-cli.mjs in installation from this commit brings it back to life again.

rjlan created this issue in ronivay/XenOrchestraInstallerUpdater

open invalid credentials on login to fresh install #133

0 julien-f committed to vatesfr/xen-orchestra

chore: update dev deps

0 julien-f committed to vatesfr/xen-orchestra

feat(async-each): add basic JsDoc typing

How should it work if job does not complete within the hour? If schedule would flex dynamically depending on last job completion time, wouldn’t it eventually be way off than actually scheduled?

In my opinion making hourly delta backups doesn’t make any sense and is something you should do in filesystem/file level inside VMs instead. Continuous replication and just snapshots should be fine, although the snapshot chain can easily grow too large and backups start to fail, so in that sense making snapshot rotation smaller would be better. Daily/weekly/monthly delta backup is understandable and makes perfect sense.

@florent thanks, yeah this seems to fix it again.

yeah it’s 7.6 i see this with

@florent I see that the issue is still open, but looks like PR https://github.com/vatesfr/xen-orchestra/pull/6191 was merged so i assume it's not WIP anymore? Just wanted to let you know that some of the changes in that PR before merge made the initial fix non functional. Tested with installation from latest master branch commit.

Errors in browser have changed slightly:

Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54
    Error util.js:44
    _fail rfb.js:578
    f rfb.js:233
    onclose websock.js:305

and

Tried changing state of a disconnected RFB object util.js:44:54
    Error util.js:44
    _updateConnectionState rfb.js:473
    disconnect rfb.js:284
    _clean react-novnc.js:66
    componentWillUnmount react-novnc.js:115
    React 139

XO doesn't log anything related to fallback method, just a normal console proxy INFO message. Host still gives the same Failure "ws_proxy: not implemented" error and seems that the fallback to TCP doesn't happen and console doesn't open. Still works for PV guests as it did before.

fbeauchamp created this issue in vatesfr/xen-orchestra

closed fix(xo-server/proxy-console): fallback on TCP if WS not available #6191

Yep. Same domain, asks A and AAAA at the same time, both being individual queries obviously.

Did some further testing if amount of DNS queries would correlate to the amount of actual connections made to the host. This doesn't seem to be the case which is even more interesting Some results below.

Ran an incremental from delta backup which took in total of 9 minutes:

• Amount of DNS queries: close to 7k
• Amount of HTTPS connects logged to host IP-address: 478.
• Amount of HTTPS connects/disconnects logged in total to host IP-address: 955

Connection counts were about the same with installation from dns.lookup branch provided by @julien-f above, without the amount of DNS queries obviously.

@julien-f this changed the situation from thousands of queries in minutes to no noticeable spike in query graphs during backup job, so huge improvement.

I think the main point to focus on here is that XO is doing totally unnecessary DNS queries with excessive frequency. I don't see this as implementing a non-core feature but a fix in the logic how application figures out where to connect and how often. How exactly and what options there are is outside of my knowledge

I'll put this to test and see tomorrow what the DNS query stats look like.

Just my two cents but i feel like one shouldn't "fix" a flaw or bad behaviour in application by relying on external dependency to deal with it, especially if it's fixable. Sure using something like nscd in XOA would kinda fix the issue in it but wouldn't possible perf issue etc still exist in node? I'm not competent to review the code so can't say anything about the actual implementation in feature branch.