@julien-f haven't seen OOM's after this so looks good, thank you.

@isener just run /opt/xo/xo-server/dist/recover-account-cli.mjs admin@admin.net

Yup, installed nodejs version is beyond LTS version, which could be the cause. It is probably installed manually outside of the script as well since it installs 16.x if missing/too old but doesn’t do anything if version is above.

@florent thanks, yeah this seems to fix it again.

Did some further testing if amount of DNS queries would correlate to the amount of actual connections made to the host. This doesn't seem to be the case which is even more interesting Some results below.

Ran an incremental from delta backup which took in total of 9 minutes:

• Amount of DNS queries: close to 7k
• Amount of HTTPS connects logged to host IP-address: 478.
• Amount of HTTPS connects/disconnects logged in total to host IP-address: 955

Connection counts were about the same with installation from dns.lookup branch provided by @julien-f above, without the amount of DNS queries obviously.

It's quite normal that any new package that adds a service user modifies passwd file or some patch could've added/modified such user. Check the passwd backup file located in /etc/passwd- if its modification time matches the time you ran updates. If yes, then you can just diff passwd and passwd- to see what was actually changed.

@isener said in Forgot password for the webgui:

and when I ping, I get:
ping: xo.my-company.net: Name or service not known

what is wrong?

There is no DNS record for xo.my-company.net or you can’t resolve it from wherever you’re trying now.

xo.my-company.net -> which ip address should this be:
of the xcp-neg server or the server where the xen-orchestar is installed?

from where to run this command:
xo-cli --register http://xo.my-company.net/api/admin@admin.net
the xcp-neg server or the server where the xen-orchestar is installed?

Answer to both would be where XO is installed.

@isener said in Forgot password for the webgui:

I read somewhere that yiu have to use this command in CLI
xo-server-recover-account youremail@here.com

but how to install CLI?

This does not mean you need xo-cli installed. CLI refers to just active shell to your XOA VM, via console or ssh. Just run the command in the shell. If you have XO from sources, there's no such command alias and you need to run the script directly from where your XO is installed, like <installdir>/packages/xo-server/dist/recover-account-cli.mjs

I'm running daily installation from sources on multiple different OS's. All have the same specs: 2vCPU/4GB RAM. This has worked flawlessly for a long time. Recently (starting from 4th/5th Nov) i've started to see OOM errors almost daily during yarn build which then cause it to fail with following error:

Using polyfills: No polyfills were added, since the `useBuiltIns` option was not set.
[01:23:25] Finished 'copyAssets' after 36 s
[01:24:33] Finished 'buildScripts' after 1.73 min
[01:24:33] Finished 'build' after 1.73 min
✖ 1
error Command failed with exit code 1.

It isn't consistent, sometimes it's debian that fails, sometimes ubuntu, sometimes centos/almalinux and so on. Something has definitely changed in the build procedure that eats more RAM than it used to.

I’m fine with increasing the RAM if needed. Just wanted to point this out if there’s something out of the ordinary with latest changes.

Increasing VM disk size from XCP-ng center/XO only makes the block device larger, partition and filesystem (what you see in df output) won’t resize automatically, you need to do it manually inside the VM.

Sounds like a client side error if it happens with two different remotes on different systems, which probably explains why there’s nothing in server side logs.

Which OS is XO running? Could this be related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949394

Could you try mv on smb share without XO involved to verify it’s not related? Sure it should report other than success if such renaming fails.

@isener just run /opt/xo/xo-server/dist/recover-account-cli.mjs admin@admin.net

@isener said in Forgot password for the webgui:

and when I ping, I get:
ping: xo.my-company.net: Name or service not known

what is wrong?

There is no DNS record for xo.my-company.net or you can’t resolve it from wherever you’re trying now.

xo.my-company.net -> which ip address should this be:
of the xcp-neg server or the server where the xen-orchestar is installed?

from where to run this command:
xo-cli --register http://xo.my-company.net/api/admin@admin.net
the xcp-neg server or the server where the xen-orchestar is installed?

Answer to both would be where XO is installed.

@isener said in Forgot password for the webgui:

I read somewhere that yiu have to use this command in CLI
xo-server-recover-account youremail@here.com

but how to install CLI?

This does not mean you need xo-cli installed. CLI refers to just active shell to your XOA VM, via console or ssh. Just run the command in the shell. If you have XO from sources, there's no such command alias and you need to run the script directly from where your XO is installed, like <installdir>/packages/xo-server/dist/recover-account-cli.mjs

@julien-f haven't seen OOM's after this so looks good, thank you.

I'm running daily installation from sources on multiple different OS's. All have the same specs: 2vCPU/4GB RAM. This has worked flawlessly for a long time. Recently (starting from 4th/5th Nov) i've started to see OOM errors almost daily during yarn build which then cause it to fail with following error:

Using polyfills: No polyfills were added, since the `useBuiltIns` option was not set.
[01:23:25] Finished 'copyAssets' after 36 s
[01:24:33] Finished 'buildScripts' after 1.73 min
[01:24:33] Finished 'build' after 1.73 min
✖ 1
error Command failed with exit code 1.

It isn't consistent, sometimes it's debian that fails, sometimes ubuntu, sometimes centos/almalinux and so on. Something has definitely changed in the build procedure that eats more RAM than it used to.

I’m fine with increasing the RAM if needed. Just wanted to point this out if there’s something out of the ordinary with latest changes.

Settings -> audit. Assuming you have it enabled. If not, nowhere afaik.

Increasing VM disk size from XCP-ng center/XO only makes the block device larger, partition and filesystem (what you see in df output) won’t resize automatically, you need to do it manually inside the VM.

There’s usually little to no reasons to expose any mgmt systems to internet in actual production environments. Especially if it’s strictly for internal mgmt purposes. XO is no exception. Not because the system would be unsecure, but you simply want to make any attack surface as small as possible. It’s just a best a practice.

Okay, nodejs version is up to date then, so XO update will probably succeed as well without much of a hickup. Sure it’s safer if you can create a installation from scratch. XO gui has config export/import feature which you can use to migrate to new installation.

@JanP so you actually didn't update anything, probably just re-ran the installation on top of old codebase. See from sources installation instructions in: https://xen-orchestra.com/docs/installation.html#from-the-sources. Take note that it's perfectly possible that you also need to upgrade your nodejs version to latest LTS.

That'd be the first thing to solve as your installation is very old. It's therefore pointless to debug any issues until you get that XO to latest version as issue you're seeing could've very well be fixed long time ago.

So the answer is no, that is like couple years old. I assume it’s from sources installation then.

Start off by upgrading it to latest version. Instructed also here: https://xen-orchestra.com/docs/community.html