Backup jobs started to fail after last XOA update

Forza

Since the latest XOA update last week I am getting failed backup jobs. They look like this:

•	UUID: 5ee88933-d28a-f4ca-7dde-7490f3564398
•	Start time: Saturday, December 3rd 2022, 4:31:43 pm
•	End time: Sunday, December 4th 2022, 4:33:09 pm
•	Duration: a day
•	Error: self-signed certificate

It seems I had to enable "Unauthorized Certificates" in the servers settings to make backups work again. Is this a new setting or is this just a coincidence somehow?

Current version: 5.77.0 - XOA build: 20210102

- node: 18.12.1
- npm: 8.19.2
- xen-orchestra-upload-ova: 0.1.4
- xo-server: 5.107.1
- xo-server-auth-github-enterprise: 0.2.2
- xo-server-auth-google-enterprise: 0.2.2
- xo-server-auth-ldap-enterprise: 0.10.6
- xo-server-auth-saml-enterprise: 0.10.1
- xo-server-backup-reports-enterprise: 0.17.2
- xo-server-netbox-enterprise: 0.3.5
- xo-server-telemetry: 0.5.0
- xo-server-transport-email-enterprise: 0.6.0
- xo-server-transport-icinga2-enterprise: 0.1.1
- xo-server-transport-nagios-enterprise: 1.0.0
- xo-server-transport-slack-enterprise: 0.0.0
- xo-server-transport-xmpp-enterprise: 0.1.1
- xo-server-usage-report-enterprise: 0.10.2
- xo-server-xoa: 0.17.0
- xo-web-enterprise: 5.108.0
- xoa-cli: 0.33.0
- xoa-updater: 0.45.0

Looking at the error log I see that XOA is trying HTTPS instead of HTTP when communicating with the XCP-ng host:

"result": {
        "code": "DEPTH_ZERO_SELF_SIGNED_CERT",
        "url": "https://10.254.9.2/export/?ref=OpaqueRef:3377f666-5fdc-41e5-aebf-39c7ffa5ad3f&use_compression=zstd&session_id=OpaqueRef:dd989ec7-60ac-45af-8fcd-faf542f8e62a&task_id=OpaqueRef:5617043e-af92-4023-9482-043094180083",
...
        "VM": "OpaqueRef:3377f666-5fdc-41e5-aebf-39c7ffa5ad3f",
        "message": "self-signed certificate",
        "name": "Error",
        "stack": "Error: self-signed certificate\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1538:34)\n    at TLSSocket.emit (node:events:513:28)\n    at TLSSocket.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:135:17)\n    at TLSSocket._finishInit (node:_tls_wrap:952:8)\n    at ssl.onhandshakedone (node:_tls_wrap:733:12)\n    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)"
      }

olivierlambert

Hi,

Are you using XO from the sources or XOA?

edit: it seems to be XOA, missed that later in your post

Does it ring bell @julien-f ?

Forza

@olivierlambert said in Backup jobs started to fail after last XOA update:

Indeed, I use XOA with Enterprise license

olivierlambert

In this case, it's better to create support tickets than use the forum, this way your issue can be treated in priority

julien-f

@Forza This is perfectly normal, XO is using HTTPS to connect to hosts by default for many years, and if you are not using an HTTPS certificate trusted by common authorities, you need to enable this setting.

Forza

@julien-f said in Backup jobs started to fail after last XOA update:

@Forza This is perfectly normal, XO is using HTTPS to connect to hosts by default for many years, and if you are not using an HTTPS certificate trusted by common authorities, you need to enable this setting.

I know https:// is default, but I remeber we could enable http:// to improve backup and migration performance. Is not possible any more?

julien-f

@Forza Yes, it's possible, simply prefix your host address by http:// in Settings/Servers.

If that does not work, open a support ticket and a tunnel and I'll investigate

Forza

@julien-f said in Backup jobs started to fail after last XOA update:

@Forza Yes, it's possible, simply prefix your host address by http:// in Settings/Servers.

If that does not work, open a support ticket and a tunnel and I'll investigate

This is why I was surprised. I have been using http:// for a long time now. But since I changed the "Unauthorized Certificates" it started to work again.

If you think it is important, I can open a tunnel for you to look?

sluflyer06

@Forza Wow, I just tried that out and backup speed increased by 48%.

julien-f

@Forza Yes please

@sluflyer06 We are currently investigating this, whether the perf issue comes from XO or XCP-ng.

julien-f

For anyone interested:

Okay, after investigation, it appears that when the pool master is an HTTP request for exporting a VDI on another host (for instance because the VDI is on a local SR), it explicitly specify HTTPS in the new URL.

I'm not sure it's a good idea to ignore this as I'm afraid it may cause issues in the future.

Anyway XO behavior has not changed recently as it comes from the host.

We are currently investing the perf impact of HTTPS and hopefully we'll find a way to improve it.

Forza

@julien-f Would it help if I add the pool member to the "Servers" list as http:// ?

julien-f

@Forza No, it's not possible to add multiple hosts of the same pool to XO.