XOA 6.8 Pool Metadata backup

flakpyro

Since upgrading to XOA 6.8 i'm experiencing XOA pool metadata backup failures. The task will run and a number of pools will successfully backup however a random set of pools will stall until the job eventually fails after 20 mins. The error is "Body Timeout Error". I have tried restarting the toolstack on the hosts as well as restarting XOA. It is not always the same hosts each time i retry either.

I would attach the log but it is quite long as it seems to show successes as well as failures

"id": "0mtln3sci-stmgt9yih8i",
      "start": 1788446895426,
      "status": "failure",
      "tasks": [
        {
          "id": "0mtln3sde-8lp6c7ei6un",
          "start": 1788446895458,
          "status": "failure",
          "end": 1788448100196,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/dispatcher/client-h1.js:703:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (19bda980-70ec-bada-5d9d-6f35c8a2122b) for the remote (5b9abfa7-f5f1-4e80-bfde-cabe82864988). (5220a296-765d-4023-acc7-70d4ae66fe1e)",
          "data": {
            "id": "5b9abfa7-f5f1-4e80-bfde-cabe82864988",
            "type": "remote",
            "progress": 0
          }

Danp

Please switch back to the Stable release channel and then let us know if the issue goes away.

Details for switching release channels can be found here -- https://help.vates.tech/help/en-us/30-versions-and-updates/21-xoa-versions

flakpyro

@Danp I would normally do this and report back however i am running a custom patch from @florent at the moment that he deployed to fix the "Fetch Failed" backup proxy issue. Once 6.8.2 has been released which that patch i should be able to switch between versions and test this!

acebmxer

@flakpyro The patch was pushed though to XO from Sources. Hasnt hit XOA yet i dont beleive.

https://github.com/vatesfr/xen-orchestra/commit/003cc6f9cc7de6bff9eaee9fa0bc14a1f7b64cc0

I am not seeing this error after patch applied to XOA...

fix(proxy): fetch failed  (#10346)
* fix(backups): fetch failed

disable compresion between xoa on proxy on xoa side

* fix(xo-proxy): disable compression on the proxy

it was disable with hrp, replaced by #10038 since it was never
sending accept-encoding header

compression can be put back later on a route by route basis after
careful checks ( for example : file restore, backup listing )

flakpyro

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.