XOA 6.8 Pool Metadata backup

flakpyro

Since upgrading to XOA 6.8 i'm experiencing XOA pool metadata backup failures. The task will run and a number of pools will successfully backup however a random set of pools will stall until the job eventually fails after 20 mins. The error is "Body Timeout Error". I have tried restarting the toolstack on the hosts as well as restarting XOA. It is not always the same hosts each time i retry either.

I would attach the log but it is quite long as it seems to show successes as well as failures

"id": "0mtln3sci-stmgt9yih8i",
      "start": 1788446895426,
      "status": "failure",
      "tasks": [
        {
          "id": "0mtln3sde-8lp6c7ei6un",
          "start": 1788446895458,
          "status": "failure",
          "end": 1788448100196,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/dispatcher/client-h1.js:703:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (19bda980-70ec-bada-5d9d-6f35c8a2122b) for the remote (5b9abfa7-f5f1-4e80-bfde-cabe82864988). (5220a296-765d-4023-acc7-70d4ae66fe1e)",
          "data": {
            "id": "5b9abfa7-f5f1-4e80-bfde-cabe82864988",
            "type": "remote",
            "progress": 0
          }

Danp

Please switch back to the Stable release channel and then let us know if the issue goes away.

Details for switching release channels can be found here -- https://help.vates.tech/help/en-us/30-versions-and-updates/21-xoa-versions

flakpyro

@Danp I would normally do this and report back however i am running a custom patch from @florent at the moment that he deployed to fix the "Fetch Failed" backup proxy issue. Once 6.8.2 has been released which that patch i should be able to switch between versions and test this!

acebmxer

@flakpyro The patch was pushed though to XO from Sources. Hasnt hit XOA yet i dont beleive.

https://github.com/vatesfr/xen-orchestra/commit/003cc6f9cc7de6bff9eaee9fa0bc14a1f7b64cc0

I am not seeing this error after patch applied to XOA...

fix(proxy): fetch failed  (#10346)
* fix(backups): fetch failed

disable compresion between xoa on proxy on xoa side

* fix(xo-proxy): disable compression on the proxy

it was disable with hrp, replaced by #10038 since it was never
sending accept-encoding header

compression can be put back later on a route by route basis after
careful checks ( for example : file restore, backup listing )

flakpyro

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

poddingue

6.8.2 went out, and it carries florent's proxy patch, so the thing that was stopping you switching channels has cleared.

Before you do, though, the 6.8.2 changelog lists exactly one bug fix, the fetch failed proxy one, with xo-server at 5.208.3 and the proxy at 0.31.10, and there's nothing in it about metadata backups or BodyTimeoutError.

Your own post above points the same way, since you're seeing the stall on sources at the latest commit, and that commit already has the proxy fix in it. So I'd treat the metadata timeout as a separate bug that 6.8.2 won't touch, and I couldn't find an open issue covering it, though I might well have searched the wrong words.

Might be worth a mention to @Team-XO-Backend, mostly so the metadata half doesn't quietly get filed as fixed by 6.8.2.

flakpyro

@poddingue This was fixed a number of weeks ago via a ticket. I believe @florent may know the exact fix.