Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

bogikornel

@poddingue httpInactivityTimeout changed the only thing for me, I received the email half an hour later that the metadata could not be saved.

marcoi

looks like im getting the xo config backup body timeout error as well.
Xen Orchestra, commit a94eb
Master, commit 9c6d1

  "result": {
    "message": "backup task failed with undefined error",
    "name": "Error",
    "stack": "Error: backup task failed with undefined error\n    at forwardResult (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/_handleBackupLog.mjs:37:25)\n    at handleBackupLog (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/_handleBackupLog.mjs:68:12)\n    at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/metadata-backups.mjs:133:13)\n    at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202608030700/@xen-orchestra/mixins/Tasks.mjs:205:23)\n    at Task.onProgress [as _onProgress] (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/combineEvents.js:61:5)\n    at Task.#emit (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:157:10)\n    at Task.#end (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:168:15)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:190:16)\n    at Jobs.runJob (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n    at Jobs.runJobSequence (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/jobs/index.mjs:339:7)\n    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:204:22)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:188:20)\n    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/api.mjs:475:18)"
  }

JB

@christopher-petzel Olá Chris!
Yes, this is a local XCP-ng management network and there are no other backups simultaneous.

JB

@poddingue Olá! The failure has a fixed duration of 5 minutes.

acebmxer

Just got a failure today but with a different pool.

    "message": "backup task failed with undefined error",
    "name": "Error",
    "stack": "Error: backup task failed with undefined error\n    at forwardResult (file:///usr/local/lib/node_modules/xo-server/src/_handleBackupLog.mjs:37:25)\n    at handleBackupLog (file:///usr/local/lib/node_modules/xo-server/src/_handleBackupLog.mjs:68:12)\n    at onTaskUpdate (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/metadata-backups.mjs:117:13)\n    at onTaskUpdate (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/mixins/Tasks.mjs:205:23)\n    at onLogFct (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/combineEvents.js:61:5)\n    at metadataBackup._executor (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/metadata-backups.mjs:122:11)\n    at Jobs.runJob (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n    at file:///usr/local/lib/node_modules/xo-server/src/api/schedule.mjs:84:9\n    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:204:22)\n    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:188:20)\n    at Xo.runSequence (file:///usr/local/lib/node_modules/xo-server/src/api/schedule.mjs:73:3)\n    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:204:22)\n    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:188:20)\n    at Api.#callApiMethod (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/api.mjs:475:18)"
  }
}

JB

Every day I update XO-CE hoping this error gets fixed, but so far, no solution!

olivierlambert

Is there a way in private message to give access to the XCP-ng host and XO VM in question OR to get all logs from 2 machines somewhere? 5min is clearly a timeout error.

acebmxer

@olivierlambert

Mine has not done it since last post but i have opened a support ticket along with tunnel being opened - Ticket#7762089

JB

@olivierlambert Yes, I can grant it.

marcoi

looks like mine also just started working.

JB

Master, commit be960

olivierlambert

@JB I'd like to test a different way of investigating the problem. So I won't do it for many people, just send me a private message with a way to SSH to your host and XOA, promise it's not a scam

JB

@marcoi Try doing it three times, and the problem comes back.

marcoi

@JB seems okay after a few back to back runs.

JB

@marcoi Does your backup include METADATA and XO CONFIG?

marcoi

@JB both
Also Xen Orchestra, commit be960

JB

@marcoi The problem is with the METADATA backup. XO Config is working fine!

JB

@marcoi So the problem is with my infrastructure.

JB

@marcoi I see that you use NFS to store the backup. I use SMB.

JB

This post is deleted!