Mirror Incremental Bkp - Error Maximum call stack size exceeded

mguimond

bkp_log.txt
Here are the log.

My mirror incremental bkp always get stuck with one specific vm.

   "result": {
            "message": "Maximum call stack size exceeded",
            "name": "RangeError",
            "stack": "RangeError: Maximum call stack size exceeded\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:418:7)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)\n    at getUsedChildChainOrDelete (file:///opt/xo/xo-builds/xen-orchestra-202401192117/@xen-orchestra/backups/_cleanVm.mjs:433:23)"
=============

olivierlambert

Hi,

Can you provide a bit more information on your setup, the XO version you use and such

mguimond

Xen Orchestra commit ec166

XCP-NG 8.2.1 (only one host)

Let me know what specific information you need if I'm missing some.

olivierlambert

commit ec1669a32 is almost 3 months old. You can't report an issue without being on the latest commit in our repo, as per our doc:

https://xen-orchestra.com/docs/community.html#current-version

Please upgrade/rebuild and if you reproduce, we'll take a look

mguimond

@olivierlambert Perfect, I will work on this and come back to you.

mguimond

@olivierlambert I did the update and I'm getting the same error.

Herer are the log 2024-04-11T11_47_11.366Z - backup NG.txt

Sense my mirror incremental backup take a while to do the copy because of internet speed. I had to do update to another commit this morning. I will retest de backup job but still getting the error on the latest commit from last week.

olivierlambert

What Node version are you using?

mguimond

@olivierlambert Node v18.20.2

mguimond

@olivierlambert any news on this ? Thank You.

olivierlambert

Sorry I don't know what's the issue, anyone else can reproduce?

mguimond

@olivierlambert

Do you know what would cause this issue with a specific vm backup ?

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1714518000004",
  "jobId": "3c8de440-0a66-4a95-a201-968d1993bd8b",
  "jobName": "Main - Delta - BRO-NAS-001 - VM1SQLSRV2012_PROD",
  "message": "backup",
  "scheduleId": "41094424-a841-4a75-bb74-edd0c5d72423",
  "start": 1714518000004,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "a23bacfb-543d-0ed7-cf90-902288f59ed6"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "a23bacfb-543d-0ed7-cf90-902288f59ed6",
        "name_label": "VM1SQLSRV2012_PROD"
      },
      "id": "1714518001609",
      "message": "backup VM",
      "start": 1714518001609,
      "status": "failure",
      "tasks": [
        {
          "id": "1714518001617",
          "message": "clean-vm",
          "start": 1714518001617,
          "status": "failure",
          "tasks": [
            {
              "id": "1714518005435",
              "message": "merge",
              "start": 1714518005435,
              "status": "failure",
              "end": 1714518010534,
              "result": {
                "errno": -5,
                "code": "Z_BUF_ERROR",
                "message": "unexpected end of file",
                "name": "Error",
                "stack": "Error: unexpected end of file\n    at BrotliDecoder.zlibOnError [as onerror] (node:zlib:189:17)\n    at BrotliDecoder.callbackTrampoline (node:internal/async_hooks:128:17)"
              }
            }
          ],
          "end": 1714518010534,
          "result": {
            "errno": -5,
            "code": "Z_BUF_ERROR",
            "message": "unexpected end of file",
            "name": "Error",
            "stack": "Error: unexpected end of file\n    at BrotliDecoder.zlibOnError [as onerror] (node:zlib:189:17)\n    at BrotliDecoder.callbackTrampoline (node:internal/async_hooks:128:17)"
          }
        },
        {
          "id": "1714518010549",
          "message": "snapshot",
          "start": 1714518010549,
          "status": "success",
          "end": 1714518014388,
          "result": "bfce73a6-ea5b-d498-f3d8-d139b9fe29d5"
        },
        {
          "data": {
            "id": "b99d8af6-1fc0-4c49-bbe8-d7c718754070",
            "isFull": true,
            "type": "remote"
          },
          "id": "1714518014392",
          "message": "export",
          "start": 1714518014392,
          "status": "success",
          "tasks": [
            {
              "id": "1714518015575",
              "message": "transfer",
              "start": 1714518015575,
              "status": "success",
              "end": 1714530612896,
              "result": {
                "size": 1668042162176
              }
            },
            {
              "id": "1714530613148",
              "message": "clean-vm",
              "start": 1714530613148,
              "status": "success",
              "end": 1714530617953,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1714530618039
        }
      ],
      "end": 1714530618039
    }
  ],
  "end": 1714530618040
}

Danp

@mguimond Could be corruption in the disk. What commit are you on for XO? Is your host fully patched?

olivierlambert

Yeah it sounds that the merge on the backup side can't succeed since the chain seems corrupted.