XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    S3 backup broken

    Scheduled Pinned Locked Moved Xen Orchestra
    31 Posts 5 Posters 9.3k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A Offline
      Andrew Top contributor
      last edited by

      @olivierlambert @julien-f @florent
      Today's (12-Aug) commits broke S3 backups... It timesout after about 10 minutes.

      {
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1660321035114",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1660321035114,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "c45dd52b-fa92-df6f-800a-10853c183c23"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "c45dd52b-fa92-df6f-800a-10853c183c23"
      },
      "id": "1660321035862:0",
      "message": "backup VM",
      "start": 1660321035862,
      "status": "failure",
      "tasks": [
        {
          "id": "1660321036264",
          "message": "clean-vm",
          "start": 1660321036264,
          "status": "success",
          "end": 1660321036533,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1660321036725",
          "message": "snapshot",
          "start": 1660321036725,
          "status": "success",
          "end": 1660321038340,
          "result": "0c702f83-aa82-3c40-25be-b630deaf23ec"
        },
        {
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1660321038341",
          "message": "export",
          "start": 1660321038341,
          "status": "failure",
          "tasks": [
            {
              "id": "1660321038370",
              "message": "transfer",
              "start": 1660321038370,
              "status": "success",
              "end": 1660321060740,
              "result": {
                "size": 1594257920
              }
            },
            {
              "id": "1660321061284",
              "message": "clean-vm",
              "start": 1660321061284,
              "status": "failure",
              "tasks": [
                {
                  "id": "1660321061470",
                  "message": "merge",
                  "start": 1660321061470,
                  "status": "failure",
                  "end": 1660321661643,
                  "result": {
                    "chain": [
                      "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220812T045411Z.alias.vhd",
                      "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220812T045949Z.alias.vhd"
                    ],
                    "message": "operation timed out",
                    "name": "TimeoutError",
                    "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208121202/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208121202/@xen-orchestra/fs/dist/abstract.js:319:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208121202/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"
                  }
                }
              ],
              "end": 1660321661644,
              "result": {
                "chain": [
                  "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220812T045411Z.alias.vhd",
                  "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220812T045949Z.alias.vhd"
                ],
                "message": "operation timed out",
                "name": "TimeoutError",
                "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208121202/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208121202/@xen-orchestra/fs/dist/abstract.js:319:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208121202/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"
              }
            }
          ],
          "end": 1660321661796
        }
      ],
      "end": 1660321661796
    }
  ],
  "end": 1660321661797
}
      julien-fJ 1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Adding @florent in the loop

        1 Reply Last reply Reply Quote 0
        • julien-fJ Offline
          julien-f Vates 🪐 Co-Founder XO Team @Andrew
          last edited by

          @Andrew Any idea which commit is problematic?

          A 1 Reply Last reply Reply Quote 0
          • A Offline
            Andrew Top contributor @julien-f
            last edited by

            @julien-f
            all of them...

            julien-fJ A 2 Replies Last reply Reply Quote 0
            • julien-fJ Offline
              julien-f Vates 🪐 Co-Founder XO Team @Andrew
              last edited by

              @Andrew Do you mean https://github.com/vatesfr/xen-orchestra/commit/d369593979cfcd81d425f873f5a8682991bbba96 introduced the issue?

              0 MathieuRA committed to vatesfr/xen-orchestra 
              fix(xo-web): from ignoreBackup to bypassBackupCheck (#6362)

Introduced by 837b06ef2ba13b561a654d2ebd58b62c4dab2ca7
              A 1 Reply Last reply Reply Quote 0
              • A Offline
                Andrew Top contributor @julien-f
                last edited by

                @julien-f I'll try to narrow it down. But, yes, since d369593979cfcd81d425f873f5a8682991bbba96

                1 Reply Last reply Reply Quote 0
                • A Offline
                  Andrew Top contributor @Andrew
                  last edited by

                  @olivierlambert @julien-f 61691ac46b50659f05159594b8c672b6187a9f9c causes my S3 backup failures.

                  packages/vhd-lib/Vhd/VhdDirectory.js

childBlockPath === undefined
                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    ping @julien-f

                    A 1 Reply Last reply Reply Quote 0
                    • A Offline
                      Andrew Top contributor @olivierlambert
                      last edited by

                      @olivierlambert @julien-f ?

                      julien-fJ 1 Reply Last reply Reply Quote 0
                      • julien-fJ Offline
                        julien-f Vates 🪐 Co-Founder XO Team @Andrew
                        last edited by

                        @Andrew Hey, sorry for the delay, I was in vacation last week and it was not an obvious issue that I could fix quickly.

                        I have started investigating, but have not found the root cause yet.

                        The issue has been triggered by a fix which (should have) make the S3 backups faster by renaming block instead of download/upload for VHD merges.

                        I could revert it but it does not seem to be the way to go, I'll keep you posted as soon as I find the cause.

                        A 1 Reply Last reply Reply Quote 0
                        • A Offline
                          Andrew Top contributor @julien-f
                          last edited by

                          @julien-f No problem, it's August! I hope you had a fun time away from all the computers.

                          julien-fJ 1 Reply Last reply Reply Quote 0
                          • julien-fJ Offline
                            julien-f Vates 🪐 Co-Founder XO Team @Andrew
                            last edited by

                            I finally have access to a correct connection and have been able to run a complete S3 backup, I did not reproduce your issue, maybe (🤞) because I've fixed it.

                            @Andrew Do you still have your issues?

                            A 1 Reply Last reply Reply Quote 0
                            • A Offline
                              Andrew Top contributor @julien-f
                              last edited by Andrew

                              @julien-f Nope... same problem.

                              I updated XO to commit 276d1. I cleared out my S3 storage. I have a Delta Backup job with retention set to 3.

                              First backup runs as Full, completes OK.
                              Second backup runs as delta, completes OK.
                              Third backup runs as delta, completes OK.
                              Fourth backup run as delta, fails to complete, timeout after 10 minutes.
                              Fifth backup runs as delta, fails to complete, takes 20 minutes to time out.

                              Looking at restore, I see only 3 restore points (as I should), the 2nd, 3rd, and 4th (or 3-5). So it looks like it did work but timed out. The meta data is updated, but the actual backup data is left behind.

                              Seems to be a merge (failure) problem.

                              Fourth run log:

                              {
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1661770802189",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1661770802189,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "c45dd52b-fa92-df6f-800a-10853c183c23"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "c45dd52b-fa92-df6f-800a-10853c183c23"
      },
      "id": "1661770803039",
      "message": "backup VM",
      "start": 1661770803039,
      "status": "failure",
      "tasks": [
        {
          "id": "1661770803448",
          "message": "clean-vm",
          "start": 1661770803448,
          "status": "success",
          "end": 1661770803693,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1661770803886",
          "message": "snapshot",
          "start": 1661770803886,
          "status": "success",
          "end": 1661770805369,
          "result": "3c014252-33ec-98da-3485-31eafc9047a5"
        },
        {
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1661770805370",
          "message": "export",
          "start": 1661770805370,
          "status": "failure",
          "tasks": [
            {
              "id": "1661770805400",
              "message": "transfer",
              "start": 1661770805400,
              "status": "success",
              "end": 1661770809191,
              "result": {
                "size": 21009920
              }
            },
            {
              "id": "1661770809728",
              "message": "clean-vm",
              "start": 1661770809728,
              "status": "failure",
              "tasks": [
                {
                  "id": "1661770809929",
                  "message": "merge",
                  "start": 1661770809929,
                  "status": "failure",
                  "end": 1661771410018,
                  "result": {
                    "chain": [
                      "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                      "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                    ],
                    "message": "operation timed out",
                    "name": "TimeoutError",
                    "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208290628/@xen-orchestra/fs/dist/abstract.js:338:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
                  }
                }
              ],
              "end": 1661771410018,
              "result": {
                "chain": [
                  "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                  "/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                ],
                "message": "operation timed out",
                "name": "TimeoutError",
                "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208290628/@xen-orchestra/fs/dist/abstract.js:338:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
              }
            }
          ],
          "end": 1661771410146
        }
      ],
      "end": 1661771410151
    }
  ],
  "end": 1661771410152
}

                              Fifth run log:

                              {
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1661771823634",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1661771823634,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "c45dd52b-fa92-df6f-800a-10853c183c23"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "c45dd52b-fa92-df6f-800a-10853c183c23"
      },
      "id": "1661771824505:0",
      "message": "backup VM",
      "start": 1661771824505,
      "status": "failure",
      "tasks": [
        {
          "id": "1661771824903",
          "message": "clean-vm",
          "start": 1661771824903,
          "status": "failure",
          "tasks": [
            {
              "id": "1661771825143",
              "message": "merge",
              "start": 1661771825143,
              "status": "failure",
              "end": 1661771825223,
              "result": {
                "name": "NoSuchKey",
                "$fault": "client",
                "$metadata": {
                  "httpStatusCode": 400,
                  "attempts": 1,
                  "totalRetryDelay": 0
                },
                "Code": "NoSuchKey",
                "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/646",
                "BucketName": "downtown",
                "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/646",
                "RequestId": "170FCCABAC0773DC",
                "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
                "message": "The specified key does not exist.",
                "chain": [
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                ],
                "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208290628/@xen-orchestra/fs/dist/s3.js:184:7)"
              }
            }
          ],
          "end": 1661771825224,
          "result": {
            "name": "NoSuchKey",
            "$fault": "client",
            "$metadata": {
              "httpStatusCode": 400,
              "attempts": 1,
              "totalRetryDelay": 0
            },
            "Code": "NoSuchKey",
            "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/646",
            "BucketName": "downtown",
            "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/646",
            "RequestId": "170FCCABAC0773DC",
            "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
            "message": "The specified key does not exist.",
            "chain": [
              "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
              "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
            ],
            "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208290628/@xen-orchestra/fs/dist/s3.js:184:7)"
          }
        },
        {
          "id": "1661772425511",
          "message": "snapshot",
          "start": 1661772425511,
          "status": "success",
          "end": 1661772426988,
          "result": "3f0dc9ae-ac68-78fb-3ce5-f72cac4369eb"
        },
        {
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1661772426988:0",
          "message": "export",
          "start": 1661772426988,
          "status": "failure",
          "tasks": [
            {
              "id": "1661772427019",
              "message": "transfer",
              "start": 1661772427019,
              "status": "success",
              "end": 1661772431443,
              "result": {
                "size": 77646848
              }
            },
            {
              "id": "1661772431906",
              "message": "clean-vm",
              "start": 1661772431906,
              "status": "failure",
              "tasks": [
                {
                  "id": "1661772432129",
                  "message": "merge",
                  "start": 1661772432129,
                  "status": "failure",
                  "end": 1661772432196,
                  "result": {
                    "name": "NoSuchKey",
                    "$fault": "client",
                    "$metadata": {
                      "httpStatusCode": 400,
                      "attempts": 1,
                      "totalRetryDelay": 0
                    },
                    "Code": "NoSuchKey",
                    "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/647",
                    "BucketName": "downtown",
                    "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/647",
                    "RequestId": "170FCD38FE8B091B",
                    "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
                    "message": "The specified key does not exist.",
                    "chain": [
                      "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                      "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                    ],
                    "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208290628/@xen-orchestra/fs/dist/s3.js:184:7)"
                  }
                }
              ],
              "end": 1661772432196,
              "result": {
                "name": "NoSuchKey",
                "$fault": "client",
                "$metadata": {
                  "httpStatusCode": 400,
                  "attempts": 1,
                  "totalRetryDelay": 0
                },
                "Code": "NoSuchKey",
                "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/647",
                "BucketName": "downtown",
                "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/647",
                "RequestId": "170FCD38FE8B091B",
                "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
                "message": "The specified key does not exist.",
                "chain": [
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                ],
                "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208290628/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208290628/@xen-orchestra/fs/dist/s3.js:184:7)"
              }
            }
          ],
          "end": 1661773032252
        }
      ],
      "end": 1661773032253
    }
  ],
  "end": 1661773032253
}
                              1 Reply Last reply Reply Quote 0
                              • florentF Offline
                                florent Vates 🪐 XO Team
                                last edited by

                                hi @Andrew ,

                                can you test this branch : fix-s3-merge ? https://github.com/vatesfr/xen-orchestra/pull/6388
                                it seems your error is related to a merge resuming not handling correctly blocks that are already merged

                                Regards

                                fbeauchamp opened this pull request in vatesfr/xen-orchestra

                                closed fix(fs/s3#copy): normalize error: no such key → ENOENT #6388

                                A 1 Reply Last reply Reply Quote 1
                                • A Offline
                                  Andrew Top contributor @florent
                                  last edited by Andrew

                                  @florent Running fix-s3-merge did not fix it.... just took twice a long to fail.

                                  {
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1661901871884",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1661901871884,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "c45dd52b-fa92-df6f-800a-10853c183c23"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "c45dd52b-fa92-df6f-800a-10853c183c23"
      },
      "id": "1661901872792:0",
      "message": "backup VM",
      "start": 1661901872792,
      "status": "failure",
      "tasks": [
        {
          "id": "1661901873179",
          "message": "clean-vm",
          "start": 1661901873179,
          "status": "failure",
          "tasks": [
            {
              "id": "1661901873464",
              "message": "merge",
              "start": 1661901873464,
              "status": "failure",
              "end": 1661901873544,
              "result": {
                "name": "NoSuchKey",
                "$fault": "client",
                "$metadata": {
                  "httpStatusCode": 400,
                  "attempts": 1,
                  "totalRetryDelay": 0
                },
                "Code": "NoSuchKey",
                "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/637",
                "BucketName": "downtown",
                "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/637",
                "RequestId": "171042F2E7ED4258",
                "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
                "message": "The specified key does not exist.",
                "chain": [
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                ],
                "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208301921/@xen-orchestra/fs/dist/s3.js:184:7)"
              }
            }
          ],
          "end": 1661901873544,
          "result": {
            "name": "NoSuchKey",
            "$fault": "client",
            "$metadata": {
              "httpStatusCode": 400,
              "attempts": 1,
              "totalRetryDelay": 0
            },
            "Code": "NoSuchKey",
            "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/637",
            "BucketName": "downtown",
            "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/637",
            "RequestId": "171042F2E7ED4258",
            "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
            "message": "The specified key does not exist.",
            "chain": [
              "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
              "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
            ],
            "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208301921/@xen-orchestra/fs/dist/s3.js:184:7)"
          }
        },
        {
          "id": "1661902473935",
          "message": "snapshot",
          "start": 1661902473935,
          "status": "success",
          "end": 1661902475409,
          "result": "70d35c7b-7f33-9657-124c-9547f8fa39e5"
        },
        {
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1661902475409:0",
          "message": "export",
          "start": 1661902475409,
          "status": "failure",
          "tasks": [
            {
              "id": "1661902475439",
              "message": "transfer",
              "start": 1661902475439,
              "status": "success",
              "end": 1661902520078,
              "result": {
                "size": 2867539968
              }
            },
            {
              "id": "1661902520515",
              "message": "clean-vm",
              "start": 1661902520515,
              "status": "failure",
              "tasks": [
                {
                  "id": "1661902520785",
                  "message": "merge",
                  "start": 1661902520785,
                  "status": "failure",
                  "end": 1661902520860,
                  "result": {
                    "name": "NoSuchKey",
                    "$fault": "client",
                    "$metadata": {
                      "httpStatusCode": 400,
                      "attempts": 1,
                      "totalRetryDelay": 0
                    },
                    "Code": "NoSuchKey",
                    "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/637",
                    "BucketName": "downtown",
                    "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/637",
                    "RequestId": "171043899F0F7665",
                    "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
                    "message": "The specified key does not exist.",
                    "chain": [
                      "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                      "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                    ],
                    "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208301921/@xen-orchestra/fs/dist/s3.js:184:7)"
                  }
                }
              ],
              "end": 1661902520861,
              "result": {
                "name": "NoSuchKey",
                "$fault": "client",
                "$metadata": {
                  "httpStatusCode": 400,
                  "attempts": 1,
                  "totalRetryDelay": 0
                },
                "Code": "NoSuchKey",
                "Key": "backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/c297fff3-6b43-42ba-8b3e-4a288f450a3f.vhd/blocks/2/637",
                "BucketName": "downtown",
                "Resource": "/downtown/backup/xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/data/8c42a2fc-02fd-498b-9d5b-2f1e3a39210c.vhd/blocks/2/637",
                "RequestId": "171043899F0F7665",
                "HostId": "7e8f4ac5-82bb-4d46-b01f-5447fdfb5d3a",
                "message": "The specified key does not exist.",
                "chain": [
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T104605Z.alias.vhd",
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220829T105224Z.alias.vhd"
                ],
                "stack": "NoSuchKey: The specified key does not exist.\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at deserializeAws_restXmlCopyObjectCommandError (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3131:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20\n    at async StandardRetryStrategy.retry (/opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)\n    at async /opt/xo/xo-builds/xen-orchestra-202208301921/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22\n    at async S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202208301921/@xen-orchestra/fs/dist/s3.js:184:7)"
              }
            }
          ],
          "end": 1661903120910
        }
      ],
      "end": 1661903120910
    }
  ],
  "end": 1661903120911
}
                                  florentF 1 Reply Last reply Reply Quote 0
                                  • florentF Offline
                                    florent Vates 🪐 XO Team @Andrew
                                    last edited by

                                    @Andrew can you retry ? my patch was incomplete
                                    thanks

                                    A 2 Replies Last reply Reply Quote 0
                                    • A Offline
                                      Andrew Top contributor @florent
                                      last edited by

                                      @florent Still fails.

                                      {
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1661943144343",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1661943144343,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "c45dd52b-fa92-df6f-800a-10853c183c23"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "c45dd52b-fa92-df6f-800a-10853c183c23"
      },
      "id": "1661943145228:0",
      "message": "backup VM",
      "start": 1661943145228,
      "status": "failure",
      "tasks": [
        {
          "id": "1661943145611",
          "message": "clean-vm",
          "start": 1661943145611,
          "status": "failure",
          "tasks": [
            {
              "id": "1661943145847",
              "message": "merge",
              "start": 1661943145847,
              "status": "failure",
              "end": 1661943745931,
              "result": {
                "chain": [
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T025925Z.alias.vhd",
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T030535Z.alias.vhd"
                ],
                "message": "operation timed out",
                "name": "TimeoutError",
                "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208310649/@xen-orchestra/fs/dist/abstract.js:338:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
              }
            }
          ],
          "end": 1661943745932,
          "result": {
            "chain": [
              "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T025925Z.alias.vhd",
              "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T030535Z.alias.vhd"
            ],
            "message": "operation timed out",
            "name": "TimeoutError",
            "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208310649/@xen-orchestra/fs/dist/abstract.js:338:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1661943746504",
          "message": "snapshot",
          "start": 1661943746504,
          "status": "success",
          "end": 1661943748063,
          "result": "af87938d-8f55-e1e9-cb12-6d0954c1bb89"
        },
        {
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1661943748064",
          "message": "export",
          "start": 1661943748064,
          "status": "failure",
          "tasks": [
            {
              "id": "1661943748108",
              "message": "transfer",
              "start": 1661943748108,
              "status": "success",
              "end": 1661943774148,
              "result": {
                "size": 1600550912
              }
            },
            {
              "id": "1661943774635",
              "message": "clean-vm",
              "start": 1661943774635,
              "status": "failure",
              "tasks": [
                {
                  "id": "1661943774868",
                  "message": "merge",
                  "start": 1661943774868,
                  "status": "failure",
                  "end": 1661944374948,
                  "result": {
                    "chain": [
                      "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T025925Z.alias.vhd",
                      "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T030535Z.alias.vhd"
                    ],
                    "message": "operation timed out",
                    "name": "TimeoutError",
                    "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208310649/@xen-orchestra/fs/dist/abstract.js:338:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
                  }
                }
              ],
              "end": 1661944374950,
              "result": {
                "chain": [
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T025925Z.alias.vhd",
                  "xo-vm-backups/c45dd52b-fa92-df6f-800a-10853c183c23/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/ae8fffde-b2bd-4205-a596-9139ef59193f/20220831T030535Z.alias.vhd"
                ],
                "message": "operation timed out",
                "name": "TimeoutError",
                "stack": "TimeoutError: operation timed out\n    at Promise.timeout (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/promise-toolbox/timeout.js:11:16)\n    at S3Handler.rename (/opt/xo/xo-builds/xen-orchestra-202208310649/@xen-orchestra/fs/dist/abstract.js:338:37)\n    at Queue.next (/opt/xo/xo-builds/xen-orchestra-202208310649/node_modules/limit-concurrency-decorator/dist/index.js:21:22)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
              }
            }
          ],
          "end": 1661944375065
        }
      ],
      "end": 1661944375065
    }
  ],
  "end": 1661944375066
}
                                      O 1 Reply Last reply Reply Quote 0
                                      • O Offline
                                        odeawan @Andrew
                                        last edited by

                                        @Andrew I had a very similar issue over the last couple of weeks producing the same error.
                                        My coworker had been running similar S3 backups with success on the same ISP from a different site and then again at a site with Starlink. The only differences between the sites were the circuit and edge router.
                                        I dove a little deeper and found that I had misconfigured a new VLAN/subnet and had IDS touching xen-orchestra and was flagging the port 80 traffic outbound to S3 as malicious. I could see the firewall actively dropping the outbound packets in the packet filter. This would make the S3 backup stop after about 30 minutes and subsequent backups would fail.
                                        It may not be your exact issue, but this post struck a chord with me.

                                        A 1 Reply Last reply Reply Quote 0
                                        • A Offline
                                          Andrew Top contributor @odeawan
                                          last edited by

                                          @odeawan Thanks for the idea, but I know that's not the problem in this case. This test setup that fails is between a XO server and a S3 server on the same LAN, so no firewall... and it only fails during the delta merge phase. My main (older version) XO server that runs backups to off site S3 storage works fine (also to the same local S3 server).

                                          1 Reply Last reply Reply Quote 0
                                          • A Offline
                                            Andrew Top contributor @florent
                                            last edited by

                                            @florent I did a git update again (to commit aa261) and it works! Maybe I missed and update?,,,,

                                            {
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1661987889299",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1661987889299,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "c45dd52b-fa92-df6f-800a-10853c183c23"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "c45dd52b-fa92-df6f-800a-10853c183c23"
      },
      "id": "1661987890202",
      "message": "backup VM",
      "start": 1661987890202,
      "status": "success",
      "tasks": [
        {
          "id": "1661987890615",
          "message": "clean-vm",
          "start": 1661987890615,
          "status": "success",
          "end": 1661987891797,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1661987891992",
          "message": "snapshot",
          "start": 1661987891992,
          "status": "success",
          "end": 1661987893470,
          "result": "ec858b82-4f64-c5fe-a258-887ba57c7458"
        },
        {
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1661987893471",
          "message": "export",
          "start": 1661987893471,
          "status": "success",
          "tasks": [
            {
              "id": "1661987893513",
              "message": "transfer",
              "start": 1661987893513,
              "status": "success",
              "end": 1661987897713,
              "result": {
                "size": 75549184
              }
            },
            {
              "id": "1661987898186",
              "message": "clean-vm",
              "start": 1661987898186,
              "status": "success",
              "tasks": [
                {
                  "id": "1661987899091",
                  "message": "merge",
                  "start": 1661987899091,
                  "status": "success",
                  "end": 1661987902420
                }
              ],
              "end": 1661987902474,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1661987902480
        }
      ],
      "end": 1661987902480
    }
  ],
  "end": 1661987902480
}
                                            florentF 1 Reply Last reply Reply Quote 0

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • First post
                                              Last post