XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Failed S3 backup success, should recover

    Scheduled Pinned Locked Moved Xen Orchestra
    4 Posts 3 Posters 254 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A Offline
      Andrew Top contributor
      last edited by

      @olivierlambert
      I'm running XO source (commit d3077) with XCP 8.2.1. S3 delta backups have been working great for a long time. Recently Wasabi has been having problems and some backups fail. Manually restarting the failed back normally works and things are fine.

      This one VM backup runs and fails but this delta backup shows in the Restore menu with the correct date.

      The restore health check for this failed backup is Successful (so, it's good, right?).

      This backup does not auto-recover and fix the S3 data storage like most others do. Maybe XO should automatically run a full backup because it fails... XO should do something other than keep failing.

      Failure log:

          {
            "data": {
              "type": "VM",
              "id": "00371171-0a0b-f32f-916e-a79e8f8a98d4",
              "name_label": "billing"
            },
            "id": "1691298603892",
            "message": "backup VM",
            "start": 1691298603892,
            "status": "failure",
            "tasks": [
              {
                "id": "1691298604051",
                "message": "clean-vm",
                "start": 1691298604051,
                "status": "failure",
                "tasks": [
                  {
                    "id": "1691298632353",
                    "message": "merge",
                    "start": 1691298632353,
                    "status": "failure",
                    "end": 1691298637075,
                    "result": {
                      "code": "ENOENT",
                      "path": "/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat",
                      "message": "ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'",
                      "name": "Error",
                      "stack": "Error: ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'\n    at S3Handler._createReadStream (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/s3.js:214:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async S3Handler.__readFile (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/abstract.js:255:18)"
                    }
                  }
                ],
                "end": 1691298637075,
                "result": {
                  "code": "ENOENT",
                  "path": "/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat",
                  "message": "ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'",
                  "name": "Error",
                  "stack": "Error: ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'\n    at S3Handler._createReadStream (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/s3.js:214:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async S3Handler.__readFile (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/abstract.js:255:18)"
                }
              },
              {
                "id": "1691298658567",
                "message": "snapshot",
                "start": 1691298658567,
                "status": "success",
                "end": 1691298661961,
                "result": "9645b1d8-3e12-dfad-5d13-d378d5fb5ca9"
              },
              {
                "data": {
                  "id": "db9ad0a8-bce6-4a2b-b9fd-5c4cecf059c4",
                  "isFull": false,
                  "type": "remote"
                },
                "id": "1691298661962",
                "message": "export",
                "start": 1691298661962,
                "status": "failure",
                "tasks": [
                  {
                    "id": "1691298664803",
                    "message": "transfer",
                    "start": 1691298664803,
                    "status": "success",
                    "end": 1691298692985,
                    "result": {
                      "size": 249655296
                    }
                  },
                  {
                    "id": "1691298696693",
                    "message": "clean-vm",
                    "start": 1691298696693,
                    "status": "failure",
                    "tasks": [
                      {
                        "id": "1691298729796",
                        "message": "merge",
                        "start": 1691298729796,
                        "status": "failure",
                        "end": 1691298734169,
                        "result": {
                          "code": "ENOENT",
                          "path": "/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat",
                          "message": "ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'",
                          "name": "Error",
                          "stack": "Error: ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'\n    at S3Handler._createReadStream (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/s3.js:214:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async S3Handler.__readFile (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/abstract.js:255:18)"
                        }
                      }
                    ],
                    "end": 1691298734170,
                    "result": {
                      "code": "ENOENT",
                      "path": "/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat",
                      "message": "ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'",
                      "name": "Error",
                      "stack": "Error: ENOENT: no such file '/xo-vm-backups/00371171-0a0b-f32f-916e-a79e8f8a98d4/vdis/f9405e51-da48-4eb1-980d-99da34359b3c/b6834642-beb3-49d5-aaf2-bcf09ba79eef/data/9e0089e5-6e24-4861-b229-5fd70a98abff.vhd/bat'\n    at S3Handler._createReadStream (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/s3.js:214:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async S3Handler.__readFile (/opt/xo/xo-builds/xen-orchestra-202307311131/@xen-orchestra/fs/dist/abstract.js:255:18)"
                    }
                  }
                ],
                "end": 1691298734171
              }
            ],
            "end": 1691298734171
          },
      

      The directory is there, but no bat file:

      drwxr-x--- 1 root root   0 Dec 31  1969 blocks
      -rw-r----- 1 root root  10 Jul 22 01:11 chunk-filters.json
      -rw-r----- 1 root root 162 Jul 22 01:11 footer
      -rw-r----- 1 root root 201 Jul 22 01:11 header
      -rw-r----- 1 root root  56 Jul 22 01:11 parentLocatorEntry0
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry1
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry2
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry3
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry4
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry5
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry6
      -rw-r----- 1 root root   1 Jul 22 01:11 parentLocatorEntry7
      
      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hello @Andrew

        Regardless the rest of the post, please try on latest master commit first 🙂 That's the rule if you are on the sources, there's no stable version there.

        If it's the same issue after that, @florent will take a look 🙂

        A 1 Reply Last reply Reply Quote 0
        • A Offline
          Andrew Top contributor @olivierlambert
          last edited by

          @olivierlambert XO source update done (commit 5bef8). Same problem.

          I thought it would be a chance for the team to look a real world failure before I delete it and start the backup over, if they are around.

          B 1 Reply Last reply Reply Quote 0
          • B Offline
            borivoj-tydlitat @Andrew
            last edited by

            We see occasional VM backup failures that match the above pattern (ENOENT on a "path" in a S3 bucket remote destination during a VM delta backup). We are running a backup of a large VM (200GB disk) to S3 bucket in Ceph Object Gateway. Our current XO is this commit https://github.com/vatesfr/xen-orchestra/commit/da51b1649c65d7d78a4eb25cc46c488ce2552800 from 2024-03-04.

            The failure is intermittent.

            Example failure log:

            {
              "data": {
                "mode": "delta",
                "reportWhen": "always"
              },
              "id": "1712008868496",
              "jobId": "49c2634a-f281-46b7-9192-aea8c786f3d4",
              "jobName": "ZABBIX VM to ceph02 Delta Backup",
              "message": "backup",
              "scheduleId": "6b1280f0-21e7-475c-9af3-330726108568",
              "start": 1712008868496,
              "status": "failure",
              "infos": [
                {
                  "data": {
                    "vms": [
                      "c020890b-c1a7-79cf-0b55-aa340ba9226b"
                    ]
                  },
                  "message": "vms"
                }
              ],
              "tasks": [
                {
                  "data": {
                    "type": "VM",
                    "id": "c020890b-c1a7-79cf-0b55-aa340ba9226b",
                    "name_label": "zabbix"
                  },
                  "id": "1712008870266",
                  "message": "backup VM",
                  "start": 1712008870266,
                  "status": "failure",
                  "tasks": [
                    {
                      "id": "1712008870283",
                      "message": "clean-vm",
                      "start": 1712008870283,
                      "status": "success",
                      "end": 1712008872364,
                      "result": {
                        "merge": false
                      }
                    },
                    {
                      "id": "1712008872791",
                      "message": "snapshot",
                      "start": 1712008872791,
                      "status": "success",
                      "end": 1712008873901,
                      "result": "902d65f0-8c9b-2460-dad7-d041ceefebdc"
                    },
                    {
                      "data": {
                        "id": "d8d6a743-8c15-4b9a-baf4-df5d7d379b6e",
                        "isFull": false,
                        "type": "remote"
                      },
                      "id": "1712008873919",
                      "message": "export",
                      "start": 1712008873919,
                      "status": "failure",
                      "tasks": [
                        {
                          "id": "1712008875215",
                          "message": "transfer",
                          "start": 1712008875215,
                          "status": "success",
                          "end": 1712009333076,
                          "result": {
                            "size": 35343973376
                          }
                        },
                        {
                          "id": "1712009335152",
                          "message": "clean-vm",
                          "start": 1712009335152,
                          "status": "failure",
                          "tasks": [
                            {
                              "id": "1712009336153",
                              "message": "merge",
                              "start": 1712009336153,
                              "status": "failure",
                              "end": 1712009585029,
                              "result": {
                                "cause": {
                                  "name": "NoSuchKey",
                                  "$fault": "client",
                                  "$metadata": {
                                    "httpStatusCode": 404,
                                    "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3",
                                    "attempts": 1,
                                    "totalRetryDelay": 0
                                  },
                                  "Code": "NoSuchKey",
                                  "BucketName": "xcp-ng-mama-pool-vm-backup",
                                  "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3",
                                  "HostId": "302aa3c-s3-s3",
                                  "message": "UnknownError"
                                },
                                "code": "ENOENT",
                                "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52",
                                "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'",
                                "name": "Error",
                                "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n    at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
                              }
                            }
                          ],
                          "end": 1712009585029,
                          "result": {
                            "cause": {
                              "name": "NoSuchKey",
                              "$fault": "client",
                              "$metadata": {
                                "httpStatusCode": 404,
                                "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3",
                                "attempts": 1,
                                "totalRetryDelay": 0
                              },
                              "Code": "NoSuchKey",
                              "BucketName": "xcp-ng-mama-pool-vm-backup",
                              "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3",
                              "HostId": "302aa3c-s3-s3",
                              "message": "UnknownError"
                            },
                            "code": "ENOENT",
                            "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52",
                            "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'",
                            "name": "Error",
                            "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n    at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
                          }
                        }
                      ],
                      "end": 1712009585031
                    }
                  ],
                  "end": 1712009585032
                }
              ],
              "end": 1712009585032
            }
            

            I know I should be on the current XO to report a problem. Not asking for a fix now, just trying to document that this defect still exists.

            0 julien-f committed to vatesfr/xen-orchestra
            fix(xo-server/robots.txt): must not require authentication
            
            Introduced by a2c36c083
            
            This file is intended for search engines, which, by definition are not signed in.
            1 Reply Last reply Reply Quote 0
            • First post
              Last post