XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Our future backup code: test it!

    Scheduled Pinned Locked Moved Backup
    128 Posts 10 Posters 10.1k Views 11 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • florentF Offline
      florent Vates 🪐 XO Team @manilx
      last edited by

      @manilx said in Our future backup code: test it!:

      @florent Here is the log of the latest mirror job. All VM's are failing.
      https://paste.vates.tech/?8b00e8353e4139af#4zVKzqXNcpxA7d9PNehV8w4Xjo45UBj9L1RRa3cgGekC

      Guess I'm reverting back.

      Did the mirror work after revert ?

      M 1 Reply Last reply Reply Quote 0
      • M Offline
        manilx @florent
        last edited by

        @florent Yes, just fine.

        florentF 1 Reply Last reply Reply Quote 0
        • florentF Offline
          florent Vates 🪐 XO Team @manilx
          last edited by

          @manilx said in Our future backup code: test it!:

          @florent Yes, just fine.

          ok, so that's an issue with the mirror. I reproduced this night on my lab
          I am working on a fix

          M 1 Reply Last reply Reply Quote 2
          • M Offline
            manilx @florent
            last edited by

            @florent Pls advise here when it's fixed, so that I can retry. I will hold back until then.
            Backups are essential.

            florentF 1 Reply Last reply Reply Quote 1
            • florentF Offline
              florent Vates 🪐 XO Team @manilx
              last edited by florent

              @manilx the PR is in review, it should be merged in a few hours

              thank you for helping us improve the quality

              M 1 Reply Last reply Reply Quote 0
              • M Offline
                manilx @florent
                last edited by

                @florent Will update and try at the end of today then (GMT)

                florentF 1 Reply Last reply Reply Quote 0
                • florentF Offline
                  florent Vates 🪐 XO Team @manilx
                  last edited by

                  @manilx this is merged

                  M 2 Replies Last reply Reply Quote 2
                  • M Offline
                    manilx @florent
                    last edited by

                    @florent Updated.

                    Did a backup job: OK
                    Did the mirror job: OK

                    Great!

                    1 Reply Last reply Reply Quote 2
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      \o/

                      1 Reply Last reply Reply Quote 0
                      • M Offline
                        manilx @florent
                        last edited by

                        This post is deleted!
                        1 Reply Last reply Reply Quote 0
                        • Tristis OrisT Offline
                          Tristis Oris Top contributor
                          last edited by Tristis Oris

                          CR still failed with
                          Error: HANDLE_INVALID(SR, OpaqueRef:cc116d5f-0a91-b487-b43d-4ebd98495fcd)
                          Error: Error in the source generator Cannot read properties of undefined (reading 'write')
                          i found that old CR copies dissapeared from all hosts.

                          florentF 1 Reply Last reply Reply Quote 0
                          • florentF Offline
                            florent Vates 🪐 XO Team @Tristis Oris
                            last edited by

                            @Tristis-Oris I still don't have the solution I am working on it .

                            M 1 Reply Last reply Reply Quote 0
                            • M Offline
                              manilx @florent
                              last edited by

                              @florent Getting errors again:

                              ScreenShot 2025-05-25 at 10.06.44.png

                              {
                                "data": {
                                  "mode": "delta",
                                  "reportWhen": "failure"
                                },
                                "id": "1748134829731",
                                "jobId": "b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6",
                                "jobName": "Mirror Mani",
                                "message": "backup",
                                "scheduleId": "9e8eeb18-65bb-4e12-b1a1-4f7c0490565c",
                                "start": 1748134829731,
                                "status": "failure",
                                "infos": [
                                  {
                                    "data": {
                                      "vms": [
                                        "0c8fd092-ab36-478e-df85-16b1a01e1ca8",
                                        "19242523-e70b-bf39-820d-5e322435012b",
                                        "1a65641e-0976-07a8-30f8-3cbd2e4bc63d",
                                        "1c596fc8-26a8-7d95-d004-1e694392fc72",
                                        "1d7614fc-65f7-4e1c-71f9-da638efd8b4d",
                                        "284b64bc-8684-eb3c-b32e-93e455f5854b",
                                        "294b867a-e241-fe7a-10b0-aef1de6f693d",
                                        "341a6155-908b-66ba-76ee-56a1d5163af1",
                                        "43a4d905-7d13-85b8-bed3-f6b805ff26ac",
                                        "48ca6a7e-7acf-74fd-4ad9-714315563652",
                                        "4d8cad25-b6e5-4d8f-5226-b5908b21a1b6",
                                        "52e64134-62e3-9682-4e3f-296a1198db4d",
                                        "58af91d1-94cf-cbe5-88cf-560e652741bf",
                                        "592e229c-76d7-c0f2-426c-4ef9e80810d0",
                                        "5f49e0a4-9397-3da5-e44a-0ab3632cd9f1",
                                        "6c9f0275-07bd-a124-e187-47f8389ed93c",
                                        "8d573273-49b1-6b84-73a6-42c3346add52",
                                        "8e76ba5c-e190-753b-75d6-e67feb1dd85c",
                                        "9017c283-51d0-8d22-d13b-328cc9fe4ff7",
                                        "903d61e0-c07f-8163-47ae-8e212147a353",
                                        "931aaab6-5f7a-d0ef-26a1-1f54063d0941",
                                        "9e0c8a72-d0fd-6632-6d57-94224149af05",
                                        "9f8088b6-9a5d-0ec5-24d5-118c23100a4b",
                                        "a060d911-bf2b-61c5-56bf-11944af25c4a",
                                        "a2562d66-00c8-3955-ab43-6a5584044b98",
                                        "ac14b70f-9958-7add-f833-64486feba4e7",
                                        "b02697d4-23b5-598f-4def-d6b197bc1085",
                                        "b5d74e0b-388c-019a-6994-e174c9ca7a51",
                                        "bcb4bcfc-c77b-2068-4bec-2826e8ece6be",
                                        "c21e8541-c5ea-840e-31aa-095789600230",
                                        "c49a5398-38db-4fcd-481b-30181a5503cc",
                                        "c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b",
                                        "cc6b00b9-16ec-1990-e44a-503b6e3921c3",
                                        "d7964354-aff6-c7e1-0067-52e8e866a965",
                                        "ef53c584-f6c2-3aa3-b4c0-802fb20683d9",
                                        "f262e69e-f345-bc52-c2dc-60eea7b84c9a",
                                        "fab6b6c0-7ce7-91da-fe60-acbcea4ca5be",
                                        "fe2dd0ea-f063-9aba-30d5-d2de508b091f"
                                      ]
                                    },
                                    "message": "vms"
                                  }
                                ],
                                "tasks": [
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "19242523-e70b-bf39-820d-5e322435012b"
                                    },
                                    "id": "1748134830645",
                                    "message": "backup VM",
                                    "start": 1748134830645,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748134830822",
                                        "message": "clean-vm",
                                        "start": 1748134830822,
                                        "status": "success",
                                        "end": 1748134831309,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748134831397",
                                        "message": "export",
                                        "start": 1748134831397,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748134831413",
                                            "message": "transfer",
                                            "start": 1748134831413,
                                            "status": "success",
                                            "end": 1748134937238,
                                            "result": {
                                              "size": 710934528
                                            }
                                          },
                                          {
                                            "id": "1748134947620",
                                            "message": "clean-vm",
                                            "start": 1748134947620,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/19242523-e70b-bf39-820d-5e322435012b/20250524T220611Z.json",
                                                  "actual": 710934528,
                                                  "expected": 713224192
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748134949566,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748134965928
                                      }
                                    ],
                                    "end": 1748134965928
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "0c8fd092-ab36-478e-df85-16b1a01e1ca8"
                                    },
                                    "id": "1748134830649",
                                    "message": "backup VM",
                                    "start": 1748134830649,
                                    "status": "interrupted",
                                    "tasks": [
                                      {
                                        "id": "1748134830750",
                                        "message": "clean-vm",
                                        "start": 1748134830750,
                                        "status": "success",
                                        "warnings": [
                                          {
                                            "data": {
                                              "path": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/20250522T161036Z.json",
                                              "actual": 71360596992,
                                              "expected": 71360597504
                                            },
                                            "message": "cleanVm: incorrect backup size in metadata"
                                          }
                                        ],
                                        "end": 1748134831421,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748134834838",
                                        "message": "export",
                                        "start": 1748134834838,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748134834899",
                                            "message": "transfer",
                                            "start": 1748134834899,
                                            "status": "success",
                                            "end": 1748135312431,
                                            "result": {
                                              "size": 4857004032
                                            }
                                          },
                                          {
                                            "id": "1748135312559",
                                            "message": "clean-vm",
                                            "start": 1748135312559,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "parent": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45",
                                                  "child": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45/20250524T220111Z.vhd"
                                                },
                                                "message": "parent VHD is missing"
                                              },
                                              {
                                                "data": {
                                                  "backup": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/20250524T220111Z.json",
                                                  "missingVhds": [
                                                    "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45/20250524T220111Z.vhd"
                                                  ]
                                                },
                                                "message": "some VHDs linked to the backup are missing"
                                              }
                                            ],
                                            "end": 1748135312819,
                                            "result": {
                                              "merge": false
                                            }
                                          }
                                        ],
                                        "end": 1748135312823
                                      }
                                    ],
                                    "warnings": [
                                      {
                                        "data": {
                                          "attempt": 1,
                                          "error": "Expected values to be strictly equal:\n\n0 !== 512\n"
                                        },
                                        "message": "Retry the VM mirror backup due to an error"
                                      }
                                    ]
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "1c596fc8-26a8-7d95-d004-1e694392fc72"
                                    },
                                    "id": "1748134966251",
                                    "message": "backup VM",
                                    "start": 1748134966251,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748134968074",
                                        "message": "clean-vm",
                                        "start": 1748134968074,
                                        "status": "success",
                                        "end": 1748134968813,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748134971622",
                                        "message": "export",
                                        "start": 1748134971622,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748134971997",
                                            "message": "transfer",
                                            "start": 1748134971997,
                                            "status": "success",
                                            "end": 1748135312426,
                                            "result": {
                                              "size": 3261071360
                                            }
                                          },
                                          {
                                            "id": "1748135314472",
                                            "message": "clean-vm",
                                            "start": 1748135314472,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/1c596fc8-26a8-7d95-d004-1e694392fc72/20250524T222347Z.json",
                                                  "actual": 3261071360,
                                                  "expected": 3264098816
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748135314652,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748135314707
                                      }
                                    ],
                                    "end": 1748135314707
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "294b867a-e241-fe7a-10b0-aef1de6f693d"
                                    },
                                    "id": "1748135312961",
                                    "message": "backup VM",
                                    "start": 1748135312961,
                                    "status": "interrupted",
                                    "tasks": [
                                      {
                                        "id": "1748135313037",
                                        "message": "clean-vm",
                                        "start": 1748135313037,
                                        "status": "success",
                                        "warnings": [
                                          {
                                            "data": {
                                              "path": "/xo-vm-backups/294b867a-e241-fe7a-10b0-aef1de6f693d/20250522T161022Z.json",
                                              "actual": 11216235520,
                                              "expected": 11216236032
                                            },
                                            "message": "cleanVm: incorrect backup size in metadata"
                                          }
                                        ],
                                        "end": 1748135313727,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135313903",
                                        "message": "export",
                                        "start": 1748135313903,
                                        "status": "failure",
                                        "tasks": [
                                          {
                                            "id": "1748135313921",
                                            "message": "transfer",
                                            "start": 1748135313921,
                                            "status": "failure",
                                            "end": 1748135318829,
                                            "result": {
                                              "errno": -9,
                                              "code": "EBADF",
                                              "syscall": "read",
                                              "message": "EBADF: bad file descriptor, read",
                                              "name": "Error",
                                              "stack": "Error: EBADF: bad file descriptor, read\nFrom:\n    at NfsHandler.addSyncStackTrace (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:21:26)\n    at NfsHandler._read (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:160:43)\n    at NfsHandler.read (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:259:17)\n    at NfsHandler.read (/opt/xen-orchestra/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at VhdFile._read (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:126:55)\n    at VhdFile.readBlock (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:212:17)\n    at RemoteVhd.readBlock (file:///opt/xen-orchestra/@xen-orchestra/backups/disks/RemoteVhd.mjs:144:38)\n    at RemoteVhd.buildDiskBlockGenerator (file:///opt/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:67:28)"
                                            }
                                          },
                                          {
                                            "id": "1748135318836",
                                            "message": "clean-vm",
                                            "start": 1748135318836,
                                            "status": "success",
                                            "end": 1748135318927,
                                            "result": {
                                              "merge": false
                                            }
                                          }
                                        ],
                                        "end": 1748135318948
                                      }
                                    ],
                                    "warnings": [
                                      {
                                        "data": {
                                          "attempt": 1,
                                          "error": "ENOENT: no such file or directory, open '/run/xo-server/mounts/337848e1-b52b-4e08-9bf0-ffce54de4276/xo-vm-backups/294b867a-e241-fe7a-10b0-aef1de6f693d/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/c87b9871-9ce2-4b39-9fec-9dea2fac3220/20250524T220036Z.vhd'"
                                        },
                                        "message": "Retry the VM mirror backup due to an error"
                                      }
                                    ]
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "43a4d905-7d13-85b8-bed3-f6b805ff26ac"
                                    },
                                    "id": "1748135318587",
                                    "message": "backup VM",
                                    "start": 1748135318587,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135318860",
                                        "message": "clean-vm",
                                        "start": 1748135318860,
                                        "status": "success",
                                        "end": 1748135319739,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135319979",
                                        "message": "export",
                                        "start": 1748135319979,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748135319993",
                                            "message": "transfer",
                                            "start": 1748135319993,
                                            "status": "success",
                                            "end": 1748135458653,
                                            "result": {
                                              "size": 870318080
                                            }
                                          },
                                          {
                                            "id": "1748135473880",
                                            "message": "clean-vm",
                                            "start": 1748135473880,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/43a4d905-7d13-85b8-bed3-f6b805ff26ac/20250524T220144Z.json",
                                                  "actual": 870318080,
                                                  "expected": 872691712
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748135483581,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748135487189
                                      }
                                    ],
                                    "end": 1748135487190
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "58af91d1-94cf-cbe5-88cf-560e652741bf"
                                    },
                                    "id": "1748135319145",
                                    "message": "backup VM",
                                    "start": 1748135319145,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135319356",
                                        "message": "clean-vm",
                                        "start": 1748135319356,
                                        "status": "success",
                                        "end": 1748135320011,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135331775",
                                        "message": "export",
                                        "start": 1748135331775,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748135331781",
                                            "message": "transfer",
                                            "start": 1748135331781,
                                            "status": "success",
                                            "end": 1748135561792,
                                            "result": {
                                              "size": 2801795072
                                            }
                                          },
                                          {
                                            "id": "1748135562458",
                                            "message": "clean-vm",
                                            "start": 1748135562458,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/58af91d1-94cf-cbe5-88cf-560e652741bf/20250524T221027Z.json",
                                                  "actual": 2801795072,
                                                  "expected": 2804644352
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748135562570,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748135562664
                                      }
                                    ],
                                    "end": 1748135562665
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "6c9f0275-07bd-a124-e187-47f8389ed93c"
                                    },
                                    "id": "1748135487646",
                                    "message": "backup VM",
                                    "start": 1748135487646,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135488697",
                                        "message": "clean-vm",
                                        "start": 1748135488697,
                                        "status": "success",
                                        "end": 1748135499685,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "id": "1748135500529:0",
                                        "message": "clean-vm",
                                        "start": 1748135500529,
                                        "status": "success",
                                        "end": 1748135506654,
                                        "result": {
                                          "merge": false
                                        }
                                      }
                                    ],
                                    "infos": [
                                      {
                                        "message": "No new data to upload for this VM"
                                      }
                                    ],
                                    "end": 1748135507365
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "931aaab6-5f7a-d0ef-26a1-1f54063d0941"
                                    },
                                    "id": "1748135507681",
                                    "message": "backup VM",
                                    "start": 1748135507681,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135509790",
                                        "message": "clean-vm",
                                        "start": 1748135509790,
                                        "status": "success",
                                        "end": 1748135517538,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "id": "1748135518256",
                                        "message": "clean-vm",
                                        "start": 1748135518256,
                                        "status": "success",
                                        "end": 1748135523041,
                                        "result": {
                                          "merge": false
                                        }
                                      }
                                    ],
                                    "infos": [
                                      {
                                        "message": "No new data to upload for this VM"
                                      }
                                    ],
                                    "end": 1748135523539
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "a060d911-bf2b-61c5-56bf-11944af25c4a"
                                    },
                                    "id": "1748135525305",
                                    "message": "backup VM",
                                    "start": 1748135525305,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135527194",
                                        "message": "clean-vm",
                                        "start": 1748135527194,
                                        "status": "success",
                                        "end": 1748135534840,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "id": "1748135535382",
                                        "message": "clean-vm",
                                        "start": 1748135535382,
                                        "status": "success",
                                        "end": 1748135538329,
                                        "result": {
                                          "merge": false
                                        }
                                      }
                                    ],
                                    "infos": [
                                      {
                                        "message": "No new data to upload for this VM"
                                      }
                                    ],
                                    "end": 1748135538791
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "a2562d66-00c8-3955-ab43-6a5584044b98"
                                    },
                                    "id": "1748135539187",
                                    "message": "backup VM",
                                    "start": 1748135539187,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135561164",
                                        "message": "clean-vm",
                                        "start": 1748135561164,
                                        "status": "success",
                                        "end": 1748135562180,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135562446",
                                        "message": "export",
                                        "start": 1748135562446,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748135562453",
                                            "message": "transfer",
                                            "start": 1748135562453,
                                            "status": "success",
                                            "end": 1748135562527,
                                            "result": {
                                              "size": 0
                                            }
                                          },
                                          {
                                            "id": "1748135562801",
                                            "message": "clean-vm",
                                            "start": 1748135562801,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/a2562d66-00c8-3955-ab43-6a5584044b98/20250524T220635Z.json",
                                                  "actual": 0,
                                                  "expected": 134144
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748135563247,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748135563304
                                      }
                                    ],
                                    "end": 1748135563304
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "b02697d4-23b5-598f-4def-d6b197bc1085"
                                    },
                                    "id": "1748135562737",
                                    "message": "backup VM",
                                    "start": 1748135562737,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135562853",
                                        "message": "clean-vm",
                                        "start": 1748135562853,
                                        "status": "success",
                                        "warnings": [
                                          {
                                            "data": {
                                              "path": "/xo-vm-backups/b02697d4-23b5-598f-4def-d6b197bc1085/20250522T161026Z.json",
                                              "actual": 11740651520,
                                              "expected": 11740652032
                                            },
                                            "message": "cleanVm: incorrect backup size in metadata"
                                          }
                                        ],
                                        "end": 1748135563349,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135563511",
                                        "message": "export",
                                        "start": 1748135563511,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748135563531",
                                            "message": "transfer",
                                            "start": 1748135563531,
                                            "status": "success",
                                            "end": 1748135898500,
                                            "result": {
                                              "size": 2713714688
                                            }
                                          },
                                          {
                                            "id": "1748135910743",
                                            "message": "clean-vm",
                                            "start": 1748135910743,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/b02697d4-23b5-598f-4def-d6b197bc1085/20250524T220039Z.json",
                                                  "actual": 2713714688,
                                                  "expected": 2716501504
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748135913811,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748135915372
                                      }
                                    ],
                                    "end": 1748135915372
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "b5d74e0b-388c-019a-6994-e174c9ca7a51"
                                    },
                                    "id": "1748135563341",
                                    "message": "backup VM",
                                    "start": 1748135563341,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135563363",
                                        "message": "clean-vm",
                                        "start": 1748135563363,
                                        "status": "success",
                                        "end": 1748135563857,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135565987",
                                        "message": "export",
                                        "start": 1748135565987,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748135565996",
                                            "message": "transfer",
                                            "start": 1748135565996,
                                            "status": "success",
                                            "end": 1748136473233,
                                            "result": {
                                              "size": 13337886720
                                            }
                                          },
                                          {
                                            "id": "1748136476298",
                                            "message": "clean-vm",
                                            "start": 1748136476298,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/b5d74e0b-388c-019a-6994-e174c9ca7a51/20250524T220255Z.json",
                                                  "actual": 13337886720,
                                                  "expected": 13343341568
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748136477094,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748136477313
                                      }
                                    ],
                                    "end": 1748136477313
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "c49a5398-38db-4fcd-481b-30181a5503cc"
                                    },
                                    "id": "1748135916054",
                                    "message": "backup VM",
                                    "start": 1748135916054,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135920380",
                                        "message": "clean-vm",
                                        "start": 1748135920380,
                                        "status": "success",
                                        "end": 1748135930666,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "id": "1748135930919",
                                        "message": "clean-vm",
                                        "start": 1748135930919,
                                        "status": "success",
                                        "end": 1748135935805,
                                        "result": {
                                          "merge": false
                                        }
                                      }
                                    ],
                                    "infos": [
                                      {
                                        "message": "No new data to upload for this VM"
                                      }
                                    ],
                                    "end": 1748135936610
                                  },
                                  {
                                    "data": {
                                      "type": "VM",
                                      "id": "c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b"
                                    },
                                    "id": "1748135937843",
                                    "message": "backup VM",
                                    "start": 1748135937843,
                                    "status": "success",
                                    "tasks": [
                                      {
                                        "id": "1748135940204",
                                        "message": "clean-vm",
                                        "start": 1748135940204,
                                        "status": "success",
                                        "end": 1748135948613,
                                        "result": {
                                          "merge": false
                                        }
                                      },
                                      {
                                        "data": {
                                          "id": "337848e1-b52b-4e08-9bf0-ffce54de4276",
                                          "type": "remote"
                                        },
                                        "id": "1748135951349",
                                        "message": "export",
                                        "start": 1748135951349,
                                        "status": "success",
                                        "tasks": [
                                          {
                                            "id": "1748135951810",
                                            "message": "transfer",
                                            "start": 1748135951810,
                                            "status": "success",
                                            "end": 1748136066962,
                                            "result": {
                                              "size": 960495616
                                            }
                                          },
                                          {
                                            "id": "1748136083459",
                                            "message": "clean-vm",
                                            "start": 1748136083459,
                                            "status": "success",
                                            "warnings": [
                                              {
                                                "data": {
                                                  "path": "/xo-vm-backups/c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b/20250524T220054Z.json",
                                                  "actual": 960495616,
                                                  "expected": 962891264
                                                },
                                                "message": "cleanVm: incorrect backup size in metadata"
                                              }
                                            ],
                                            "end": 1748136093352,
                                            "result": {
                                              "merge": true
                                            }
                                          }
                                        ],
                                        "end": 1748136096139
                                      }
                                    ],
                                    "end": 1748136096139
                                  }
                                ],
                                "end": 1748136477317,
                                "result": {
                                  "errno": -9,
                                  "code": "EBADF",
                                  "syscall": "scandir",
                                  "path": "/run/xo-server/mounts/5dba60dc-c4fc-4d49-9a1e-453522c0d8f7/xo-vm-backups/341a6155-908b-66ba-76ee-56a1d5163af1"
                                }
                              }
                              
                              1 Reply Last reply Reply Quote 0
                              • florentF Offline
                                florent Vates 🪐 XO Team
                                last edited by

                                I am checking it
                                Can you confirm that this a delta backup job and you are not using block.

                                M 1 Reply Last reply Reply Quote 0
                                • M Offline
                                  manilx @florent
                                  last edited by manilx

                                  @florent The original backup is delta. Using NBD+CBT to a NFS storage.

                                  Also get these frequently:

                                  ScreenShot 2025-05-26 at 09.18.45.png

                                  But nothing is running in parallel.

                                  1 Reply Last reply Reply Quote 0
                                  • florentF Offline
                                    florent Vates 🪐 XO Team
                                    last edited by

                                    this error is known : this when a mirror backup start before the end of the asynchronoux cleaning of previous backup is not done

                                    you can check the "merge backup synchronously" on the advanced setting of the first backup

                                    M 2 Replies Last reply Reply Quote 1
                                    • M Offline
                                      manilx @florent
                                      last edited by manilx

                                      @florent All my backups have this set! And this only happens with the new backup code, never with the old.

                                      1 Reply Last reply Reply Quote 0
                                      • M Offline
                                        manilx @florent
                                        last edited by

                                        @florent Giving up on mirror backups. Switching to direct backups to remore storage.
                                        At least until this works again as expected.

                                        1 Reply Last reply Reply Quote 0
                                        • florentF Offline
                                          florent Vates 🪐 XO Team
                                          last edited by

                                          At least I reproduce it locally , when going from a non block remote to an encrypted remote

                                          Is your target remote encrypted ?

                                          M 1 Reply Last reply Reply Quote 0
                                          • M Offline
                                            manilx @florent
                                            last edited by

                                            @florent Simple NFS, no encryption.

                                            florentF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post