XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Missing vdi which is a base for a delta

    Scheduled Pinned Locked Moved Unsolved Backup
    18 Posts 3 Posters 965 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • poddingueP
      poddingue Vates 🪐 @blueh2o
      last edited by

      @blueh2o same VMs and same disks every day is useful, I think, since whoever looks at it can then trigger it on demand instead of waiting for a bad night. 🤔

      While you experiment, do the failing VMs have something in common? The same source SR, for example, or a mirror target keeping a different retention than the source.
      I don't know the mirror code well enough to say which of those would matter. 🤷

      B 1 Reply Last reply
      Reply Quote 0
      • B
        blueh2o @poddingue
        last edited by

        @poddingue said:

        While you experiment, do the failing VMs have something in common? The same source SR, for example, or a mirror target keeping a different retention than the source.

        There are two backup jobs that are failing. One from Ark (local) to Synology, and one from Ark to S3. It seems to be the same VMs failing each night, but interestingly, there are some that fail to Synology but succeed to S3 and vice-versa. There are 9 VMs that fail for both. They do not all share the same SR. The backups on Ark are coming from several delta jobs that run daily, weekly, and monthly. Retention are different for all those.

        poddingueP 1 Reply Last reply
        Reply Quote 0
        • poddingueP
          poddingue Vates 🪐 @blueh2o
          last edited by poddingue

          Thanks, that helps. Some VMs fail to the Synology but pass to S3, and some the other way round. 🤔
          So I think the backups on Ark are probably fine (they pass health checks, as you said at the start) and it has more to do with what each target already holds, but I could be wrong. 🤷
          I'd also flag the several delta jobs with different retentions all feeding Ark, I just don't know the mirror code well enough to say whether that matters.

          @Team-XO-Backend, posts 6 and 10 have the log and the pattern if someone gets to this one.

          1 Reply Last reply
          Reply Quote 0
          • B
            blueh2o
            last edited by

            I don't know how, but there do seem to be chains on one remote that have no key. When I delete those, I can get a successful mirror run. I guess I'll just have to throw out all the 'bad' backup chains and start over.

            poddingueP 1 Reply Last reply
            Reply Quote 0
            • poddingueP
              poddingue Vates 🪐 @blueh2o
              last edited by

              Glad you got a clean run out of it, and thanks for coming back to say what worked. If I'm reading the mirror code right, it checks that the parent disk of each delta is already on every target, and stops with this exact error when one isn't there. How a chain ends up without its key in the first place, I can't tell you. Which remote were those chains on, Ark or one of the mirror targets? And before you wipe everything and start over, could you jot down which VMs had them? I think that's what someone would need to chase the cause.

              B 1 Reply Last reply
              Reply Quote 0
              • B
                blueh2o @poddingue
                last edited by

                @poddingue They were all on Ark. VMs:

                7da38127-0ff5-1f76-d8b6-a5159a37d658
                7eb23022-4f0b-f73e-b544-c4d860e54453
                865f4634-76f6-48a3-938e-3adff4fb9e3f
                9a64cc4e-0b46-d01e-bfcf-3fbedbd7e334
                a1f1bee1-4076-b37f-006c-630d5c3a9f39
                a86bb37b-a0ec-3d94-b52d-196da40b317d
                a925c4c2-b3f3-6cc3-3c40-a836e3a24914
                b27a40e4-ea4f-7706-fb4e-98551fa85d0d
                b92636e5-c9f1-47d6-644f-70158dc39f0e
                bbe64766-97f3-de14-43aa-77867b7f1fb0
                d1a39422-78d6-56d1-28c3-65299cdc1a5e
                d77691ef-a507-e120-e2b8-2c1807f0ef99
                edc46393-eee4-6a39-89c0-0b4486df8879
                f8ed66aa-e753-1271-84c6-a73819ff25f7
                fb7fca7d-7355-4a48-f35c-d32c8dfa71ee

                B 1 Reply Last reply
                Reply Quote 0
                • B
                  blueh2o @blueh2o
                  last edited by

                  Well, mirror backup failed last night, missing vdi d08f1acc-a2a9-40c1-a129-e87e151cb268 which is a base for a delta. I checked the VM backups and this is what I see:image.jpeg
                  I don't understand why there is no key in Ark.

                  B 1 Reply Last reply
                  Reply Quote 0
                  • B
                    blueh2o @blueh2o
                    last edited by

                    2026-11-19T07_00_00.018Z - backup NG.txt

                    B 1 Reply Last reply
                    Reply Quote 0
                    • B
                      blueh2o @blueh2o
                      last edited by

                      And the health check passes. How can that be possible?

                      {
                      "id": "0mwnnmk6y-jd7jumsiui",
                      "start": 1795099050682,
                      "status": "success",
                      "updatedAt": 1795103578854,
                      "tasks": [
                      {
                      "id": "0mwnnmmmn-jvm3zflms38",
                      "start": 1795099053839,
                      "status": "success",
                      "end": 1795103578854,
                      "result": {
                      "size": 302621130752,
                      "id": "b656b0d4-b575-c805-5738-6fd50d150951",
                      "liveMounts": []
                      },
                      "message": "transfer"
                      }
                      ],
                      "end": 1795103578854,
                      "result": {
                      "size": 302621130752,
                      "id": "b656b0d4-b575-c805-5738-6fd50d150951",
                      "liveMounts": []
                      },
                      "message": "restore",
                      "data": {
                      "backupId": "dc0a417c-e1cb-4b4b-8fd9-d59e2564999f//xo-vm-backups/fb7fca7d-7355-4a48-f35c-d32c8dfa71ee/20261119T051216Z.json",
                      "jobId": "73d695c7-901b-45bd-94ba-f38682705d35",
                      "srId": "caba922c-ac32-67e8-f83b-a5907d880179",
                      "time": 1795065136373
                      }
                      }

                      poddingueP 1 Reply Last reply
                      Reply Quote 0
                      • poddingueP
                        poddingue Vates 🪐 @blueh2o
                        last edited by

                        @blueh2o I think the two things you saw fit together, at least going by what I see in the mirror code. The 1 differencing tag only says that this one backup's disk is a delta, which is normal for an incremental; the key sits further back in the chain, which I think is why the health check could still rebuild the whole ~280 GiB disk.
                        And as I wrote a few posts up, the mirror reads the parent of each delta on Ark and then asks each target whether it already has it, so last night's error would mean the Synology is missing d08f1acc for that one VM (fb7fca7d), not that Ark is broken.
                        That brings us back to Florent's suggestion from the older thread, deleting that VM's most recent backups on the Synology so they get sent again, though I can't tell you why the Synology lost that base in the first place. At least it's down to one VM out of 56 last night, against 16 in October.

                        1 Reply Last reply
                        Reply Quote 0

                        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                        With your input, this post could be even better 💗

                        Register Login
                        • First post
                          Last post