@Bastien-Nollet here is the log..
2025-07-09T01_00_00.011Z - backup NG.json.txt

From today morning...

Yesterday all was ok.

@lsouai-vates
Sure, I understand issues might be related to changes of backup processing under the hood.
I hope my report going to help with identification of bugs.
Does the EEXIST error are also related with this?

Hi,

I see that changes of backup engine causing a lot of new errors.

1. There are many "Error: EEXIST: file already exists" which never happened in the past. Restart of the same backup usually just works.
2. Due to 1 I've added option "retry" to each backup and now even if error occurs second attempt is successful but overall status of backup tasks is set to failed.

This is how it looks like.

This backup job is an old one which running at my environment for months if not years.
Backups are stored on NAS via NFS.

Other VMs processed in this job are successfully processed.

Backup job logs attached.
2025-07-01T01_00_00.011Z - backup NG.json.txt

@DustinB

Here is the backup job configuration:

@DustinB

Look I have xs-node1 and xs-node2 as you can see DC1 running on node1 and have replica to separate, replica dedicated, SR at node2... DC2 running on node2 and have replica also to this replica dedicated SR.

Both xs-node1 and xs-node2 have only local storage NVMe for running VMs and xs-node2 have also ZFS pool for replicas.

When I'm trying to migrate DC1 to xs-node2 on NVMe SR I'm getting this DUPLICATE_MAC_SEED error.

Only one DC1 VM running across both xs-nodes...

@DustinB Main reason is related with OS license costs... I agree that it would be easier to have single pool.

I have 2 XCP-NG hosts one is used to run VMs and other is a target for VMs replica.

When I try migrate running VM to other host which is a target of backup replica, so there are VM objects created during replication, I get error DUPLICATE_MAC_SEED.

I need to move all running VMs to this second host to perform maintenance of main one.
It was possible before.

Xen Orchestra, commit 6d993
Master, commit ec782

@icompit
Xen Orchestra, commit a5967
Master, commit a5967

@icompit Full backups of the same VMs are processed without a problem.

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1728349200005",
  "jobId": "af0c1c01-8101-4dc6-806f-a4d3cf381cf7",
  "jobName": "ISO",
  "message": "backup",
  "scheduleId": "0e073624-a7eb-47ca-aca5-5f1cd0fc996b",
  "start": 1728349200005,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "81dbf29f-adfa-5bc4-9bc1-021f8eb46b9e",
          "fc1c7067-69f3-8949-ab68-780024a49a75",
          "07c32e8d-ea85-ffaa-3509-2dabff58e4af",
          "66949c31-5544-88fe-5e49-f0e0c7946347"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "81dbf29f-adfa-5bc4-9bc1-021f8eb46b9e",
        "name_label": "MX"
      },
      "id": "1728349200726",
      "message": "backup VM",
      "start": 1728349200726,
      "status": "success",
      "tasks": [
        {
          "id": "1728349200738",
          "message": "clean-vm",
          "start": 1728349200738,
          "status": "success",
          "end": 1728349200773,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1728349200944",
          "message": "snapshot",
          "start": 1728349200944,
          "status": "success",
          "end": 1728349202402,
          "result": "c9285da2-1b13-e8a7-7652-316fbfde76fd"
        },
        {
          "data": {
            "id": "22d6a348-ae2b-4783-a77d-a456e508ba64",
            "isFull": true,
            "type": "remote"
          },
          "id": "1728349202402:0",
          "message": "export",
          "start": 1728349202402,
          "status": "success",
          "tasks": [
            {
              "id": "1728349202859",
              "message": "transfer",
              "start": 1728349202859,
              "status": "success",
              "end": 1728355287481,
              "result": {
                "size": 358491499520
              }
            },
            {
              "id": "1728355310406",
              "message": "health check",
              "start": 1728355310406,
              "status": "success",
              "tasks": [
                {
                  "id": "1728355311238",
                  "message": "transfer",
                  "start": 1728355311238,
                  "status": "success",
                  "end": 1728359446575,
                  "result": {
                    "size": 358491498496,
                    "id": "17402cd8-bd52-98e1-7951-3d47a03dcbf4"
                  }
                },
                {
                  "id": "1728359446575:0",
                  "message": "vmstart",
                  "start": 1728359446575,
                  "status": "success",
                  "end": 1728359477580
                }
              ],
              "end": 1728359492584
            },
            {
              "id": "1728359492598",
              "message": "clean-vm",
              "start": 1728359492598,
              "status": "success",
              "end": 1728359492625,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1728359492627
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:dea183a0-abf2-422b-b7fd-4d0ed1d2ac12"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:e1c325ac-e8c9-4a75-83d0-a0ee50289665"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1728359492628
    },
    {
      "data": {
        "type": "VM",
        "id": "fc1c7067-69f3-8949-ab68-780024a49a75",
        "name_label": "ISH"
      },
      "id": "1728359492631",
      "message": "backup VM",
      "start": 1728359492631,
      "status": "success",
      "tasks": [
        {
          "id": "1728359492636",
          "message": "clean-vm",
          "start": 1728359492636,
          "status": "success",
          "end": 1728359492649,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1728359492792",
          "message": "snapshot",
          "start": 1728359492792,
          "status": "success",
          "end": 1728359493650,
          "result": "8c4a61e7-9f4c-a58b-f1d7-e9edb86e73bb"
        },
        {
          "data": {
            "id": "22d6a348-ae2b-4783-a77d-a456e508ba64",
            "isFull": true,
            "type": "remote"
          },
          "id": "1728359493650:0",
          "message": "export",
          "start": 1728359493650,
          "status": "success",
          "tasks": [
            {
              "id": "1728359494085",
              "message": "transfer",
              "start": 1728359494085,
              "status": "success",
              "end": 1728360044219,
              "result": {
                "size": 49962294784
              }
            },
            {
              "id": "1728360047118",
              "message": "health check",
              "start": 1728360047118,
              "status": "success",
              "tasks": [
                {
                  "id": "1728360047256",
                  "message": "transfer",
                  "start": 1728360047256,
                  "status": "success",
                  "end": 1728360544642,
                  "result": {
                    "size": 49962294272,
                    "id": "3d7e38b1-2366-e657-884e-67f33bee9b56"
                  }
                },
                {
                  "id": "1728360544642:0",
                  "message": "vmstart",
                  "start": 1728360544642,
                  "status": "success",
                  "end": 1728360567171
                }
              ],
              "end": 1728360568598
            },
            {
              "id": "1728360568628",
              "message": "clean-vm",
              "start": 1728360568628,
              "status": "success",
              "end": 1728360568663,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1728360568667
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:18e03c56-b537-490c-8c1d-0f87fef61940"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1728360568667
    },
    {
      "data": {
        "type": "VM",
        "id": "07c32e8d-ea85-ffaa-3509-2dabff58e4af",
        "name_label": "VPS1"
      },
      "id": "1728360568671",
      "message": "backup VM",
      "start": 1728360568671,
      "status": "success",
      "tasks": [
        {
          "id": "1728360568677",
          "message": "clean-vm",
          "start": 1728360568677,
          "status": "success",
          "end": 1728360568702,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1728360568803",
          "message": "snapshot",
          "start": 1728360568803,
          "status": "success",
          "end": 1728360569642,
          "result": "eac56d7c-fa78-f6ae-41c3-d316fbc597a0"
        },
        {
          "data": {
            "id": "22d6a348-ae2b-4783-a77d-a456e508ba64",
            "isFull": true,
            "type": "remote"
          },
          "id": "1728360569643",
          "message": "export",
          "start": 1728360569643,
          "status": "success",
          "tasks": [
            {
              "id": "1728360570055",
              "message": "transfer",
              "start": 1728360570055,
              "status": "success",
              "end": 1728360787031,
              "result": {
                "size": 15826942976
              }
            },
            {
              "id": "1728360788525",
              "message": "health check",
              "start": 1728360788525,
              "status": "success",
              "tasks": [
                {
                  "id": "1728360788592",
                  "message": "transfer",
                  "start": 1728360788592,
                  "status": "success",
                  "end": 1728360987674,
                  "result": {
                    "size": 15826942464,
                    "id": "52962bc5-2ed4-c397-9131-8894f45523bd"
                  }
                },
                {
                  "id": "1728360987675",
                  "message": "vmstart",
                  "start": 1728360987675,
                  "status": "success",
                  "end": 1728361007502
                }
              ],
              "end": 1728361008575
            },
            {
              "id": "1728361008588",
              "message": "clean-vm",
              "start": 1728361008588,
              "status": "success",
              "end": 1728361008622,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1728361008636
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:00c688b0-555d-42bb-93e9-db8b09fea339"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1728361008636
    },
    {
      "data": {
        "type": "VM",
        "id": "66949c31-5544-88fe-5e49-f0e0c7946347",
        "name_label": "DNS"
      },
      "id": "1728361008638",
      "message": "backup VM",
      "start": 1728361008638,
      "status": "success",
      "tasks": [
        {
          "id": "1728361008643",
          "message": "clean-vm",
          "start": 1728361008643,
          "status": "success",
          "end": 1728361008669,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1728361008780",
          "message": "snapshot",
          "start": 1728361008780,
          "status": "success",
          "end": 1728361009626,
          "result": "20408aa8-5979-23a5-e8db-c32eb5ec52a8"
        },
        {
          "data": {
            "id": "22d6a348-ae2b-4783-a77d-a456e508ba64",
            "isFull": true,
            "type": "remote"
          },
          "id": "1728361009626:0",
          "message": "export",
          "start": 1728361009626,
          "status": "success",
          "tasks": [
            {
              "id": "1728361010081",
              "message": "transfer",
              "start": 1728361010081,
              "status": "success",
              "end": 1728361179509,
              "result": {
                "size": 14742417920
              }
            },
            {
              "id": "1728361181070:0",
              "message": "health check",
              "start": 1728361181070,
              "status": "success",
              "tasks": [
                {
                  "id": "1728361181124",
                  "message": "transfer",
                  "start": 1728361181124,
                  "status": "success",
                  "end": 1728361321066,
                  "result": {
                    "size": 14742417408,
                    "id": "cd3ce194-28ca-0c3f-8a78-6873ebac0da2"
                  }
                },
                {
                  "id": "1728361321067",
                  "message": "vmstart",
                  "start": 1728361321067,
                  "status": "success",
                  "end": 1728361368268
                }
              ],
              "end": 1728361369283
            },
            {
              "id": "1728361369296",
              "message": "clean-vm",
              "start": 1728361369296,
              "status": "success",
              "end": 1728361369327,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1728361369340
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:b6f2b416-83b1-4f50-84f1-662f03aa626d"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1728361369340
    }
  ],
  "end": 1728361369340
}