Continuous replication problem with multiple SR

Gheppy

Hello,
I have problems with continuous replication, meaning that if I choose to do it on a single SR everything is ok but if I add two or more SRs it crashes and gives the error message below.
The problem is that it does nothing, it stays stuck and it is always the first SR in the list.
Info:

XCP-ng 8.3 up to date,
XOCE commit 6ecab

Here it is with only one SR and everything is ok:

configuration:
transfer:
message

Here it is with two SRs

configuration:
transfer:
message:
log

{
  "data": {
    "mode": "delta",
    "reportWhen": "always"
  },
  "id": "1748599936065",
  "jobId": "109e74e9-b59f-483b-860f-8f36f5223789",
  "jobName": "********-vrs7",
  "message": "backup",
  "scheduleId": "40f57bd8-2557-4cf5-8322-705ec1d811d2",
  "start": 1748599936065,
  "status": "pending",
  "infos": [
    {
      "data": {
        "vms": [
          "629bdfeb-7700-561c-74ac-e151068721c2"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "629bdfeb-7700-561c-74ac-e151068721c2",
        "name_label": "********-vrs7"
      },
      "id": "1748599941103",
      "message": "backup VM",
      "start": 1748599941103,
      "status": "pending",
      "tasks": [
        {
          "id": "1748599941671",
          "message": "snapshot",
          "start": 1748599941671,
          "status": "success",
          "end": 1748599945608,
          "result": "5c030f40-0b34-d1b4-10aa-f849548aa0b7"
        },
        {
          "data": {
            "id": "1afcdfda-6ede-3cb5-ecbf-29dc09ea605c",
            "isFull": true,
            "name_label": "********-RAID",
            "type": "SR"
          },
          "id": "1748599945609",
          "message": "export",
          "start": 1748599945609,
          "status": "pending",
          "tasks": [
            {
              "id": "1748599948875",
              "message": "transfer",
              "start": 1748599948875,
              "status": "failure",
              "end": 1748599949159,
              "result": {
                "code": "HANDLE_INVALID",
                "params": [
                  "SR",
                  "OpaqueRef:e6bbcba7-6a86-4d00-8391-3c1722b3552f"
                ],
                "call": {
                  "duration": 6,
                  "method": "VDI.create",
                  "params": [
                    "* session id *",
                    {
                      "name_description": "********-sdb-256gb",
                      "name_label": "********-sdb-256gb",
                      "other_config": {
                        "xo:backup:vm": "629bdfeb-7700-561c-74ac-e151068721c2",
                        "xo:copy_of": "59ee458d-99c8-4a45-9c91-263c9729208b"
                      },
                      "read_only": false,
                      "sharable": false,
                      "SR": "OpaqueRef:e6bbcba7-6a86-4d00-8391-3c1722b3552f",
                      "tags": [],
                      "type": "user",
                      "virtual_size": 274877906944,
                      "xenstore_data": {}
                    }
                  ]
                },
                "message": "HANDLE_INVALID(SR, OpaqueRef:e6bbcba7-6a86-4d00-8391-3c1722b3552f)",
                "name": "XapiError",
                "stack": "XapiError: HANDLE_INVALID(SR, OpaqueRef:e6bbcba7-6a86-4d00-8391-3c1722b3552f)\n    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)"
              }
            }
          ]
        },
        {
          "data": {
            "id": "a5d2b22e-e4be-c384-9187-879aa41dd70f",
            "isFull": true,
            "name_label": "********-vrs6-RAID",
            "type": "SR"
          },
          "id": "1748599945617",
          "message": "export",
          "start": 1748599945617,
          "status": "pending",
          "tasks": [
            {
              "id": "1748599948887",
              "message": "transfer",
              "start": 1748599948887,
              "status": "pending"
            }
          ]
        }
      ]
    }
  ]
}

olivierlambert

@Gheppy Hi,

Worth an investigation for @florent (adding also @lsouai-vates in the loop)

florent

@olivierlambert @Gheppy that is a nice catch and it gives us a interesting clue

I am currently working on it

florent

I have a branch that fix this case, at least on our lab fix_multiple_replication , PR is here : https://github.com/vatesfr/xen-orchestra/pull/8668

it should be in master tomorrow , and hopefully in latest by the end of the week

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(backups): properly clone data on incremental backup #8668

florent

said in Continuous replication problem with multiple SR:

I have a branch that fix this case, at least on our lab fix_multiple_replication , PR is here : https://github.com/vatesfr/xen-orchestra/pull/8668

it should be in master tomorrow , and hopefully in latest by the end of the week

it's merged in master (also with the same fix for mirror backups)

Gheppy

I tested with commit e64c434 and it is ok.
Thank you

olivierlambert

florent

@Gheppy it's always easier to fix when we can build a minimal setup that reproduce the issue
thanks for your report