Continuous Replication job fails "TypeError: Cannot read properties of undefined (reading 'uuid')" at #isAlreadyOnHealthCheckSr

stevewest15

@florent Thank you for your assistance!

I'm also running into this issue after upgrading to the latest build (Xen Orchestra, commit d3077):

        {
          "data": {
            "id": "349e7130-da43-6550-e7c0-aa0a84102e0f",
            "isFull": false,
            "name_label": "Local storage",
            "type": "SR"
          },
          "id": "1690857015572:0",
          "message": "export",
          "start": 1690857015572,
          "status": "failure",
          "tasks": [
            {
              "id": "1690857016890",
              "message": "transfer",
              "start": 1690857016890,
              "status": "success",
              "end": 1690857173019,
              "result": {
                "size": 8317447680
              }
            },
            {
              "id": "1690857209204:0",
              "message": "health check",
              "start": 1690857209204,
              "status": "failure",
              "end": 1690857209217,
              "result": {
                "message": "Cannot read properties of undefined (reading 'uuid')",
                "name": "TypeError",
                "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
              }
            }
          ],
          "end": 1690857209217,
          "result": {
            "message": "Cannot read properties of undefined (reading 'uuid')",
            "name": "TypeError",
            "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
          }
        }
      ],
      "end": 1690857647768
    },
    {
      "data": {
        "type": "VM",
        "id": "a0ba50c3-f6d3-32ff-cda7-9243d291a6c0",
        "name_label": "XOCE"
      },
      "id": "1690857647776",
      "message": "backup VM",
      "start": 1690857647776,
      "status": "failure",
      "tasks": [
        {
          "id": "1690857647781",
          "message": "clean-vm",
          "start": 1690857647781,
          "status": "success",
          "end": 1690857647865,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1690857647964",
          "message": "snapshot",
          "start": 1690857647964,
          "status": "success",
          "end": 1690857649748,
          "result": "a25a21f6-cfdd-5f8d-9753-36a7f1eb7302"
        },
        {
          "data": {
            "id": "45ff0925-797b-43dd-a95c-7e298c61ad02",
            "isFull": false,
            "type": "remote"
          },
          "id": "1690857649748:0",
          "message": "export",
          "start": 1690857649748,
          "status": "success",
          "tasks": [
            {
              "id": "1690857651540",
              "message": "transfer",
              "start": 1690857651540,
              "status": "success",
              "end": 1690857709369,
              "result": {
                "size": 1015333376
              }
            },
            {
              "id": "1690857713360",
              "message": "health check",
              "start": 1690857713360,
              "status": "success",
              "tasks": [
                {
                  "id": "1690857713479",
                  "message": "transfer",
                  "start": 1690857713479,
                  "status": "success",
                  "end": 1690857820321,
                  "result": {
                    "size": 18864355840,
                    "id": "2d68d34a-7a93-39fb-0095-a505a761c590"
                  }
                },
                {
                  "id": "1690857820321:0",
                  "message": "vmstart",
                  "start": 1690857820321,
                  "status": "success",
                  "end": 1690857840697
                }
              ],
              "end": 1690857842923
            },
            {
              "id": "1690857842923:0",
              "message": "clean-vm",
              "start": 1690857842923,
              "status": "success",
              "end": 1690857842950,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1690857842962
        },
        {
          "data": {
            "id": "349e7130-da43-6550-e7c0-aa0a84102e0f",
            "isFull": false,
            "name_label": "Local storage",
            "type": "SR"
          },
          "id": "1690857649749",
          "message": "export",
          "start": 1690857649749,
          "status": "failure",
          "tasks": [
            {
              "id": "1690857651536",
              "message": "transfer",
              "start": 1690857651536,
              "status": "success",
              "end": 1690857708807,
              "result": {
                "size": 1015333376
              }
            },
            {
              "id": "1690857713360:0",
              "message": "health check",
              "start": 1690857713360,
              "status": "failure",
              "end": 1690857713370,
              "result": {
                "message": "Cannot read properties of undefined (reading 'uuid')",
                "name": "TypeError",
                "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
              }
            }
          ],
          "end": 1690857713370,
          "result": {
            "message": "Cannot read properties of undefined (reading 'uuid')",
            "name": "TypeError",
            "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
          }
        }
      ],
      "end": 1690857842962
    },

Thank you again!

SW

techjeff

Thank you for your assistance, @florent! Please let me know if you need additional information or want me to perform tests.

florent

@techjeff Are your replication done with or without memory ?

techjeff

Hi, @florent. The CR job is configured for "Normal" snapshots, not with memory.

Also, for curiosity's sake, I decided to rebuild this same rule from scratch--the hypothesis was that the record of the original CR job could have been somehow "broken" or now incompatible. The backup job was rebuilt with exactly the same settings apart from "Rebuild of " being added to the beginning of the job and schedule names.

Because it's a new CR backup of ~20 VMs to really cheap spinning disks, the backup is still in progress after 8 hours (another delta-backup job also started ~30 minutes after I started the new job), but so far it has yielded the same results: apart from the 2 VMs that are in progress, all other VMs have failed with the same TypeError.

I'm currently running Xen Orchestra, commit abd0a built from sources using ronivay's XenOrchestraInstallerUpdater.

florent

@techjeff thanks for your effort, I found the problem

can you test this branch : fix_cr_healthcheck ? (git checkout fix_cr_healthcheck from xen-orchestra folder) it will be merged soon

techjeff

@florent thank you for your effort as well!

I will checkout that branch, attempt to restart the backup job for one of the small VMs, then I'll report back here, hopefully before the end of the day (UTC-7).

techjeff

@florent I reviewed the change you made and got slightly embarrassed that my brain didn't notice the missing "l" when I had previously reviewed MixinXapiWriter.mjs

After checking out the fix_cr_healthcheck branch and rebuilding, I restarted the backup of just one VM in the backup job and it was successful!

Now, I will run the full backup job and report back for the sake of being thorough.

techjeff

@florent the CR job was completed with health checks. The issue appears to be fixed in the fix_cr_healthcheck branch.

olivierlambert

Thanks for your feedback @techjeff !

techjeff

@olivierlambert Thank you and your team again for your commitment to this fantastic FOSS tool and for allowing me to build it myself!

I very much appreciate the personal touch of my issues being triaged by the CEO and Co-Founder. It's refreshing to see an executive officer stay in touch with their customer base.