Continuous Replication job fails "TypeError: Cannot read properties of undefined (reading 'uuid')" at #isAlreadyOnHealthCheckSr
-
@techjeff are you up to date to master/latest ? I think it have already been fixed
-
@florent I just updated to Xen Orchestra, commit 6b936, then attempted to restart the failed backup (only a single VM, not the whole job) and I am still getting the error.
It looks like there was another commit about an hour ago, but it doesn't look to be related to this at all.
Here's the full JSON error log:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1690565798795", "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d", "jobName": "Weekly continuous replication of core services and Work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays", "message": "backup", "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6", "start": 1690565798795, "status": "failure", "infos": [ { "data": { "vms": [ "69ed2d54-b28a-4ec5-195f-dee283f730cf" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf", "name_label": "mgmt-win10.jeff.intangible.home.arpa" }, "id": "1690565809617", "message": "backup VM", "start": 1690565809617, "status": "failure", "tasks": [ { "id": "1690565811948", "message": "snapshot", "start": 1690565811948, "status": "success", "end": 1690565818268, "result": "97927cb2-10fc-e0ff-5c6a-0e3a0b3e405d" }, { "data": { "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9", "isFull": false, "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups", "type": "SR" }, "id": "1690565818269", "message": "export", "start": 1690565818269, "status": "failure", "tasks": [ { "id": "1690565821451", "message": "transfer", "start": 1690565821451, "status": "success", "end": 1690565843248, "result": { "size": 312320 } }, { "id": "1690565868762", "message": "health check", "start": 1690565868762, "status": "failure", "end": 1690565868794, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690565868794, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690565868806, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690565868807 } -
@techjeff strange
I will look into this -
@florent Thank you for your assistance!
I'm also running into this issue after upgrading to the latest build (Xen Orchestra, commit d3077):
{ "data": { "id": "349e7130-da43-6550-e7c0-aa0a84102e0f", "isFull": false, "name_label": "Local storage", "type": "SR" }, "id": "1690857015572:0", "message": "export", "start": 1690857015572, "status": "failure", "tasks": [ { "id": "1690857016890", "message": "transfer", "start": 1690857016890, "status": "success", "end": 1690857173019, "result": { "size": 8317447680 } }, { "id": "1690857209204:0", "message": "health check", "start": 1690857209204, "status": "failure", "end": 1690857209217, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857209217, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857647768 }, { "data": { "type": "VM", "id": "a0ba50c3-f6d3-32ff-cda7-9243d291a6c0", "name_label": "XOCE" }, "id": "1690857647776", "message": "backup VM", "start": 1690857647776, "status": "failure", "tasks": [ { "id": "1690857647781", "message": "clean-vm", "start": 1690857647781, "status": "success", "end": 1690857647865, "result": { "merge": false } }, { "id": "1690857647964", "message": "snapshot", "start": 1690857647964, "status": "success", "end": 1690857649748, "result": "a25a21f6-cfdd-5f8d-9753-36a7f1eb7302" }, { "data": { "id": "45ff0925-797b-43dd-a95c-7e298c61ad02", "isFull": false, "type": "remote" }, "id": "1690857649748:0", "message": "export", "start": 1690857649748, "status": "success", "tasks": [ { "id": "1690857651540", "message": "transfer", "start": 1690857651540, "status": "success", "end": 1690857709369, "result": { "size": 1015333376 } }, { "id": "1690857713360", "message": "health check", "start": 1690857713360, "status": "success", "tasks": [ { "id": "1690857713479", "message": "transfer", "start": 1690857713479, "status": "success", "end": 1690857820321, "result": { "size": 18864355840, "id": "2d68d34a-7a93-39fb-0095-a505a761c590" } }, { "id": "1690857820321:0", "message": "vmstart", "start": 1690857820321, "status": "success", "end": 1690857840697 } ], "end": 1690857842923 }, { "id": "1690857842923:0", "message": "clean-vm", "start": 1690857842923, "status": "success", "end": 1690857842950, "result": { "merge": true } } ], "end": 1690857842962 }, { "data": { "id": "349e7130-da43-6550-e7c0-aa0a84102e0f", "isFull": false, "name_label": "Local storage", "type": "SR" }, "id": "1690857649749", "message": "export", "start": 1690857649749, "status": "failure", "tasks": [ { "id": "1690857651536", "message": "transfer", "start": 1690857651536, "status": "success", "end": 1690857708807, "result": { "size": 1015333376 } }, { "id": "1690857713360:0", "message": "health check", "start": 1690857713360, "status": "failure", "end": 1690857713370, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857713370, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857842962 },Thank you again!
SW
-
Thank you for your assistance, @florent! Please let me know if you need additional information or want me to perform tests.
-
@techjeff Are your replication done with or without memory ?
-
Hi, @florent. The CR job is configured for "Normal" snapshots, not with memory.
Also, for curiosity's sake, I decided to rebuild this same rule from scratch--the hypothesis was that the record of the original CR job could have been somehow "broken" or now incompatible. The backup job was rebuilt with exactly the same settings apart from "Rebuild of " being added to the beginning of the job and schedule names.
Because it's a new CR backup of ~20 VMs to really cheap spinning disks, the backup is still in progress after 8 hours (another delta-backup job also started ~30 minutes after I started the new job), but so far it has yielded the same results: apart from the 2 VMs that are in progress, all other VMs have failed with the same TypeError.
I'm currently running Xen Orchestra, commit abd0a built from sources using ronivay's XenOrchestraInstallerUpdater.
-
@techjeff thanks for your effort, I found the problem
can you test this branch : fix_cr_healthcheck ? (
git checkout fix_cr_healthcheckfrom xen-orchestra folder) it will be merged soon -
@florent thank you for your effort as well!
I will checkout that branch, attempt to restart the backup job for one of the small VMs, then I'll report back here, hopefully before the end of the day (UTC-7).
-
@florent I reviewed the change you made and got slightly embarrassed that my brain didn't notice the missing "l" when I had previously reviewed
MixinXapiWriter.mjs
After checking out the
fix_cr_healthcheckbranch and rebuilding, I restarted the backup of just one VM in the backup job and it was successful!Now, I will run the full backup job and report back for the sake of being thorough.
-
@florent the CR job was completed with health checks. The issue appears to be fixed in the
fix_cr_healthcheckbranch. -
Thanks for your feedback @techjeff !
-
@olivierlambert Thank you and your team again for your commitment to this fantastic FOSS tool and for allowing me to build it myself!
I very much appreciate the personal touch of my issues being triaged by the CEO and Co-Founder. It's refreshing to see an executive officer stay in touch with their customer base.
