Continuous Replication job fails "TypeError: Cannot read properties of undefined (reading 'uuid')" at #isAlreadyOnHealthCheckSr
-
Hello,
Thanks again for this awesome open-source project!
I discovered that one of my continuous replication backup jobs is failing to perform health checks despite completing transfers. I have two other CR jobs that are still working just fine. I updated Xen Orchestra this morning to commit 5f71e. This XO community edition (built from sources) instance is running on Debian 10 which has all of its system updates installed.
Is this something that I can fix on my own or might this be a symptom of some of the recent changes to backup nomenclature?
Here's the log file for the last run of this job:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1688145264192", "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d", "jobName": "Weekly continuous replication of core services and work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays", "message": "backup", "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6", "start": 1688145264192, "status": "failure", "infos": [ { "data": { "vms": [ "69ed2d54-b28a-4ec5-195f-dee283f730cf" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf", "name_label": "mgmt-win10.jeff.intangible.home.arpa" }, "id": "1688145336401", "message": "backup VM", "start": 1688145336401, "status": "failure", "tasks": [ { "id": "1688145386549", "message": "snapshot", "start": 1688145386549, "status": "success", "end": 1688145393151, "result": "88a6830c-b17c-1076-e6b9-6240888b778a" }, { "data": { "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9", "isFull": false, "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups", "type": "SR" }, "id": "1688145393152", "message": "export", "start": 1688145393152, "status": "failure", "tasks": [ { "id": "1688145396765", "message": "transfer", "start": 1688145396765, "status": "success", "end": 1688145408511, "result": { "size": 312320 } }, { "id": "1688145423393", "message": "health check", "start": 1688145423393, "status": "failure", "end": 1688145423434, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17" } } ], "end": 1688145423434, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17" } } ], "end": 1688145423447, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17" } } ], "end": 1688145423449 } -
Hi!
Can you try again (pulling last commit and rebuild) and tell us if you still have the issue?
-
@olivierlambert, thanks again for your help!
I see that there are new commits since my post--I guess I got a bit excited to update before the official announcement was made
As requested, I have updated and rebuilt to commit f32742225, but I am still getting the same issue when I attempt to retry a single VM in that CR job.
Below is the error log:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1688151228295", "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d", "jobName": "Weekly continuous replication of core services and work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays", "message": "backup", "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6", "start": 1688151228295, "status": "failure", "infos": [ { "data": { "vms": [ "69ed2d54-b28a-4ec5-195f-dee283f730cf" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf", "name_label": "mgmt-win10.jeff.intangible.home.arpa" }, "id": "1688151279573", "message": "backup VM", "start": 1688151279573, "status": "failure", "tasks": [ { "id": "1688151282519", "message": "snapshot", "start": 1688151282519, "status": "success", "end": 1688151288614, "result": "d74cc31c-5287-81f3-176c-ffb3998c9694" }, { "data": { "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9", "isFull": false, "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups", "type": "SR" }, "id": "1688151288616", "message": "export", "start": 1688151288616, "status": "failure", "tasks": [ { "id": "1688151292272", "message": "transfer", "start": 1688151292272, "status": "success", "end": 1688151313070, "result": { "size": 312320 } }, { "id": "1688151326976", "message": "health check", "start": 1688151326976, "status": "failure", "end": 1688151327016, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17" } } ], "end": 1688151327016, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17" } } ], "end": 1688151327029, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17" } } ], "end": 1688151327034 } -
@julien-f does this ring a bell?
-
Ping @florent
-
@techjeff hi
This can happen if either the SR of one of the disk or the one used for the health check don't exists
Can you check it ?
-
Hi, @florent. Thank you for your assistance!
The SR used for the health check is available and functional (most of my running VMs boot from this SR). This SR has only 225GB free, but the VM failing the health check has 1x 50GB disk and 1x 100GB disk, so that should be enough to perform a single VM health check. The destination SR for the CR job is not the same as the SR used for the health check. Do they need to be the same? I assume they do not need to be the same since I can choose which SR to use for the health check.
I have 3 hosts in my pool; one is powered down unless needed, so the SR shows an orange color dot because it is not connected to all hosts. Perhaps this could be the reason.
Thanks again for your assistance. FYI, I am about to go on a camping trip (after sending this message) and won't return until July 12. I'll check this thread when I return.
-
Hi, @florent. I'm just checking in to confirm that the SR I used for health checks is available and in use. When I disable the health check, the backup job works just fine. I configured the CR job with a different health check SR, but the job failed with the same error.
Please let me know if there is any other information that you would like from me or if there is anything else I can do to assist.
-
@techjeff are you up to date to master/latest ? I think it have already been fixed
-
@florent I just updated to Xen Orchestra, commit 6b936, then attempted to restart the failed backup (only a single VM, not the whole job) and I am still getting the error.
It looks like there was another commit about an hour ago, but it doesn't look to be related to this at all.
Here's the full JSON error log:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1690565798795", "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d", "jobName": "Weekly continuous replication of core services and Work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays", "message": "backup", "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6", "start": 1690565798795, "status": "failure", "infos": [ { "data": { "vms": [ "69ed2d54-b28a-4ec5-195f-dee283f730cf" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf", "name_label": "mgmt-win10.jeff.intangible.home.arpa" }, "id": "1690565809617", "message": "backup VM", "start": 1690565809617, "status": "failure", "tasks": [ { "id": "1690565811948", "message": "snapshot", "start": 1690565811948, "status": "success", "end": 1690565818268, "result": "97927cb2-10fc-e0ff-5c6a-0e3a0b3e405d" }, { "data": { "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9", "isFull": false, "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups", "type": "SR" }, "id": "1690565818269", "message": "export", "start": 1690565818269, "status": "failure", "tasks": [ { "id": "1690565821451", "message": "transfer", "start": 1690565821451, "status": "success", "end": 1690565843248, "result": { "size": 312320 } }, { "id": "1690565868762", "message": "health check", "start": 1690565868762, "status": "failure", "end": 1690565868794, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690565868794, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690565868806, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690565868807 } -
@techjeff strange
I will look into this -
@florent Thank you for your assistance!
I'm also running into this issue after upgrading to the latest build (Xen Orchestra, commit d3077):
{ "data": { "id": "349e7130-da43-6550-e7c0-aa0a84102e0f", "isFull": false, "name_label": "Local storage", "type": "SR" }, "id": "1690857015572:0", "message": "export", "start": 1690857015572, "status": "failure", "tasks": [ { "id": "1690857016890", "message": "transfer", "start": 1690857016890, "status": "success", "end": 1690857173019, "result": { "size": 8317447680 } }, { "id": "1690857209204:0", "message": "health check", "start": 1690857209204, "status": "failure", "end": 1690857209217, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857209217, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857647768 }, { "data": { "type": "VM", "id": "a0ba50c3-f6d3-32ff-cda7-9243d291a6c0", "name_label": "XOCE" }, "id": "1690857647776", "message": "backup VM", "start": 1690857647776, "status": "failure", "tasks": [ { "id": "1690857647781", "message": "clean-vm", "start": 1690857647781, "status": "success", "end": 1690857647865, "result": { "merge": false } }, { "id": "1690857647964", "message": "snapshot", "start": 1690857647964, "status": "success", "end": 1690857649748, "result": "a25a21f6-cfdd-5f8d-9753-36a7f1eb7302" }, { "data": { "id": "45ff0925-797b-43dd-a95c-7e298c61ad02", "isFull": false, "type": "remote" }, "id": "1690857649748:0", "message": "export", "start": 1690857649748, "status": "success", "tasks": [ { "id": "1690857651540", "message": "transfer", "start": 1690857651540, "status": "success", "end": 1690857709369, "result": { "size": 1015333376 } }, { "id": "1690857713360", "message": "health check", "start": 1690857713360, "status": "success", "tasks": [ { "id": "1690857713479", "message": "transfer", "start": 1690857713479, "status": "success", "end": 1690857820321, "result": { "size": 18864355840, "id": "2d68d34a-7a93-39fb-0095-a505a761c590" } }, { "id": "1690857820321:0", "message": "vmstart", "start": 1690857820321, "status": "success", "end": 1690857840697 } ], "end": 1690857842923 }, { "id": "1690857842923:0", "message": "clean-vm", "start": 1690857842923, "status": "success", "end": 1690857842950, "result": { "merge": true } } ], "end": 1690857842962 }, { "data": { "id": "349e7130-da43-6550-e7c0-aa0a84102e0f", "isFull": false, "name_label": "Local storage", "type": "SR" }, "id": "1690857649749", "message": "export", "start": 1690857649749, "status": "failure", "tasks": [ { "id": "1690857651536", "message": "transfer", "start": 1690857651536, "status": "success", "end": 1690857708807, "result": { "size": 1015333376 } }, { "id": "1690857713360:0", "message": "health check", "start": 1690857713360, "status": "failure", "end": 1690857713370, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857713370, "result": { "message": "Cannot read properties of undefined (reading 'uuid')", "name": "TypeError", "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n at #isAlreadyOnHealthCheckSr (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17" } } ], "end": 1690857842962 },Thank you again!
SW
-
Thank you for your assistance, @florent! Please let me know if you need additional information or want me to perform tests.
-
@techjeff Are your replication done with or without memory ?
-
Hi, @florent. The CR job is configured for "Normal" snapshots, not with memory.
Also, for curiosity's sake, I decided to rebuild this same rule from scratch--the hypothesis was that the record of the original CR job could have been somehow "broken" or now incompatible. The backup job was rebuilt with exactly the same settings apart from "Rebuild of " being added to the beginning of the job and schedule names.
Because it's a new CR backup of ~20 VMs to really cheap spinning disks, the backup is still in progress after 8 hours (another delta-backup job also started ~30 minutes after I started the new job), but so far it has yielded the same results: apart from the 2 VMs that are in progress, all other VMs have failed with the same TypeError.
I'm currently running Xen Orchestra, commit abd0a built from sources using ronivay's XenOrchestraInstallerUpdater.
-
@techjeff thanks for your effort, I found the problem
can you test this branch : fix_cr_healthcheck ? (
git checkout fix_cr_healthcheckfrom xen-orchestra folder) it will be merged soon -
@florent thank you for your effort as well!
I will checkout that branch, attempt to restart the backup job for one of the small VMs, then I'll report back here, hopefully before the end of the day (UTC-7).
-
@florent I reviewed the change you made and got slightly embarrassed that my brain didn't notice the missing "l" when I had previously reviewed
MixinXapiWriter.mjs
After checking out the
fix_cr_healthcheckbranch and rebuilding, I restarted the backup of just one VM in the backup job and it was successful!Now, I will run the full backup job and report back for the sake of being thorough.
-
@florent the CR job was completed with health checks. The issue appears to be fixed in the
fix_cr_healthcheckbranch. -
Thanks for your feedback @techjeff !
