Continuous Replication fails with XO update (June-20)

Andrew

@olivierlambert @florent
Continuous Replication fails for me now (XO source).... 2 VMs worked, 59 failed the same way using current commit ba7a29eb5e0ee1b66b06933ba953335e70bae21f

I think the problem started with: a85f44d699a3e581fe654cfd1118a28fd85e9945

This version works: 31461118e0e30afa2c15594744529357fd4f3696

 Retry the VM backup due to an error
 the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot destructure property 'other_config' of 'undefined' as it is undefined.. It won't be used anymore in this job execution.
 Retry the VM backup due to an error
 the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot destructure property 'other_config' of 'undefined' as it is undefined.. It won't be used anymore in this job execution.

Here's part of the log....

              "message": "transfer",
              "start": 1718906431567,
              "status": "failure",
              "end": 1718906431567,
              "result": {
                "message": "Cannot destructure property 'other_config' of 'undefined' as it is undefined.",
                "name": "TypeError",
                "stack": "TypeError: Cannot destructure property 'other_config' of 'undefined' as it is undefined.\n    at file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:101:47\n    at Array.filter (<anonymous>)\n    at #decorateVmMetadata (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:101:37)\n    at file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:134:71\n    at file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/Task.mjs:108:24\n    at Zone.run (/opt/xo/xo-builds/xen-orchestra-202406201339/node_modules/node-zone/index.js:80:23)\n    at Task.run (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/Task.mjs:106:23)\n    at Task.run (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/Task.mjs:38:27)\n    at IncrementalXapiWriter._transfer (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:133:16)\n    at IncrementalXapiWriter.transfer (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/_AbstractIncrementalWriter.mjs:22:25)"
              }
            }
          ],
          "end": 1718906431567,
          "result": {
            "message": "Cannot destructure property 'other_config' of 'undefined' as it is undefined.",
            "name": "TypeError",
            "stack": "TypeError: Cannot destructure property 'other_config' of 'undefined' as it is undefined.\n    at file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:101:47\n    at Array.filter (<anonymous>)\n    at #decorateVmMetadata (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:101:37)\n    at file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:134:71\n    at file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/Task.mjs:108:24\n    at Zone.run (/opt/xo/xo-builds/xen-orchestra-202406201339/node_modules/node-zone/index.js:80:23)\n    at Task.run (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/Task.mjs:106:23)\n    at Task.run (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/Task.mjs:38:27)\n    at IncrementalXapiWriter._transfer (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:133:16)\n    at IncrementalXapiWriter.transfer (file:///opt/xo/xo-builds/xen-orchestra-202406201339/@xen-orchestra/backups/_runners/_writers/_AbstractIncrementalWriter.mjs:22:25)"
          }

Danp

Hi @Andrew,

Thanks for the report. I've flagged this for the developers to take a closer look.

Regards,
Dan

florent

@Andrew said in Continuous Replication fails with XO update (June-20):

Here's part of the log....

          "message": "transfer",
          "start": 1718906431567,
          "status": "failure",

HI Andrew,

can you try this branch https://github.com/vatesfr/xen-orchestra/pull/7763 ?

regards

fbeauchamp opened this pull request in vatesfr/xen-orchestra

draft fix(backups): handle undefined VDI #7763

scot1297tutaio

I am also having issues with Continuous Replication:

️ the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot destructure property 'other_config' of 'undefined' as it is undefined.. It won't be used anymore in this job execution.
️ Retry the VM backup due to an error
️ the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot destructure property 'other_config' of 'undefined' as it is undefined.. It won't be used anymore in this job execution.
Error: Cannot destructure property 'other_config' of 'undefined' as it is undefined.

I am on commit a14c4 using XO from sources.

Scot

olivierlambert

@scot1297tutaio please try the branch @florent gave

scot1297tutaio

@olivierlambert I updated to that branch and it is working again. Thank you!

Andrew

@florent @olivierlambert That branch works again for me. No errors any more.

Delgado

I can confirm I upgraded to this branch and replication works again.

olivierlambert

Great news, this is helpful for our dev team, thanks for your feedback! Keep an eye on the PR to see when merged

Andrew

@florent @olivierlambert While this fixes one problem it adds a new one.

I am now getting the error VDI must be free or attached to exactly one VM on more than one VM. Which is correct because the CR process (and also sometimes delta backup) is leaving a VM attached to the Control Domain.

This problem shows up under the Dashboard->Health page. Manually deleting these stuck VDIs resolves the backup issue.

Now that the backup/CR process checks for this condition it becomes more of a problem that the process leaves VDIs attached to the control domain.

So, this update does fix the other_config issue but adds checks for other unrelated problems that exist and need to be resolved (Stuck VDI on Control Domain).

Andrew

@olivierlambert Any updates on this issue?

olivierlambert

Sadly I'm not supervising this part, maybe @julien-f is aware

Andrew

@julien-f @florent I tried the current XO commit 818a5b7f5bf29c021bc1612608c607022bac9c2b and still have the same problems, so I went back to fix_sr_vdi_undefined.

olivierlambert

Got it. We might ship that in a patch release if it's bundled in our tomorrow's latest XOA release.