XO Backup Error: VDI_IN_USE(OpaqueRef:.., destroy)
-
Hi there.
Given is a community edition XO on commit 3500d.
Since June 6th one out of seven VMs is no longer backing up.
It is always the same VM which is failing.
The remote is working (six VMs back up without issue).
It seems XO has trouble cleaning up the backup directory:
Here is the full backup log:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1780936394962", "jobId": "21bbca2a-59ad-4fa2-bb73-f60c1e9216e7", "jobName": "VM-Backup", "message": "backup", "scheduleId": "15044139-f4b7-4078-92bc-891e9b641d52", "start": 1780936394962, "status": "failure", "tasks": [ { "id": "0mq5fjqwf-22j12nbve5e", "start": 1780936397295, "status": "failure", "tasks": [ { "id": "0mq5fjqwy-ymsrhlbl67s", "start": 1780936397314, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/011114a9-b622-92eb-581f-36f961efe622/vdis/21bbca2a-59ad-4fa2-bb73-f60c1e9216e7/63a73be5-96ff-4c25-b7dd-c93284719916/20260608T150007Z.vhd", "error": {} }, "message": "failed to read disk parent info" } ], "end": 1780936399044, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fku8o-i3wwl2zfkza", "start": 1780936448280, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/011114a9-b622-92eb-581f-36f961efe622/vdis/21bbca2a-59ad-4fa2-bb73-f60c1e9216e7/63a73be5-96ff-4c25-b7dd-c93284719916/20260608T150007Z.vhd", "error": {} }, "message": "failed to read disk parent info" } ], "end": 1780936454149, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1780936454187, "result": { "code": "VDI_IN_USE", "params": [ "OpaqueRef:ce1e2c1b-9b52-d813-88fc-18513a51306f", "destroy" ], "task": { "uuid": "173eb901-7e4b-90d5-48cb-1306493d61d5", "name_label": "Async.VDI.destroy", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20260608T16:34:06Z", "finished": "20260608T16:34:08Z", "status": "failure", "resident_on": "OpaqueRef:8a7ca417-3c20-466f-8ebc-f19cbb63384f", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "VDI_IN_USE", "OpaqueRef:ce1e2c1b-9b52-d813-88fc-18513a51306f", "destroy" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 5212))((process xapi)(filename fun.ml)(line 33))((process xapi)(filename fun.ml)(line 38))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1821))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 5201))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))" }, "message": "VDI_IN_USE(OpaqueRef:ce1e2c1b-9b52-d813-88fc-18513a51306f, destroy)", "name": "XapiError", "stack": "XapiError: VDI_IN_USE(OpaqueRef:ce1e2c1b-9b52-d813-88fc-18513a51306f, destroy)\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202606081816/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202606081816/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202606081816/packages/xen-api/index.mjs:1078:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202606081816/packages/xen-api/index.mjs:1112:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202606081816/packages/xen-api/index.mjs:1102:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202606081816/packages/xen-api/index.mjs:1275:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)" }, "message": "backup VM", "data": { "id": "011114a9-b622-92eb-581f-36f961efe622", "type": "VM", "name_label": "ITServices" } }, { "id": "0mq5fjqwh-hhuivj17nmu", "start": 1780936397297, "status": "success", "tasks": [ { "id": "0mq5fjqx1-nwsciopgbxh", "start": 1780936397317, "status": "success", "end": 1780936399283, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fjsl1-29b0qivtg74", "start": 1780936399477, "status": "success", "end": 1780936401784, "result": "0d56ceb1-a627-1c97-e25d-ab641d5d3317", "message": "snapshot" }, { "id": "0mq5fjud4-an4b4yrzygv", "start": 1780936401784, "status": "success", "tasks": [ { "id": "0mq5fjvy2-qfouxpigsx", "start": 1780936403834, "status": "success", "end": 1780936418474, "result": { "size": 652498432 }, "message": "transfer", "data": { "progress": 99 } }, { "id": "0mq5fkcfx-f456mkr06ov", "start": 1780936425213, "status": "success", "end": 1780936426551, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780936428639, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": false, "type": "remote" } } ], "end": 1780936428639, "message": "backup VM", "data": { "id": "44ae09b4-47d2-ab1c-e187-3020f0626d74", "type": "VM", "name_label": "Unifi_OS_Server", "progress": 0 } }, { "id": "0mq5fjqwi-vahgy7faqum", "start": 1780936397298, "status": "success", "tasks": [ { "id": "0mq5fjqx0-7cex64zgf22", "start": 1780936397316, "status": "success", "end": 1780936399539, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fjswh-ubn0k70kwtb", "start": 1780936399889, "status": "success", "end": 1780936402821, "result": "5dc9f1c0-e790-c9cd-1309-ec58f64beedb", "message": "snapshot" }, { "id": "0mq5fjv5x-4cl03s5xrcs", "start": 1780936402821, "status": "success", "tasks": [ { "id": "0mq5fjwif-1eoxjx0ass2", "start": 1780936404567, "status": "success", "end": 1780936434954, "result": { "size": 1021871616 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mq5fkklg-ubob0jzzxb", "start": 1780936435780, "status": "success", "end": 1780936441161, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780936441176, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": false, "type": "remote" } } ], "end": 1780936441176, "message": "backup VM", "data": { "id": "e764ee17-19de-29fc-802f-e97385d0635d", "type": "VM", "name_label": "Graylog", "progress": 0 } }, { "id": "0mq5fjqwi-eyuvf4i18bv", "start": 1780936397298, "status": "success", "tasks": [ { "id": "0mq5fjqwx-kykb7016aud", "start": 1780936397313, "status": "success", "end": 1780936399348, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fjsl0-1jdpq922616", "start": 1780936399476, "status": "success", "end": 1780936400960, "result": "d403776f-e797-5af2-2428-35963e1af590", "message": "snapshot" }, { "id": "0mq5fjtq9-o2xr8f7gelf", "start": 1780936400961, "status": "success", "tasks": [ { "id": "0mq5fjvcs-7jaedl39bym", "start": 1780936403068, "status": "success", "end": 1780936471627, "result": { "size": 4130425344 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mq5fld9j-pzmbsqgh56", "start": 1780936472935, "status": "success", "end": 1780936476387, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780936476416, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": false, "type": "remote" } } ], "end": 1780936476416, "message": "backup VM", "data": { "id": "a329942c-10e9-70d4-4155-95c70e6c25c8", "type": "VM", "name_label": "XenOrchestra_Prod", "progress": 0 } }, { "id": "0mq5fkf41-weadm4wk1xp", "start": 1780936428673, "status": "success", "tasks": [ { "id": "0mq5fkf5n-2ponux8v3bm", "start": 1780936428731, "status": "success", "end": 1780936430525, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fkgor-fwufkt4zm1f", "start": 1780936430715, "status": "success", "end": 1780936431803, "result": "649ca352-0aab-dbd9-d75b-aae8748fb6a8", "message": "snapshot" }, { "id": "0mq5fkhiz-u5o8qh9lpoh", "start": 1780936431803, "status": "success", "tasks": [ { "id": "0mq5fklec-1glqm0n1642", "start": 1780936436820, "status": "success", "end": 1780936437167, "result": { "size": 43008 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mq5fkn5y-g2hl3nyuc2f", "start": 1780936439110, "status": "success", "end": 1780936440668, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780936440676, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": false, "type": "remote" } } ], "end": 1780936440676, "message": "backup VM", "data": { "id": "8ec416aa-4885-b82b-e9ac-24df17672916", "type": "VM", "name_label": "RustDesk", "progress": 0 } }, { "id": "0mq5fkodk-tkwoc2uthf", "start": 1780936440680, "status": "success", "tasks": [ { "id": "0mq5fkodq-e3kpftr1kr", "start": 1780936440686, "status": "success", "end": 1780936443708, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fkqvi-egmgr1fdy0t", "start": 1780936443918, "status": "success", "end": 1780936447236, "result": "8bec4422-5b08-5410-eff2-4f8d8c6d0ac7", "message": "snapshot" }, { "id": "0mq5fktfo-iuhc7r90uvc", "start": 1780936447236, "status": "success", "tasks": [ { "id": "0mq5fkv4r-dz2833iynb", "start": 1780936449435, "status": "success", "end": 1780936457597, "result": { "size": 180524032 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mq5fl3af-hde5wx2577", "start": 1780936460007, "status": "success", "end": 1780936461168, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780936461396, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": false, "type": "remote" } } ], "end": 1780936461396, "message": "backup VM", "data": { "id": "db880663-5c76-ad1d-ce9c-126db11c4822", "type": "VM", "name_label": "Nextcloud", "progress": 0 } }, { "id": "0mq5fkorf-psnurqj48qp", "start": 1780936441179, "status": "success", "tasks": [ { "id": "0mq5fkork-ui1kc8oo0i9", "start": 1780936441184, "status": "success", "end": 1780936443835, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5fkr16-ve61u2lg6yr", "start": 1780936444122, "status": "success", "end": 1780936448319, "result": "4436636b-5f56-347c-45d9-339dfbde7809", "message": "snapshot" }, { "id": "0mq5fku9r-4i3dvxltfoh", "start": 1780936448319, "status": "success", "tasks": [ { "id": "0mq5fkuzl-zbg2ht2gowa", "start": 1780936449249, "status": "success", "end": 1780936568326, "result": { "size": 8392878592 }, "message": "transfer", "data": { "progress": 99 } }, { "id": "0mq5fnfo4-lh7uyjgbpw", "start": 1780936569364, "status": "success", "end": 1780936570543, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780936570549, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": true, "type": "remote" } } ], "end": 1780936570549, "message": "backup VM", "data": { "id": "d01f5065-7ce2-cdfe-a11d-066000777c63", "type": "VM", "name_label": "Netbox", "progress": 0 } } ], "end": 1780936570551, "infos": [ { "data": { "vms": [ "44ae09b4-47d2-ab1c-e187-3020f0626d74", "e764ee17-19de-29fc-802f-e97385d0635d", "011114a9-b622-92eb-581f-36f961efe622", "a329942c-10e9-70d4-4155-95c70e6c25c8", "8ec416aa-4885-b82b-e9ac-24df17672916", "db880663-5c76-ad1d-ce9c-126db11c4822", "d01f5065-7ce2-cdfe-a11d-066000777c63" ] }, "message": "vms" } ] }Maybe this is linked to the thread by @andrew :
I upgraded today to commit 3500d, rebooted the XO VM and retried the backup. Still failing.
Thanks for the assistance.
Regards,
Marc -
@itservices Can you create a new backup job with just this VM and see if it's able to create a new chain?
-
Hi @DustinB
The VM could be backed up successfully with the same warning about the "disk parent":
The backup log:
{ "data": { "mode": "delta", "reportWhen": "never" }, "id": "1780940083480", "jobId": "8326600e-9863-4afa-b724-ae493f564ec8", "jobName": "ITServices-Detla-Test", "message": "backup", "scheduleId": "98f60867-3ac9-4ec5-b0fc-c3fe5520dc1b", "start": 1780940083480, "status": "success", "tasks": [ { "id": "0mq5hqsy6-9uqj9yfb22", "start": 1780940085774, "status": "success", "tasks": [ { "id": "0mq5hqsyx-i5pdr4zkson", "start": 1780940085801, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/011114a9-b622-92eb-581f-36f961efe622/vdis/21bbca2a-59ad-4fa2-bb73-f60c1e9216e7/63a73be5-96ff-4c25-b7dd-c93284719916/20260608T150007Z.vhd", "error": {} }, "message": "failed to read disk parent info" } ], "end": 1780940085910, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mq5hqtyu-9l484fbage", "start": 1780940087094, "status": "success", "end": 1780940087947, "result": "396960d5-93e9-670e-e8b5-345283148c0f", "message": "snapshot" }, { "id": "0mq5hqumk-7yfetdkltc2", "start": 1780940087948, "status": "success", "tasks": [ { "id": "0mq5hqvn3-wfuuk0sz1mi", "start": 1780940089263, "status": "success", "end": 1780940089285, "result": { "size": 104448 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mq5hr0dj-1l495zk6iys", "start": 1780940095399, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/011114a9-b622-92eb-581f-36f961efe622/vdis/21bbca2a-59ad-4fa2-bb73-f60c1e9216e7/63a73be5-96ff-4c25-b7dd-c93284719916/20260608T150007Z.vhd", "error": {} }, "message": "failed to read disk parent info" } ], "end": 1780940095573, "result": { "merge": true, "size": 0 }, "message": "clean-vm" } ], "end": 1780940095590, "message": "export", "data": { "id": "442a4fc2-70ff-474d-aae9-88d0249ef5f2", "isFull": false, "type": "remote" } } ], "end": 1780940095590, "message": "backup VM", "data": { "id": "011114a9-b622-92eb-581f-36f961efe622", "type": "VM", "name_label": "ITServices", "progress": 0 } } ], "end": 1780940095591, "infos": [ { "data": { "vms": [ "011114a9-b622-92eb-581f-36f961efe622" ] }, "message": "vms" } ] }I am using "smart mode" for backup jobs. The standalone job went through with and without "smart mode".
Removing the tag from the VM in question and adding it again did not help. I have also deleted the snapshot linked to the backup job and retried, with no change in behavior.
The VM is shutdown and I only turn it on when I need to. During backup it is usually not running.
Thanks again.
Regards,
Marc -
@itservices Hi,
It seems the parent of the backup chain is not found (a delta backup links to another delta and another, until you have a full backup, it seems a backup on this chain is missing).
Can you restart a backup chain to have a full backup? You can do it by reducing the retention in your backup job. -
Hi @pierrebrunet.
Thanks for the tip.
Didn't help, unfortunately. Same error message regarding VDI_IN_USE.
Regards,
Marc -
@itservices It may be a storage or a xo issue. You can try to restart xo-server and retry the backup. If it is solved, it was an issue with xo-server not freeing your vdi. If it still happens, it will be a storage issue
-
I have restarted XO as well as my NAS several times since Saturday.
Still no change.
Regards,
Marc -
The problem still persists with commit e376a.
After installation of updates to XenOrchestra I reboot the VM. Still no backup on this one VM possible.
For now I have created a separate backup job which seems to be working.
Regards,
Marc -
Commit 1ba82 did change something. It transferred the VDI but then fails with the VDI_IN_USE on that particular machine.
Regards,
Marc -
Problem still persists with commit e9b66.
Thanks for the support!
Regards,
Marc -
@itservices Hi,
We did not forget you, working on it! -
I am giving a short "report", when there is new version out and I have tested it.
Nice weekend @y'all
Regards,
Marc -
Problem still present in commit 04deb.
Here is the log from the XO6 interface:
"XapiError: VDI_IN_USE(OpaqueRef:ce1e2c1b-9b52-d813-88fc-18513a51306f, destroy)\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1078:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1112:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1102:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1275:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)"Thanks for your assistance ! ! !
Regards,
Marc -
@itservices Can you check your health dashboard where you should see " Unhealthy VDIs" ? If there are too many VDI in it, they need to be coalesced before restarting a backup.
-
Hi @pierrebrunet.
I have deleted the "parentless" VDIs and snapshots.
So the "Unhealthy VDI" page is empty for me.
So everytime I retry with the VM in question it is creating a new (full) backup-chain.
Regards,
Marc -
Quick Update:
Was looking good with commit cf26d.
After the transfer completed the VDI of that particular VM was still in use, according to XO.
So still failure on the job.
Regards,
Marc -
With commit caa17 the transfer did not happen again and fails the with the known message.
Regards,
Marc -
@itservices
Can you export your logs withxen-bugtool --yestoallplease? If you prefer to avoid exporting everything, we especially need/var/log/xensource.logand/var/log/SMlog
@Team-Storage do you have an idea about the source of the issue? -
With commit c74a8 the problem is still present but the transfer was completed again.
Unfortunately my installation does not recognize the "xen-bugtool" command. And both log files are not present.
Is this something that needs enabling after install of XCP-NG?
Unfortunately and unnoticed my syslog-server did not receive any messages from XCP-NG in the last 30 days.
Other inputs work
**Edit:
I have deleted and re-added the syslog-server with port and now logs are being send to Graylog.**
Thanks for your assistance y'all.
Regards,
Marc -
@itservices Hi, xen-bugtools is in xcp-ng hosts. You can directly get the logs from the hosts if you don't have the command.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login