"Backup fell back to a full" on delta backups
-
We recently updated Xen Orchestra to d77d6d0d9. When performing delta backups we started receiving the following "Backup fell back to a full" error:
We are using the NBD backup option, and have selected to have CBT snapshot data purged. This affects every VM. We have done testing and have tried to removing all backups, along with the task, and start over from fresh. With the second backup, and every backup there after, this error is received. And we have confirmed a full backup is being taken. We upgraded XO to the latest d7fe4f172, and this has not resolved the issue either.
Any thoughts on steps to resolve this issue? We are seeing it on two separate systems.
-
The error is no longer received when we disable "Purge snapshot data when using CBT".
-
Hi,
What kind of storage are you using? NFS, local LVM, local file base or iSCSI?
-
@olivierlambert It is an NFS share on a Synology NAS
-
Adding @florent in the loop
-
@MBNext is it possible to post the full json (you can download it on the top of the windows that shows the backup progress )
-
@florent see below:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1759651200002", "jobId": "ea94a502-b7f7-4e21-bd5f-093a11caa22a", "jobName": "Delta Backup", "message": "backup", "scheduleId": "dc6f41bf-2c5e-49e8-8f73-c1b7f0d1735c", "start": 1759651200002, "status": "success", "infos": [ { "data": { "vms": [ "55795ce2-ed10-4d85-c365-6eae35d0c094", "3e351bfb-d758-75c6-1a17-564e6560c537", "6750c30b-ce59-0d09-cbdb-f7136f793d10", "97f1af10-f2d0-ea89-9097-f92503c15813", "16653035-869a-46fc-98c4-3a18f70ab38c", "d3f07943-5779-5c75-9436-422b453ddd4c", "4f932e56-9125-6512-955d-e5f6249c8c8b", "fab1e47e-6b23-ac6e-2f6f-cadee708a13b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "55795ce2-ed10-4d85-c365-6eae35d0c094", "name_label": "Debian 12 Template" }, "id": "1759651206787", "message": "backup VM", "start": 1759651206787, "status": "success", "tasks": [ { "id": "1759651206839", "message": "clean-vm", "start": 1759651206839, "status": "success", "end": 1759651206987, "result": { "merge": false } }, { "id": "1759651207473:1", "message": "snapshot", "start": 1759651207473, "status": "success", "end": 1759651221770, "result": "d028ce1a-df34-a99d-945b-ed6cb7232d45" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651221770:0", "message": "export", "start": 1759651221770, "status": "success", "tasks": [ { "id": "1759651224150", "message": "transfer", "start": 1759651224150, "status": "success", "end": 1759651376950, "result": { "size": 4234149888 } }, { "id": "1759651379861:0", "message": "health check", "start": 1759651379861, "status": "success", "tasks": [ { "id": "1759651380004", "message": "transfer", "start": 1759651380004, "status": "success", "end": 1759651433663, "result": { "size": 0, "id": "2e5a80ab-ebc0-6fac-3978-ac2ffcc7bf11" } }, { "id": "1759651433664", "message": "vmstart", "start": 1759651433664, "status": "success", "end": 1759651436824 } ], "end": 1759651439096 }, { "id": "1759651439178", "message": "clean-vm", "start": 1759651439178, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/55795ce2-ed10-4d85-c365-6eae35d0c094/20251005T080024Z.json", "actual": 4234149888, "expected": 4235226624 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759651441127, "result": { "merge": false } } ], "end": 1759651441162 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:aa49c079-148e-69c9-f384-3a189e815d64" }, "message": "Snapshot data has been deleted" } ], "end": 1759651441162 }, { "data": { "type": "VM", "id": "97f1af10-f2d0-ea89-9097-f92503c15813", "name_label": "Xen Orchestra (Do Not Use)" }, "id": "1759651206781", "message": "backup VM", "start": 1759651206781, "status": "success", "tasks": [ { "id": "1759651206827", "message": "clean-vm", "start": 1759651206827, "status": "success", "end": 1759651206940, "result": { "merge": false } }, { "id": "1759651207473:2", "message": "snapshot", "start": 1759651207473, "status": "success", "end": 1759651208532, "result": "cf7376b2-1c46-16f6-7d8f-c842e6fdb024" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651208532:0", "message": "export", "start": 1759651208532, "status": "success", "tasks": [ { "id": "1759651210883", "message": "transfer", "start": 1759651210883, "status": "success", "end": 1759651450600, "result": { "size": 6847201280 } }, { "id": "1759651452327", "message": "health check", "start": 1759651452327, "status": "success", "tasks": [ { "id": "1759651452473", "message": "transfer", "start": 1759651452473, "status": "success", "end": 1759651545489, "result": { "size": 0, "id": "390e08f7-f932-7e54-8298-b9e9d411add0" } }, { "id": "1759651545490", "message": "vmstart", "start": 1759651545490, "status": "success", "end": 1759651548464 } ], "end": 1759651550936 }, { "id": "1759651550982", "message": "clean-vm", "start": 1759651550982, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/97f1af10-f2d0-ea89-9097-f92503c15813/20251005T080010Z.json", "actual": 6847201280, "expected": 6848915968 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759651562369, "result": { "merge": false } } ], "end": 1759651562536 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:75dd9c34-63db-f101-c57f-b3c0455f9b21" }, "message": "Snapshot data has been deleted" } ], "end": 1759651562536 }, { "data": { "type": "VM", "id": "6750c30b-ce59-0d09-cbdb-f7136f793d10", "name_label": "Docker/Host - Internal" }, "id": "1759651206790", "message": "backup VM", "start": 1759651206790, "status": "success", "tasks": [ { "id": "1759651206878", "message": "clean-vm", "start": 1759651206878, "status": "success", "end": 1759651207059, "result": { "merge": false } }, { "id": "1759651207473:0", "message": "snapshot", "start": 1759651207473, "status": "success", "end": 1759651215912, "result": "64184769-fb72-2b12-3da0-b1b5470a6419" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651215913", "message": "export", "start": 1759651215913, "status": "success", "tasks": [ { "id": "1759651219060", "message": "transfer", "start": 1759651219060, "status": "success", "end": 1759651494041, "result": { "size": 6427770880 } }, { "id": "1759651521940:0", "message": "health check", "start": 1759651521940, "status": "success", "tasks": [ { "id": "1759651523602", "message": "transfer", "start": 1759651523602, "status": "success", "end": 1759651609100, "result": { "size": 0, "id": "280aac5f-8b1e-c38e-4d61-2027641eebcc" } }, { "id": "1759651609101", "message": "vmstart", "start": 1759651609101, "status": "success", "end": 1759651612049 } ], "end": 1759651614305 }, { "id": "1759651614396", "message": "clean-vm", "start": 1759651614396, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/6750c30b-ce59-0d09-cbdb-f7136f793d10/20251005T080019Z.json", "actual": 6427770880, "expected": 6429383168 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759651617142, "result": { "merge": false } } ], "end": 1759651617181 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:940dbaf0-d954-bce0-a958-03f836759696" }, "message": "Snapshot data has been deleted" } ], "end": 1759651617181 }, { "data": { "type": "VM", "id": "16653035-869a-46fc-98c4-3a18f70ab38c", "name_label": "Xen Orchestra" }, "id": "1759651441167", "message": "backup VM", "start": 1759651441167, "status": "success", "tasks": [ { "id": "1759651441379", "message": "clean-vm", "start": 1759651441379, "status": "success", "end": 1759651442462, "result": { "merge": false } }, { "id": "1759651442738", "message": "snapshot", "start": 1759651442738, "status": "success", "end": 1759651444439, "result": "e6505f2e-3466-5967-e6b7-1673ab4db303" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651444439:0", "message": "export", "start": 1759651444439, "status": "success", "tasks": [ { "id": "1759651446955", "message": "transfer", "start": 1759651446955, "status": "success", "end": 1759651789366, "result": { "size": 7417626624 } }, { "id": "1759651792212:0", "message": "health check", "start": 1759651792212, "status": "success", "tasks": [ { "id": "1759651792315", "message": "transfer", "start": 1759651792315, "status": "success", "end": 1759651884950, "result": { "size": 0, "id": "4dc9df52-0114-5a1a-2fa0-86387e2eefff" } }, { "id": "1759651884951", "message": "vmstart", "start": 1759651884951, "status": "success", "end": 1759651888100 } ], "end": 1759651890459 }, { "id": "1759651890561", "message": "clean-vm", "start": 1759651890561, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/16653035-869a-46fc-98c4-3a18f70ab38c/20251005T080406Z.json", "actual": 7417626624, "expected": 7419480576 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759651905413, "result": { "merge": false } } ], "end": 1759651905551 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:0c719039-ea3c-e018-add3-8a84c9ae003b" }, "message": "Snapshot data has been deleted" } ], "end": 1759651905551 }, { "data": { "type": "VM", "id": "4f932e56-9125-6512-955d-e5f6249c8c8b", "name_label": "Docker/Portainer - Internal" }, "id": "1759651617205", "message": "backup VM", "start": 1759651617205, "status": "success", "tasks": [ { "id": "1759651617568", "message": "clean-vm", "start": 1759651617568, "status": "success", "end": 1759651619005, "result": { "merge": false } }, { "id": "1759651619272", "message": "snapshot", "start": 1759651619272, "status": "success", "end": 1759651620992, "result": "2b88ceb6-da16-12f0-847e-7d983e1f7299" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651620992:0", "message": "export", "start": 1759651620992, "status": "success", "tasks": [ { "id": "1759651623553", "message": "transfer", "start": 1759651623553, "status": "success", "end": 1759651886681, "result": { "size": 5857345536 } }, { "id": "1759651889014:0", "message": "health check", "start": 1759651889014, "status": "success", "tasks": [ { "id": "1759651889149", "message": "transfer", "start": 1759651889149, "status": "success", "end": 1759651960344, "result": { "size": 0, "id": "3d6c77dd-0662-1c42-6f98-f5ad3036e418" } }, { "id": "1759651960344:0", "message": "vmstart", "start": 1759651960344, "status": "success", "end": 1759651963364 } ], "end": 1759651965880 }, { "id": "1759651965983", "message": "clean-vm", "start": 1759651965983, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/4f932e56-9125-6512-955d-e5f6249c8c8b/20251005T080703Z.json", "actual": 5857345536, "expected": 5858818560 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759651967921, "result": { "merge": false } } ], "end": 1759651967951 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:a08c7415-b754-a97d-b0ba-c13a2e2117ec" }, "message": "Snapshot data has been deleted" } ], "end": 1759651967951 }, { "data": { "type": "VM", "id": "d3f07943-5779-5c75-9436-422b453ddd4c", "name_label": "Docker/Host - External" }, "id": "1759651562562", "message": "backup VM", "start": 1759651562562, "status": "success", "tasks": [ { "id": "1759651564305", "message": "clean-vm", "start": 1759651564305, "status": "success", "end": 1759651579194, "result": { "merge": false } }, { "id": "1759651580906", "message": "snapshot", "start": 1759651580906, "status": "success", "end": 1759651582598, "result": "1a16d72e-84ba-a635-7b11-6ff41fa6bb1a" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651582598:0", "message": "export", "start": 1759651582598, "status": "success", "tasks": [ { "id": "1759651595900", "message": "transfer", "start": 1759651595900, "status": "success", "end": 1759651962775, "result": { "size": 8115978240 } }, { "id": "1759651970651:0", "message": "health check", "start": 1759651970651, "status": "success", "tasks": [ { "id": "1759651970777", "message": "transfer", "start": 1759651970777, "status": "success", "end": 1759652060873, "result": { "size": 0, "id": "a94697da-0178-4540-b619-d493137cd87e" } }, { "id": "1759652060874", "message": "vmstart", "start": 1759652060874, "status": "success", "end": 1759652063833 } ], "end": 1759652066091 }, { "id": "1759652066182", "message": "clean-vm", "start": 1759652066182, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/d3f07943-5779-5c75-9436-422b453ddd4c/20251005T080635Z.json", "actual": 8115978240, "expected": 8118002688 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759652069227, "result": { "merge": false } } ], "end": 1759652069280 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:ab410ff8-77d4-b549-3837-a87e932423e9" }, "message": "Snapshot data has been deleted" } ], "end": 1759652069280 }, { "data": { "type": "VM", "id": "fab1e47e-6b23-ac6e-2f6f-cadee708a13b", "name_label": "Docker/Nextcloud - External" }, "id": "1759651905562", "message": "backup VM", "start": 1759651905562, "status": "success", "tasks": [ { "id": "1759651906819", "message": "clean-vm", "start": 1759651906819, "status": "success", "end": 1759651922004, "result": { "merge": false } }, { "id": "1759651923623", "message": "snapshot", "start": 1759651923623, "status": "success", "end": 1759651925135, "result": "079a3129-dc62-3a24-cde5-5816576dbe50" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651925136", "message": "export", "start": 1759651925136, "status": "success", "tasks": [ { "id": "1759651939259", "message": "transfer", "start": 1759651939259, "status": "success", "end": 1759652398143, "result": { "size": 20923285504 } }, { "id": "1759652400451", "message": "health check", "start": 1759652400451, "status": "success", "tasks": [ { "id": "1759652400580", "message": "transfer", "start": 1759652400580, "status": "success", "end": 1759652608594, "result": { "size": 0, "id": "6a1f30b6-c8fb-1c60-f5b0-c809099a1a6e" } }, { "id": "1759652608595", "message": "vmstart", "start": 1759652608595, "status": "success", "end": 1759652611594 } ], "end": 1759652614160 }, { "id": "1759652614248", "message": "clean-vm", "start": 1759652614248, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/fab1e47e-6b23-ac6e-2f6f-cadee708a13b/20251005T081219Z.json", "actual": 20923285504, "expected": 20928436736 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759652616019, "result": { "merge": false } } ], "end": 1759652616061 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:ed0c85b7-0bcc-e3bd-6a35-a401c6f091d2" }, "message": "Snapshot data has been deleted" } ], "end": 1759652616061 }, { "data": { "type": "VM", "id": "3e351bfb-d758-75c6-1a17-564e6560c537", "name_label": "Win11Pro1" }, "id": "1759651206786", "message": "backup VM", "start": 1759651206786, "status": "success", "tasks": [ { "id": "1759651206831", "message": "clean-vm", "start": 1759651206831, "status": "success", "end": 1759651207013, "result": { "merge": false } }, { "id": "1759651207473", "message": "snapshot", "start": 1759651207473, "status": "success", "end": 1759651209165, "result": "0048397c-9151-c83e-04d5-878955f8bc62" }, { "data": { "id": "fde89f48-d384-4208-8243-de5755ea7434", "isFull": false, "type": "remote" }, "id": "1759651209166", "message": "export", "start": 1759651209166, "status": "success", "tasks": [ { "id": "1759651214533", "message": "transfer", "start": 1759651214533, "status": "success", "end": 1759653069800, "result": { "size": 92809461760 } }, { "id": "1759653071177:0", "message": "health check", "start": 1759653071177, "status": "success", "tasks": [ { "id": "1759653071187", "message": "transfer", "start": 1759653071187, "status": "success", "end": 1759653899955, "result": { "size": 0, "id": "566c0b93-d3a0-b69c-351e-0d371f79c098" } }, { "id": "1759653899956", "message": "vmstart", "start": 1759653899956, "status": "success", "end": 1759653903489 } ], "end": 1759653906216 }, { "id": "1759653906330", "message": "clean-vm", "start": 1759653906330, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/3e351bfb-d758-75c6-1a17-564e6560c537/20251005T080014Z.json", "actual": 92809461760, "expected": 92832646656 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1759653906510, "result": { "merge": false } } ], "end": 1759653906511 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:b974e97e-1b63-cb25-1ba1-01c5ef9d9977" }, "message": "Snapshot data has been deleted" } ], "end": 1759653906511 } ], "end": 1759653906511 } -
Edited your post to use the right Markdown syntax, otherwise it's a pain to read
