Our future backup code: test it!
-
@florent Will update and try at the end of today then (GMT)
-
@manilx this is merged
-
-
\o/
-
This post is deleted! -
CR still failed with
Error: HANDLE_INVALID(SR, OpaqueRef:cc116d5f-0a91-b487-b43d-4ebd98495fcd)
Error: Error in the source generator Cannot read properties of undefined (reading 'write')
i found that old CR copies dissapeared from all hosts. -
@Tristis-Oris I still don't have the solution I am working on it .
-
@florent Getting errors again:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1748134829731", "jobId": "b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6", "jobName": "Mirror Mani", "message": "backup", "scheduleId": "9e8eeb18-65bb-4e12-b1a1-4f7c0490565c", "start": 1748134829731, "status": "failure", "infos": [ { "data": { "vms": [ "0c8fd092-ab36-478e-df85-16b1a01e1ca8", "19242523-e70b-bf39-820d-5e322435012b", "1a65641e-0976-07a8-30f8-3cbd2e4bc63d", "1c596fc8-26a8-7d95-d004-1e694392fc72", "1d7614fc-65f7-4e1c-71f9-da638efd8b4d", "284b64bc-8684-eb3c-b32e-93e455f5854b", "294b867a-e241-fe7a-10b0-aef1de6f693d", "341a6155-908b-66ba-76ee-56a1d5163af1", "43a4d905-7d13-85b8-bed3-f6b805ff26ac", "48ca6a7e-7acf-74fd-4ad9-714315563652", "4d8cad25-b6e5-4d8f-5226-b5908b21a1b6", "52e64134-62e3-9682-4e3f-296a1198db4d", "58af91d1-94cf-cbe5-88cf-560e652741bf", "592e229c-76d7-c0f2-426c-4ef9e80810d0", "5f49e0a4-9397-3da5-e44a-0ab3632cd9f1", "6c9f0275-07bd-a124-e187-47f8389ed93c", "8d573273-49b1-6b84-73a6-42c3346add52", "8e76ba5c-e190-753b-75d6-e67feb1dd85c", "9017c283-51d0-8d22-d13b-328cc9fe4ff7", "903d61e0-c07f-8163-47ae-8e212147a353", "931aaab6-5f7a-d0ef-26a1-1f54063d0941", "9e0c8a72-d0fd-6632-6d57-94224149af05", "9f8088b6-9a5d-0ec5-24d5-118c23100a4b", "a060d911-bf2b-61c5-56bf-11944af25c4a", "a2562d66-00c8-3955-ab43-6a5584044b98", "ac14b70f-9958-7add-f833-64486feba4e7", "b02697d4-23b5-598f-4def-d6b197bc1085", "b5d74e0b-388c-019a-6994-e174c9ca7a51", "bcb4bcfc-c77b-2068-4bec-2826e8ece6be", "c21e8541-c5ea-840e-31aa-095789600230", "c49a5398-38db-4fcd-481b-30181a5503cc", "c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b", "cc6b00b9-16ec-1990-e44a-503b6e3921c3", "d7964354-aff6-c7e1-0067-52e8e866a965", "ef53c584-f6c2-3aa3-b4c0-802fb20683d9", "f262e69e-f345-bc52-c2dc-60eea7b84c9a", "fab6b6c0-7ce7-91da-fe60-acbcea4ca5be", "fe2dd0ea-f063-9aba-30d5-d2de508b091f" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "19242523-e70b-bf39-820d-5e322435012b" }, "id": "1748134830645", "message": "backup VM", "start": 1748134830645, "status": "success", "tasks": [ { "id": "1748134830822", "message": "clean-vm", "start": 1748134830822, "status": "success", "end": 1748134831309, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748134831397", "message": "export", "start": 1748134831397, "status": "success", "tasks": [ { "id": "1748134831413", "message": "transfer", "start": 1748134831413, "status": "success", "end": 1748134937238, "result": { "size": 710934528 } }, { "id": "1748134947620", "message": "clean-vm", "start": 1748134947620, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/19242523-e70b-bf39-820d-5e322435012b/20250524T220611Z.json", "actual": 710934528, "expected": 713224192 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748134949566, "result": { "merge": true } } ], "end": 1748134965928 } ], "end": 1748134965928 }, { "data": { "type": "VM", "id": "0c8fd092-ab36-478e-df85-16b1a01e1ca8" }, "id": "1748134830649", "message": "backup VM", "start": 1748134830649, "status": "interrupted", "tasks": [ { "id": "1748134830750", "message": "clean-vm", "start": 1748134830750, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/20250522T161036Z.json", "actual": 71360596992, "expected": 71360597504 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748134831421, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748134834838", "message": "export", "start": 1748134834838, "status": "success", "tasks": [ { "id": "1748134834899", "message": "transfer", "start": 1748134834899, "status": "success", "end": 1748135312431, "result": { "size": 4857004032 } }, { "id": "1748135312559", "message": "clean-vm", "start": 1748135312559, "status": "success", "warnings": [ { "data": { "parent": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45", "child": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45/20250524T220111Z.vhd" }, "message": "parent VHD is missing" }, { "data": { "backup": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/20250524T220111Z.json", "missingVhds": [ "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45/20250524T220111Z.vhd" ] }, "message": "some VHDs linked to the backup are missing" } ], "end": 1748135312819, "result": { "merge": false } } ], "end": 1748135312823 } ], "warnings": [ { "data": { "attempt": 1, "error": "Expected values to be strictly equal:\n\n0 !== 512\n" }, "message": "Retry the VM mirror backup due to an error" } ] }, { "data": { "type": "VM", "id": "1c596fc8-26a8-7d95-d004-1e694392fc72" }, "id": "1748134966251", "message": "backup VM", "start": 1748134966251, "status": "success", "tasks": [ { "id": "1748134968074", "message": "clean-vm", "start": 1748134968074, "status": "success", "end": 1748134968813, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748134971622", "message": "export", "start": 1748134971622, "status": "success", "tasks": [ { "id": "1748134971997", "message": "transfer", "start": 1748134971997, "status": "success", "end": 1748135312426, "result": { "size": 3261071360 } }, { "id": "1748135314472", "message": "clean-vm", "start": 1748135314472, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/1c596fc8-26a8-7d95-d004-1e694392fc72/20250524T222347Z.json", "actual": 3261071360, "expected": 3264098816 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135314652, "result": { "merge": true } } ], "end": 1748135314707 } ], "end": 1748135314707 }, { "data": { "type": "VM", "id": "294b867a-e241-fe7a-10b0-aef1de6f693d" }, "id": "1748135312961", "message": "backup VM", "start": 1748135312961, "status": "interrupted", "tasks": [ { "id": "1748135313037", "message": "clean-vm", "start": 1748135313037, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/294b867a-e241-fe7a-10b0-aef1de6f693d/20250522T161022Z.json", "actual": 11216235520, "expected": 11216236032 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135313727, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135313903", "message": "export", "start": 1748135313903, "status": "failure", "tasks": [ { "id": "1748135313921", "message": "transfer", "start": 1748135313921, "status": "failure", "end": 1748135318829, "result": { "errno": -9, "code": "EBADF", "syscall": "read", "message": "EBADF: bad file descriptor, read", "name": "Error", "stack": "Error: EBADF: bad file descriptor, read\nFrom:\n at NfsHandler.addSyncStackTrace (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:21:26)\n at NfsHandler._read (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:160:43)\n at NfsHandler.read (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:259:17)\n at NfsHandler.read (/opt/xen-orchestra/node_modules/limit-concurrency-decorator/index.js:97:24)\n at VhdFile._read (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:126:55)\n at VhdFile.readBlock (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:212:17)\n at RemoteVhd.readBlock (file:///opt/xen-orchestra/@xen-orchestra/backups/disks/RemoteVhd.mjs:144:38)\n at RemoteVhd.buildDiskBlockGenerator (file:///opt/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:67:28)" } }, { "id": "1748135318836", "message": "clean-vm", "start": 1748135318836, "status": "success", "end": 1748135318927, "result": { "merge": false } } ], "end": 1748135318948 } ], "warnings": [ { "data": { "attempt": 1, "error": "ENOENT: no such file or directory, open '/run/xo-server/mounts/337848e1-b52b-4e08-9bf0-ffce54de4276/xo-vm-backups/294b867a-e241-fe7a-10b0-aef1de6f693d/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/c87b9871-9ce2-4b39-9fec-9dea2fac3220/20250524T220036Z.vhd'" }, "message": "Retry the VM mirror backup due to an error" } ] }, { "data": { "type": "VM", "id": "43a4d905-7d13-85b8-bed3-f6b805ff26ac" }, "id": "1748135318587", "message": "backup VM", "start": 1748135318587, "status": "success", "tasks": [ { "id": "1748135318860", "message": "clean-vm", "start": 1748135318860, "status": "success", "end": 1748135319739, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135319979", "message": "export", "start": 1748135319979, "status": "success", "tasks": [ { "id": "1748135319993", "message": "transfer", "start": 1748135319993, "status": "success", "end": 1748135458653, "result": { "size": 870318080 } }, { "id": "1748135473880", "message": "clean-vm", "start": 1748135473880, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/43a4d905-7d13-85b8-bed3-f6b805ff26ac/20250524T220144Z.json", "actual": 870318080, "expected": 872691712 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135483581, "result": { "merge": true } } ], "end": 1748135487189 } ], "end": 1748135487190 }, { "data": { "type": "VM", "id": "58af91d1-94cf-cbe5-88cf-560e652741bf" }, "id": "1748135319145", "message": "backup VM", "start": 1748135319145, "status": "success", "tasks": [ { "id": "1748135319356", "message": "clean-vm", "start": 1748135319356, "status": "success", "end": 1748135320011, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135331775", "message": "export", "start": 1748135331775, "status": "success", "tasks": [ { "id": "1748135331781", "message": "transfer", "start": 1748135331781, "status": "success", "end": 1748135561792, "result": { "size": 2801795072 } }, { "id": "1748135562458", "message": "clean-vm", "start": 1748135562458, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/58af91d1-94cf-cbe5-88cf-560e652741bf/20250524T221027Z.json", "actual": 2801795072, "expected": 2804644352 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135562570, "result": { "merge": true } } ], "end": 1748135562664 } ], "end": 1748135562665 }, { "data": { "type": "VM", "id": "6c9f0275-07bd-a124-e187-47f8389ed93c" }, "id": "1748135487646", "message": "backup VM", "start": 1748135487646, "status": "success", "tasks": [ { "id": "1748135488697", "message": "clean-vm", "start": 1748135488697, "status": "success", "end": 1748135499685, "result": { "merge": false } }, { "id": "1748135500529:0", "message": "clean-vm", "start": 1748135500529, "status": "success", "end": 1748135506654, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135507365 }, { "data": { "type": "VM", "id": "931aaab6-5f7a-d0ef-26a1-1f54063d0941" }, "id": "1748135507681", "message": "backup VM", "start": 1748135507681, "status": "success", "tasks": [ { "id": "1748135509790", "message": "clean-vm", "start": 1748135509790, "status": "success", "end": 1748135517538, "result": { "merge": false } }, { "id": "1748135518256", "message": "clean-vm", "start": 1748135518256, "status": "success", "end": 1748135523041, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135523539 }, { "data": { "type": "VM", "id": "a060d911-bf2b-61c5-56bf-11944af25c4a" }, "id": "1748135525305", "message": "backup VM", "start": 1748135525305, "status": "success", "tasks": [ { "id": "1748135527194", "message": "clean-vm", "start": 1748135527194, "status": "success", "end": 1748135534840, "result": { "merge": false } }, { "id": "1748135535382", "message": "clean-vm", "start": 1748135535382, "status": "success", "end": 1748135538329, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135538791 }, { "data": { "type": "VM", "id": "a2562d66-00c8-3955-ab43-6a5584044b98" }, "id": "1748135539187", "message": "backup VM", "start": 1748135539187, "status": "success", "tasks": [ { "id": "1748135561164", "message": "clean-vm", "start": 1748135561164, "status": "success", "end": 1748135562180, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135562446", "message": "export", "start": 1748135562446, "status": "success", "tasks": [ { "id": "1748135562453", "message": "transfer", "start": 1748135562453, "status": "success", "end": 1748135562527, "result": { "size": 0 } }, { "id": "1748135562801", "message": "clean-vm", "start": 1748135562801, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/a2562d66-00c8-3955-ab43-6a5584044b98/20250524T220635Z.json", "actual": 0, "expected": 134144 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135563247, "result": { "merge": true } } ], "end": 1748135563304 } ], "end": 1748135563304 }, { "data": { "type": "VM", "id": "b02697d4-23b5-598f-4def-d6b197bc1085" }, "id": "1748135562737", "message": "backup VM", "start": 1748135562737, "status": "success", "tasks": [ { "id": "1748135562853", "message": "clean-vm", "start": 1748135562853, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b02697d4-23b5-598f-4def-d6b197bc1085/20250522T161026Z.json", "actual": 11740651520, "expected": 11740652032 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135563349, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135563511", "message": "export", "start": 1748135563511, "status": "success", "tasks": [ { "id": "1748135563531", "message": "transfer", "start": 1748135563531, "status": "success", "end": 1748135898500, "result": { "size": 2713714688 } }, { "id": "1748135910743", "message": "clean-vm", "start": 1748135910743, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b02697d4-23b5-598f-4def-d6b197bc1085/20250524T220039Z.json", "actual": 2713714688, "expected": 2716501504 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135913811, "result": { "merge": true } } ], "end": 1748135915372 } ], "end": 1748135915372 }, { "data": { "type": "VM", "id": "b5d74e0b-388c-019a-6994-e174c9ca7a51" }, "id": "1748135563341", "message": "backup VM", "start": 1748135563341, "status": "success", "tasks": [ { "id": "1748135563363", "message": "clean-vm", "start": 1748135563363, "status": "success", "end": 1748135563857, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135565987", "message": "export", "start": 1748135565987, "status": "success", "tasks": [ { "id": "1748135565996", "message": "transfer", "start": 1748135565996, "status": "success", "end": 1748136473233, "result": { "size": 13337886720 } }, { "id": "1748136476298", "message": "clean-vm", "start": 1748136476298, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b5d74e0b-388c-019a-6994-e174c9ca7a51/20250524T220255Z.json", "actual": 13337886720, "expected": 13343341568 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748136477094, "result": { "merge": true } } ], "end": 1748136477313 } ], "end": 1748136477313 }, { "data": { "type": "VM", "id": "c49a5398-38db-4fcd-481b-30181a5503cc" }, "id": "1748135916054", "message": "backup VM", "start": 1748135916054, "status": "success", "tasks": [ { "id": "1748135920380", "message": "clean-vm", "start": 1748135920380, "status": "success", "end": 1748135930666, "result": { "merge": false } }, { "id": "1748135930919", "message": "clean-vm", "start": 1748135930919, "status": "success", "end": 1748135935805, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135936610 }, { "data": { "type": "VM", "id": "c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b" }, "id": "1748135937843", "message": "backup VM", "start": 1748135937843, "status": "success", "tasks": [ { "id": "1748135940204", "message": "clean-vm", "start": 1748135940204, "status": "success", "end": 1748135948613, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135951349", "message": "export", "start": 1748135951349, "status": "success", "tasks": [ { "id": "1748135951810", "message": "transfer", "start": 1748135951810, "status": "success", "end": 1748136066962, "result": { "size": 960495616 } }, { "id": "1748136083459", "message": "clean-vm", "start": 1748136083459, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b/20250524T220054Z.json", "actual": 960495616, "expected": 962891264 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748136093352, "result": { "merge": true } } ], "end": 1748136096139 } ], "end": 1748136096139 } ], "end": 1748136477317, "result": { "errno": -9, "code": "EBADF", "syscall": "scandir", "path": "/run/xo-server/mounts/5dba60dc-c4fc-4d49-9a1e-453522c0d8f7/xo-vm-backups/341a6155-908b-66ba-76ee-56a1d5163af1" } } -
I am checking it
Can you confirm that this a delta backup job and you are not using block. -
@florent The original backup is delta. Using NBD+CBT to a NFS storage.
Also get these frequently:
But nothing is running in parallel.
-
this error is known : this when a mirror backup start before the end of the asynchronoux cleaning of previous backup is not done
you can check the "merge backup synchronously" on the advanced setting of the first backup
-
@florent All my backups have this set! And this only happens with the new backup code, never with the old.
-
@florent Giving up on mirror backups. Switching to direct backups to remore storage.
At least until this works again as expected. -
At least I reproduce it locally , when going from a non block remote to an encrypted remote
Is your target remote encrypted ?
-
@florent Simple NFS, no encryption.
-
@manilx said in Our future backup code: test it!:
@florent Simple NFS, no encryption.
I have a potential fix here on the branch fix_backup
( https://github.com/vatesfr/xen-orchestra/pull/8635/files)open fix(disk-transform): backup ominibus fix #8635
-
@florent waiting for final master
-
@manilx said in Our future backup code: test it!:
@florent waiting for final master
this will reach master today, and probably be released in the may version of XO
-
So are we now at the point that Delta backups, Mirrors and CR backups are all working and considered stable? I have only been able to test delta backups with my home lab setup which have been working without missing a beat since the May 22nd fix for tasks not clearing. (MS-01 running XCP-NG, TrueNAS for storage)
We do use make use of CR jobs pretty extensively with the full XOA appliance for our DR site at work but because its full XOA and "production" i have not tested anything there. That said hoping for no nasty surprises with the may XO update if this is planned on being pushed to the stable XOA release!
-
@flakpyro Very good question. I really don't need the aggravation of any backup job breaking on our production site......
