Our future backup code: test it!
-
@florent Getting errors again:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1748134829731", "jobId": "b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6", "jobName": "Mirror Mani", "message": "backup", "scheduleId": "9e8eeb18-65bb-4e12-b1a1-4f7c0490565c", "start": 1748134829731, "status": "failure", "infos": [ { "data": { "vms": [ "0c8fd092-ab36-478e-df85-16b1a01e1ca8", "19242523-e70b-bf39-820d-5e322435012b", "1a65641e-0976-07a8-30f8-3cbd2e4bc63d", "1c596fc8-26a8-7d95-d004-1e694392fc72", "1d7614fc-65f7-4e1c-71f9-da638efd8b4d", "284b64bc-8684-eb3c-b32e-93e455f5854b", "294b867a-e241-fe7a-10b0-aef1de6f693d", "341a6155-908b-66ba-76ee-56a1d5163af1", "43a4d905-7d13-85b8-bed3-f6b805ff26ac", "48ca6a7e-7acf-74fd-4ad9-714315563652", "4d8cad25-b6e5-4d8f-5226-b5908b21a1b6", "52e64134-62e3-9682-4e3f-296a1198db4d", "58af91d1-94cf-cbe5-88cf-560e652741bf", "592e229c-76d7-c0f2-426c-4ef9e80810d0", "5f49e0a4-9397-3da5-e44a-0ab3632cd9f1", "6c9f0275-07bd-a124-e187-47f8389ed93c", "8d573273-49b1-6b84-73a6-42c3346add52", "8e76ba5c-e190-753b-75d6-e67feb1dd85c", "9017c283-51d0-8d22-d13b-328cc9fe4ff7", "903d61e0-c07f-8163-47ae-8e212147a353", "931aaab6-5f7a-d0ef-26a1-1f54063d0941", "9e0c8a72-d0fd-6632-6d57-94224149af05", "9f8088b6-9a5d-0ec5-24d5-118c23100a4b", "a060d911-bf2b-61c5-56bf-11944af25c4a", "a2562d66-00c8-3955-ab43-6a5584044b98", "ac14b70f-9958-7add-f833-64486feba4e7", "b02697d4-23b5-598f-4def-d6b197bc1085", "b5d74e0b-388c-019a-6994-e174c9ca7a51", "bcb4bcfc-c77b-2068-4bec-2826e8ece6be", "c21e8541-c5ea-840e-31aa-095789600230", "c49a5398-38db-4fcd-481b-30181a5503cc", "c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b", "cc6b00b9-16ec-1990-e44a-503b6e3921c3", "d7964354-aff6-c7e1-0067-52e8e866a965", "ef53c584-f6c2-3aa3-b4c0-802fb20683d9", "f262e69e-f345-bc52-c2dc-60eea7b84c9a", "fab6b6c0-7ce7-91da-fe60-acbcea4ca5be", "fe2dd0ea-f063-9aba-30d5-d2de508b091f" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "19242523-e70b-bf39-820d-5e322435012b" }, "id": "1748134830645", "message": "backup VM", "start": 1748134830645, "status": "success", "tasks": [ { "id": "1748134830822", "message": "clean-vm", "start": 1748134830822, "status": "success", "end": 1748134831309, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748134831397", "message": "export", "start": 1748134831397, "status": "success", "tasks": [ { "id": "1748134831413", "message": "transfer", "start": 1748134831413, "status": "success", "end": 1748134937238, "result": { "size": 710934528 } }, { "id": "1748134947620", "message": "clean-vm", "start": 1748134947620, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/19242523-e70b-bf39-820d-5e322435012b/20250524T220611Z.json", "actual": 710934528, "expected": 713224192 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748134949566, "result": { "merge": true } } ], "end": 1748134965928 } ], "end": 1748134965928 }, { "data": { "type": "VM", "id": "0c8fd092-ab36-478e-df85-16b1a01e1ca8" }, "id": "1748134830649", "message": "backup VM", "start": 1748134830649, "status": "interrupted", "tasks": [ { "id": "1748134830750", "message": "clean-vm", "start": 1748134830750, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/20250522T161036Z.json", "actual": 71360596992, "expected": 71360597504 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748134831421, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748134834838", "message": "export", "start": 1748134834838, "status": "success", "tasks": [ { "id": "1748134834899", "message": "transfer", "start": 1748134834899, "status": "success", "end": 1748135312431, "result": { "size": 4857004032 } }, { "id": "1748135312559", "message": "clean-vm", "start": 1748135312559, "status": "success", "warnings": [ { "data": { "parent": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45", "child": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45/20250524T220111Z.vhd" }, "message": "parent VHD is missing" }, { "data": { "backup": "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/20250524T220111Z.json", "missingVhds": [ "/xo-vm-backups/0c8fd092-ab36-478e-df85-16b1a01e1ca8/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/5cb71c46-ddf5-4058-b29c-3bb436816d45/20250524T220111Z.vhd" ] }, "message": "some VHDs linked to the backup are missing" } ], "end": 1748135312819, "result": { "merge": false } } ], "end": 1748135312823 } ], "warnings": [ { "data": { "attempt": 1, "error": "Expected values to be strictly equal:\n\n0 !== 512\n" }, "message": "Retry the VM mirror backup due to an error" } ] }, { "data": { "type": "VM", "id": "1c596fc8-26a8-7d95-d004-1e694392fc72" }, "id": "1748134966251", "message": "backup VM", "start": 1748134966251, "status": "success", "tasks": [ { "id": "1748134968074", "message": "clean-vm", "start": 1748134968074, "status": "success", "end": 1748134968813, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748134971622", "message": "export", "start": 1748134971622, "status": "success", "tasks": [ { "id": "1748134971997", "message": "transfer", "start": 1748134971997, "status": "success", "end": 1748135312426, "result": { "size": 3261071360 } }, { "id": "1748135314472", "message": "clean-vm", "start": 1748135314472, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/1c596fc8-26a8-7d95-d004-1e694392fc72/20250524T222347Z.json", "actual": 3261071360, "expected": 3264098816 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135314652, "result": { "merge": true } } ], "end": 1748135314707 } ], "end": 1748135314707 }, { "data": { "type": "VM", "id": "294b867a-e241-fe7a-10b0-aef1de6f693d" }, "id": "1748135312961", "message": "backup VM", "start": 1748135312961, "status": "interrupted", "tasks": [ { "id": "1748135313037", "message": "clean-vm", "start": 1748135313037, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/294b867a-e241-fe7a-10b0-aef1de6f693d/20250522T161022Z.json", "actual": 11216235520, "expected": 11216236032 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135313727, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135313903", "message": "export", "start": 1748135313903, "status": "failure", "tasks": [ { "id": "1748135313921", "message": "transfer", "start": 1748135313921, "status": "failure", "end": 1748135318829, "result": { "errno": -9, "code": "EBADF", "syscall": "read", "message": "EBADF: bad file descriptor, read", "name": "Error", "stack": "Error: EBADF: bad file descriptor, read\nFrom:\n at NfsHandler.addSyncStackTrace (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:21:26)\n at NfsHandler._read (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:160:43)\n at NfsHandler.read (/opt/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:259:17)\n at NfsHandler.read (/opt/xen-orchestra/node_modules/limit-concurrency-decorator/index.js:97:24)\n at VhdFile._read (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:126:55)\n at VhdFile.readBlock (/opt/xen-orchestra/packages/vhd-lib/Vhd/VhdFile.js:212:17)\n at RemoteVhd.readBlock (file:///opt/xen-orchestra/@xen-orchestra/backups/disks/RemoteVhd.mjs:144:38)\n at RemoteVhd.buildDiskBlockGenerator (file:///opt/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:67:28)" } }, { "id": "1748135318836", "message": "clean-vm", "start": 1748135318836, "status": "success", "end": 1748135318927, "result": { "merge": false } } ], "end": 1748135318948 } ], "warnings": [ { "data": { "attempt": 1, "error": "ENOENT: no such file or directory, open '/run/xo-server/mounts/337848e1-b52b-4e08-9bf0-ffce54de4276/xo-vm-backups/294b867a-e241-fe7a-10b0-aef1de6f693d/vdis/b7215ad7-5616-4620-a6fb-1c5b5b1bd3c6/c87b9871-9ce2-4b39-9fec-9dea2fac3220/20250524T220036Z.vhd'" }, "message": "Retry the VM mirror backup due to an error" } ] }, { "data": { "type": "VM", "id": "43a4d905-7d13-85b8-bed3-f6b805ff26ac" }, "id": "1748135318587", "message": "backup VM", "start": 1748135318587, "status": "success", "tasks": [ { "id": "1748135318860", "message": "clean-vm", "start": 1748135318860, "status": "success", "end": 1748135319739, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135319979", "message": "export", "start": 1748135319979, "status": "success", "tasks": [ { "id": "1748135319993", "message": "transfer", "start": 1748135319993, "status": "success", "end": 1748135458653, "result": { "size": 870318080 } }, { "id": "1748135473880", "message": "clean-vm", "start": 1748135473880, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/43a4d905-7d13-85b8-bed3-f6b805ff26ac/20250524T220144Z.json", "actual": 870318080, "expected": 872691712 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135483581, "result": { "merge": true } } ], "end": 1748135487189 } ], "end": 1748135487190 }, { "data": { "type": "VM", "id": "58af91d1-94cf-cbe5-88cf-560e652741bf" }, "id": "1748135319145", "message": "backup VM", "start": 1748135319145, "status": "success", "tasks": [ { "id": "1748135319356", "message": "clean-vm", "start": 1748135319356, "status": "success", "end": 1748135320011, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135331775", "message": "export", "start": 1748135331775, "status": "success", "tasks": [ { "id": "1748135331781", "message": "transfer", "start": 1748135331781, "status": "success", "end": 1748135561792, "result": { "size": 2801795072 } }, { "id": "1748135562458", "message": "clean-vm", "start": 1748135562458, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/58af91d1-94cf-cbe5-88cf-560e652741bf/20250524T221027Z.json", "actual": 2801795072, "expected": 2804644352 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135562570, "result": { "merge": true } } ], "end": 1748135562664 } ], "end": 1748135562665 }, { "data": { "type": "VM", "id": "6c9f0275-07bd-a124-e187-47f8389ed93c" }, "id": "1748135487646", "message": "backup VM", "start": 1748135487646, "status": "success", "tasks": [ { "id": "1748135488697", "message": "clean-vm", "start": 1748135488697, "status": "success", "end": 1748135499685, "result": { "merge": false } }, { "id": "1748135500529:0", "message": "clean-vm", "start": 1748135500529, "status": "success", "end": 1748135506654, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135507365 }, { "data": { "type": "VM", "id": "931aaab6-5f7a-d0ef-26a1-1f54063d0941" }, "id": "1748135507681", "message": "backup VM", "start": 1748135507681, "status": "success", "tasks": [ { "id": "1748135509790", "message": "clean-vm", "start": 1748135509790, "status": "success", "end": 1748135517538, "result": { "merge": false } }, { "id": "1748135518256", "message": "clean-vm", "start": 1748135518256, "status": "success", "end": 1748135523041, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135523539 }, { "data": { "type": "VM", "id": "a060d911-bf2b-61c5-56bf-11944af25c4a" }, "id": "1748135525305", "message": "backup VM", "start": 1748135525305, "status": "success", "tasks": [ { "id": "1748135527194", "message": "clean-vm", "start": 1748135527194, "status": "success", "end": 1748135534840, "result": { "merge": false } }, { "id": "1748135535382", "message": "clean-vm", "start": 1748135535382, "status": "success", "end": 1748135538329, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135538791 }, { "data": { "type": "VM", "id": "a2562d66-00c8-3955-ab43-6a5584044b98" }, "id": "1748135539187", "message": "backup VM", "start": 1748135539187, "status": "success", "tasks": [ { "id": "1748135561164", "message": "clean-vm", "start": 1748135561164, "status": "success", "end": 1748135562180, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135562446", "message": "export", "start": 1748135562446, "status": "success", "tasks": [ { "id": "1748135562453", "message": "transfer", "start": 1748135562453, "status": "success", "end": 1748135562527, "result": { "size": 0 } }, { "id": "1748135562801", "message": "clean-vm", "start": 1748135562801, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/a2562d66-00c8-3955-ab43-6a5584044b98/20250524T220635Z.json", "actual": 0, "expected": 134144 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135563247, "result": { "merge": true } } ], "end": 1748135563304 } ], "end": 1748135563304 }, { "data": { "type": "VM", "id": "b02697d4-23b5-598f-4def-d6b197bc1085" }, "id": "1748135562737", "message": "backup VM", "start": 1748135562737, "status": "success", "tasks": [ { "id": "1748135562853", "message": "clean-vm", "start": 1748135562853, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b02697d4-23b5-598f-4def-d6b197bc1085/20250522T161026Z.json", "actual": 11740651520, "expected": 11740652032 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135563349, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135563511", "message": "export", "start": 1748135563511, "status": "success", "tasks": [ { "id": "1748135563531", "message": "transfer", "start": 1748135563531, "status": "success", "end": 1748135898500, "result": { "size": 2713714688 } }, { "id": "1748135910743", "message": "clean-vm", "start": 1748135910743, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b02697d4-23b5-598f-4def-d6b197bc1085/20250524T220039Z.json", "actual": 2713714688, "expected": 2716501504 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748135913811, "result": { "merge": true } } ], "end": 1748135915372 } ], "end": 1748135915372 }, { "data": { "type": "VM", "id": "b5d74e0b-388c-019a-6994-e174c9ca7a51" }, "id": "1748135563341", "message": "backup VM", "start": 1748135563341, "status": "success", "tasks": [ { "id": "1748135563363", "message": "clean-vm", "start": 1748135563363, "status": "success", "end": 1748135563857, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135565987", "message": "export", "start": 1748135565987, "status": "success", "tasks": [ { "id": "1748135565996", "message": "transfer", "start": 1748135565996, "status": "success", "end": 1748136473233, "result": { "size": 13337886720 } }, { "id": "1748136476298", "message": "clean-vm", "start": 1748136476298, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b5d74e0b-388c-019a-6994-e174c9ca7a51/20250524T220255Z.json", "actual": 13337886720, "expected": 13343341568 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748136477094, "result": { "merge": true } } ], "end": 1748136477313 } ], "end": 1748136477313 }, { "data": { "type": "VM", "id": "c49a5398-38db-4fcd-481b-30181a5503cc" }, "id": "1748135916054", "message": "backup VM", "start": 1748135916054, "status": "success", "tasks": [ { "id": "1748135920380", "message": "clean-vm", "start": 1748135920380, "status": "success", "end": 1748135930666, "result": { "merge": false } }, { "id": "1748135930919", "message": "clean-vm", "start": 1748135930919, "status": "success", "end": 1748135935805, "result": { "merge": false } } ], "infos": [ { "message": "No new data to upload for this VM" } ], "end": 1748135936610 }, { "data": { "type": "VM", "id": "c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b" }, "id": "1748135937843", "message": "backup VM", "start": 1748135937843, "status": "success", "tasks": [ { "id": "1748135940204", "message": "clean-vm", "start": 1748135940204, "status": "success", "end": 1748135948613, "result": { "merge": false } }, { "data": { "id": "337848e1-b52b-4e08-9bf0-ffce54de4276", "type": "remote" }, "id": "1748135951349", "message": "export", "start": 1748135951349, "status": "success", "tasks": [ { "id": "1748135951810", "message": "transfer", "start": 1748135951810, "status": "success", "end": 1748136066962, "result": { "size": 960495616 } }, { "id": "1748136083459", "message": "clean-vm", "start": 1748136083459, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/c5fbe7d1-9f9c-0167-5ed2-1dff1104ef4b/20250524T220054Z.json", "actual": 960495616, "expected": 962891264 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1748136093352, "result": { "merge": true } } ], "end": 1748136096139 } ], "end": 1748136096139 } ], "end": 1748136477317, "result": { "errno": -9, "code": "EBADF", "syscall": "scandir", "path": "/run/xo-server/mounts/5dba60dc-c4fc-4d49-9a1e-453522c0d8f7/xo-vm-backups/341a6155-908b-66ba-76ee-56a1d5163af1" } } -
I am checking it
Can you confirm that this a delta backup job and you are not using block. -
@florent The original backup is delta. Using NBD+CBT to a NFS storage.
Also get these frequently:
But nothing is running in parallel.
-
this error is known : this when a mirror backup start before the end of the asynchronoux cleaning of previous backup is not done
you can check the "merge backup synchronously" on the advanced setting of the first backup
-
@florent All my backups have this set! And this only happens with the new backup code, never with the old.
-
@florent Giving up on mirror backups. Switching to direct backups to remore storage.
At least until this works again as expected. -
At least I reproduce it locally , when going from a non block remote to an encrypted remote
Is your target remote encrypted ?
-
@florent Simple NFS, no encryption.
-
@manilx said in Our future backup code: test it!:
@florent Simple NFS, no encryption.
I have a potential fix here on the branch fix_backup
( https://github.com/vatesfr/xen-orchestra/pull/8635/files)open fix(disk-transform): backup ominibus fix #8635
-
@florent waiting for final master
-
@manilx said in Our future backup code: test it!:
@florent waiting for final master
this will reach master today, and probably be released in the may version of XO
-
So are we now at the point that Delta backups, Mirrors and CR backups are all working and considered stable? I have only been able to test delta backups with my home lab setup which have been working without missing a beat since the May 22nd fix for tasks not clearing. (MS-01 running XCP-NG, TrueNAS for storage)
We do use make use of CR jobs pretty extensively with the full XOA appliance for our DR site at work but because its full XOA and "production" i have not tested anything there. That said hoping for no nasty surprises with the may XO update if this is planned on being pushed to the stable XOA release!
-
@flakpyro Very good question. I really don't need the aggravation of any backup job breaking on our production site......
-
@flakpyro the stable branch, is the stable xoa. The latest release is " it is tested, but on limited hardware". The "pull from master and use it" is on best effort, I wouldn't advise to use it on any critical use.
the april's release is stable and will be promoted to stable today.
the may's release probably won't be promoted to stable in june, given the massive changes. my guess is that the june's release maybe , if we deem it stable enough -
@Tristis-Oris have CR jobs started working for you with the latest master code?
@florent Understod! Our XOA appliance has been on the latest branch since we deployed it almost a year and a half ago. Though with all these changes i may consider dropping it down to stable for a bit as backups are pretty critical. We only use Delta jobs (Which seem to be working well) and Continuous replication jobs. (Which i have seen some issues with recently in this thread). -
@flakpyro nope. same error.
-
@olivierlambert @florent I still hit a single thread limit in the XO VM that makes backups CPU bound limiting performance as before.
-
@florent so testing a CR job with the latest XOA release. The job runs successfully and i can see the virtual disk on the remote SR however the replica does not appear in XOAs VM inventory view.
Edits:
They do appear in XCP-NG Center,and XO Lite
They do NOT appear to show up using xo-cli list-objects
This also appears to ONLY happen to Windows VMs. Maybe related to missing templates?VMs that have not yet even ran their daily replication job also seem affected. Replicas that ran last night and appeared in inventory before the update now now longer appear in the VM inventory list. This might be related to something else that was changed in the new XOA update?
-
@flakpyro Nice catch! I see the same problem.
@olivierlambert @florent Running XO source (master 6ed35). I see the Windows VMs in the CR backup job report, but they no longer appear on the destination host as a VM. Some have updated tools, others do not. It worked before... Non-windows VMs seems to be fine. These are all old VMs, so it's a new problem on existing systems.
-
@Andrew weird, never seen that.
too many tasks maybe?
