Mirror failed
-
I was trying to migrate a vm between two xcp-ng hosts and i had an error.
Mirror failed.....vm.migrate { "vm": "9e4fcc69-e775-bea9-b594-f240349f51c9", "mapVifsNetworks": { "98153036-d104-153d-ccfc-bb42bc5b1f29": "4031637d-4e28-1283-d507-3309ca111d21" }, "migrationNetwork": "02c65c2f-5e03-6f57-bf24-f6c4e9e37ea1", "sr": "e9c7c5ea-764a-5423-f9fd-8a8d65a8b762", "targetHost": "77e49410-ce48-4cba-bc5c-64e5e0e95854" } { "code": "MIRROR_FAILED", "params": [ "OpaqueRef:4b19ca12-3786-4623-abd4-2b504575c52b" ], "task": { "uuid": "cf203270-b0eb-7a57-5c03-76fc00c49b86", "name_label": "Async.VM.migrate_send", "name_description": "", "allowed_operations": [], "current_operations": { "DummyRef:|9ca67190-904c-4dd8-9312-69c630720397|task.cancel": "cancel" }, "created": "20250118T23:32:18Z", "finished": "20250119T12:32:42Z", "status": "failure", "resident_on": "OpaqueRef:49f0db32-a775-4d37-83a0-b20a996bbb9d", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "MIRROR_FAILED", "OpaqueRef:4b19ca12-3786-4623-abd4-2b504575c52b" ], "other_config": { "mirror_failed": "dfb3cae3-395d-4ab1-ad32-96316da874a7" }, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1556))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "MIRROR_FAILED(OpaqueRef:4b19ca12-3786-4623-abd4-2b504575c52b)", "name": "XapiError", "stack": "XapiError: MIRROR_FAILED(OpaqueRef:4b19ca12-3786-4623-abd4-2b504575c52b) at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/_XapiError.mjs:16:12) at default (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/_getTaskResult.mjs:13:29) at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1068:24) at file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1102:14 at Array.forEach (<anonymous>) at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1092:12) at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1265:14)" }
can anyone help me pinpointing the issue and fixing it?
Thanks in advance -
When you do a live migration, the blocks are mirrored on both sides (because you continue to write in the VM, so those blocks has to be both on source and destination, otherwise if the migration failed you would lose your VM).
If you write more blocks in the VM than mirroring them, the live migration will fail.
-
@olivierlambert so it is recommended to migrate a vm while being powered off?
-
It depends of so many factors than there's no universal answer. If you have this error message, you might to reduce the number of write operations (ie disabling a busy service in the VM). Warm migration might be also interesting.
-
@olivierlambert
Tried with vm powered off and again the same results.
How can i check what was the error?{ "vm": "9e4fcc69-e775-bea9-b594-f240349f51c9", "mapVifsNetworks": { "98153036-d104-153d-ccfc-bb42bc5b1f29 ": "4031637d-4e28-1283-d507-3309ca111d21 " }, "migrationNetwork": "02c65c2f-5e03-6f57-bf24-f6c4e9e37ea1 ", "sr": "5308a7b4-4a93-ec6a-c7c2-5deae25da964", "targetHost": "77e49410-ce48-4cba-bc5c-64e5e0e95854" } { "code": "MIRROR_FAILED", "params": [ "OpaqueRef:67ec3260-50c1-43f4-9eef-3faa7f5c5963 " ], "task": { "uuid": "c326ee2d-8aa3-b6e1-f855-d397bafbf646 ", "name_label": "Async.VM.migrate_send", "name_description": "", "allowed_operations": [], "current_operations": { "DummyRef:|9b9c2df8-7a6d-4647-93a2-41939e9ce5dc |task.cancel": "cancel" }, "created": "20250119T14:41:54Z", "finished": "20250120T06:29:55Z", "status": "failure", "resident_on": "OpaqueRef:49f0db32-a775-4d37-83a0-b20a996bbb9d", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "MIRROR_FAILED", "OpaqueRef:67ec3260-50c1-43f4-9eef-3faa7f5c5963 " ], "other_config": { "mirror_failed": "447ff17a-82bc-4d49-a177-02f9086e9a04 " }, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1556))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "MIRROR_FAILED(OpaqueRef:67ec3260-50c1-43f4-9eef-3faa7f5c5963 )", "name": "XapiError", "stack": "XapiError: MIRROR_FAILED(OpaqueRef:67ec3260-50c1-43f4-9eef-3faa7f5c5963 ) at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/_XapiError.mjs:16:12) at default (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/_getTaskResult.mjs:13:29) at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1068:24) at file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1102:14 at Array.forEach (<anonymous>) at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1092:12) at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202501180626/packages/xen-api/index.mjs:1265:14)" }
-
i started searching logs.
and i have found at xensource.log something about a uuid that is missing from a db something.Jan 20 08:29:44 r5700g xapi: [debug||26 |sm_events D:4663d9557151|storage_access] sm event on mirror: 7df83df8-cec3-85ae-fe89-e046962e44fc/447ff17a-82bc-4d49-a177-02f9086e9a04 Jan 20 08:29:44 r5700g xapi: [debug||417814 ||storage_access] Received update: ["Mirror","7df83df8-cec3-85ae-fe89-e046962e44fc/447ff17a-82bc-4d49-a177-02f9086e9a04"] Jan 20 08:29:44 r5700g xapi: [debug||417814 ||storage_access] Calling UPDATES.get Async.VM.migrate_send R:5802f743fd80 5819 30 Jan 20 08:29:44 r5700g xapi: [ info||26 |sm_events D:4663d9557151|storage_impl] DATA.MIRROR.stat dbg:sm_events D:4663d9557151 id:7df83df8-cec3-85ae-fe89-e046962e44fc/447ff17a-82bc-4d49-a177-02f9086e9a04 Jan 20 08:29:44 r5700g xapi: [debug||26 |sm_events D:4663d9557151|storage_access] Mirror 7df83df8-cec3-85ae-fe89-e046962e44fc/447ff17a-82bc-4d49-a177-02f9086e9a04 has failed Jan 20 08:29:44 r5700g xapi: [debug||26 |sm_events D:4663d9557151|storage_access] Mirror associated with task: OpaqueRef:5802f743-fd80-428b-b958-54a3356e0d45 Jan 20 08:29:44 r5700g xapi: [debug||538435 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.slave_login D:60aac40fd05a created by task D:4663d9557151 Jan 20 08:29:44 r5700g xapi: [ info||538435 /var/lib/xcp/xapi|session.slave_login D:d6baad700c72|xapi_session] Session.create trackid=0a8831f8101cac2c3ea301dfc55c7287 pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Jan 20 08:29:44 r5700g xapi: [debug||538436 /var/lib/xcp/xapi||api_effect] VDI.remove_from_sm_config Jan 20 08:29:44 r5700g xapi: [debug||538437 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:abede6dff903 created by task D:d6baad700c72 Jan 20 08:29:44 r5700g xapi: [debug||538439 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:task.cancel D:a952ceb453f4 created by task D:4663d9557151 Jan 20 08:29:44 r5700g xapi: [ info||538439 /var/lib/xcp/xapi|task.cancel D:9b9c2df87a6d|storage_access] storage_access: TASK.cancel 64 Jan 20 08:29:44 r5700g xapi: [debug||538440 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:f8b332f41f0d created by task D:4663d9557151 Jan 20 08:29:44 r5700g xapi: [ info||538440 /var/lib/xcp/xapi|session.logout D:2413f5ba515c|xapi_session] Session.destroy trackid=0a8831f8101cac2c3ea301dfc55c7287 Jan 20 08:29:44 r5700g xapi: [error||537909 ||sparse_dd_wrapper] Failure from sparse_dd: Using protocol: nbd\x0A raising End_of_file Jan 20 08:29:44 r5700g xapi: [ info||537721 |Async.VM.migrate_send R:5802f743fd80|task_server] Task 64 has been cancelled: raising Cancelled exception Jan 20 08:29:44 r5700g xapi: [ info||538444 HTTP 127.0.0.1->:::80|session.slave_login D:8701fa8c7182|xapi_session] Session.create trackid=01b5ed95bc6e2739654f111dda914de8 pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Jan 20 08:29:44 r5700g xapi: [debug||538445 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:9b894b8793d5 created by task D:8701fa8c7182 Jan 20 08:29:44 r5700g xapi: [ info||538442 HTTP 127.0.0.1->:::80|Querying services D:c11c1df69605|storage_impl] VDI.deactivate dbg:Async.VM.migrate_send R:5802f743fd80 dp:7a3bc730-ff9f-8330-6703-a4262b45a87a sr:7df83df8-cec3-85ae-fe89-e046962e44fc vdi:a36e4cfc-0752-4e9a-a406-3ca2591842d4 vm:0 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] VDI.get_by_uuid D:d32e2301995c failed with exception Db_exn.Read_missing_uuid("VDI", "", "OLD_447ff17a-82bc-4d49-a177-02f9086e9a04") Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] Raised Db_exn.Read_missing_uuid("VDI", "", "OLD_447ff17a-82bc-4d49-a177-02f9086e9a04") Jan 20 08:29:44 r5700g xapi: [debug||538447 ||dummytaskhelper] task VDI.deactivate D:9d7c5e51800d created by task O:Async.VM.mig Jan 20 08:29:44 r5700g xapi: [debug||538447 |VDI.deactivate D:9d7c5e51800d|dummytaskhelper] task VDI.deactivate D:cfe816796d8c created by task O:Async.VM.mig Jan 20 08:29:44 r5700g xapi: [debug||538447 ||sm] SM zfs vdi_deactivate sr=OpaqueRef:8a3e83cf-edcd-4245-bae3-c7e9a1b82f52 vdi=OpaqueRef:18bc6036-965a-4b2f-916d-0e63af3b4c47 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 1/9 xapi Raised at file ocaml/database/db_cache_impl.ml, line 237 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 2/9 xapi Called from file ocaml/xapi/db_actions.ml, line 14236 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 3/9 xapi Called from file ocaml/xapi/rbac.ml, line 197 Jan 20 08:29:44 r5700g xapi: [ info||538447 |sm_exec D:cb10c59db6fb|xapi_session] Session.create trackid=649de188b31f824f116654cb9e48b1f1 pool=false uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 4/9 xapi Called from file ocaml/xapi/rbac.ml, line 205 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 5/9 xapi Called from file ocaml/xapi/server_helpers.ml, line 92 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 6/9 xapi Called from file ocaml/xapi/server_helpers.ml, line 113 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 7/9 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 8/9 xapi Called from file map.ml, line 135 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] 9/9 xapi Called from file src/sexp_conv.ml, line 156 Jan 20 08:29:44 r5700g xapi: [error||538446 /var/lib/xcp/xapi||backtrace] Jan 20 08:29:44 r5700g xapi: [debug||538448 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:862d88832033 created by task D:cb10c59db6fb Jan 20 08:29:44 r5700g xapi: [debug||538449 /var/lib/xcp/xapi||api_effect] VDI.remove_from_other_config Jan 20 08:29:44 r5700g xapi: [ info||538451 |Async.SR.update R:b8c9a436a67e|dispatcher] spawning a new thread to handle the current task (trackid=7292f93df5ef9dcea30043b9e9ee9d0d) Jan 20 08:29:44 r5700g xapi: [debug||538451 |Async.SR.update R:b8c9a436a67e|audit] SR.update: SR = '7df83df8-cec3-85ae-fe89-e046962e44fc (zfs8tbN)' Jan 20 08:29:44 r5700g xapi: [ info||538451 |Async.SR.update R:b8c9a436a67e|storage_impl] SR.stat dbg:OpaqueRef:b8c9a436-a67e-44f8-baf5-2c57507b5fcf sr:7df83df8-cec3-85ae-fe89-e046962e44fc Jan 20 08:29:44 r5700g xapi: [debug||538453 ||dummytaskhelper] task SR.stat D:1b9aca2ed792 created by task R:b8c9a436a67e Jan 20 08:29:44 r5700g xapi: [debug||538453 |SR.stat D:1b9aca2ed792|sm] SM zfs sr_update sr=OpaqueRef:8a3e83cf-edcd-4245-bae3-c7e9a1b82f52 Jan 20 08:29:44 r5700g xapi: [ info||538453 |sm_exec D:785e90c63116|xapi_session] Session.create trackid=da9f0ee8b310ded041f35c0f3a2077d3 pool=false uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Jan 20 08:29:44 r5700g xapi: [debug||538454 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:e0adb632a6e7 created by task D:785e90c63116 Jan 20 08:29:44 r5700g xapi: [debug||538455 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:host.get_other_config D:d96016a921ae created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538457 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:host.get_other_config D:4051952cabd6 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538456 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_other_config D:08532c8dce21 created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538458 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_other_config D:131315f131e8 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538459 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_virtual_allocation D:3372be18b899 created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538460 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_by_uuid D:26c5cf78e11a created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538461 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.set_virtual_allocation D:87f1d70b5c67 created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538462 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.set_physical_size D:406d9faba266 created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538463 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.set_physical_utilisation D:693779743b34 created by task D:1b9aca2ed792 Jan 20 08:29:44 r5700g xapi: [debug||538464 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.get_by_uuid D:6bd34c8274fb created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538465 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.get_sm_config D:f023160b240c created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [ info||538453 |sm_exec D:785e90c63116|xapi_session] Session.destroy trackid=da9f0ee8b310ded041f35c0f3a2077d3 Jan 20 08:29:44 r5700g xapi: [debug||538468 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.get_by_uuid D:004985ae55e3 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538469 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.get_SR D:453f5a0a6250 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538470 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_uuid D:ca13ec8d52d6 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538471 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_by_uuid D:0e572ba88be3 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538472 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.get_type D:c59d1842ed93 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538473 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SM.get_all_records_where D:647e1bf6220d created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538474 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SM.get_driver_filename D:cb57b02c8a4a created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538475 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VM.get_all_records_where D:2b914e27a456 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538476 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:PBD.get_all_records_where D:5e8bf1ad7394 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538477 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:PBD.get_device_config D:88a9dc144eba created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [ info||538480 /var/lib/xcp/xapi|session.login_with_password D:3bd414ecb223|xapi_session] Session.create trackid=8d3463a90979acea8829effb2ccc222b pool=false uname=root originator=SM is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Jan 20 08:29:44 r5700g xapi: [debug||538481 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:fff533196eb7 created by task D:3bd414ecb223 Jan 20 08:29:44 r5700g xapi: [ info||538489 /var/lib/xcp/xapi|session.logout D:73e6f6349afa|xapi_session] Session.destroy trackid=8d3463a90979acea8829effb2ccc222b Jan 20 08:29:44 r5700g xapi: [debug||538490 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.get_by_uuid D:3978732c426b created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538491 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:host.get_by_uuid D:f5698a8adb36 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538492 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.get_sm_config D:c6032cef5383 created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538493 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VDI.remove_from_sm_config D:c124551af92c created by task D:9d7c5e51800d Jan 20 08:29:44 r5700g xapi: [debug||538493 /var/lib/xcp/xapi||api_effect] VDI.remove_from_sm_config Jan 20 08:29:44 r5700g xapi: [ info||538447 |sm_exec D:cb10c59db6fb|xapi_session] Session.destroy trackid=649de188b31f824f116654cb9e48b1f1
-
@AlexanderK said in Mirror failed:
"OLD_447ff17a-82bc-4d49-a177-02f9086e9a04
That's the missing bit reported by XAPI/SM, indeed.
-
@olivierlambert said in Mirror failed:
@AlexanderK said in Mirror failed:
"OLD_447ff17a-82bc-4d49-a177-02f9086e9a04
That's the missing bit reported by XAPI/SM, indeed.
Does this mean it is a bug?