XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Restoring Full Backup Error: Stream_vdi.Invalid_checksum

    Scheduled Pinned Locked Moved Xen Orchestra
    10 Posts 3 Posters 893 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • lawrencesystemsL Offline
      lawrencesystems Ambassador
      last edited by

      I am running xo-server 5.103.1 xo-web 5.104.0 built from sources on Debian GNU/Linux bookworm/sid with all packages up to date. The XO instance has 6GB of memory and 16 CPU's assigned.

      I have several "Full Backup" jobs going to a TrueNAS server via an SMB remote and they are backing up successfully without any errors. I am not using the "Store backup as multiple data blocks" for this remote. While I was doing a test restore of one of the larger VM's which is about 220 GiB it failed and gave me the error below.

      I keep a few versions and the older backup worked so I was assuming that backup was somehow corrupted. I ran the "Full Backup" job again and it completed successfully then tried to restore and same error message but different "Block Ref:" number in the error.

      I have backed up and restored this VM from a NFS remote on the TrueNAS which is using the "Store backup as multiple data blocks" options and it is working just fine.

      I know lots of people are using SMB for the remotes so I am looking for some insights as where to look to figure out if this is an issue with using SMB for the remote with larger VM's or if this is just something broken in my XO instance.

      {"code":"IMPORT_ERROR","params":["INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ]"],"url":"https://192.168.3.95/import/?sr_id=OpaqueRef%3A8e6025c8-30e6-4744-a1da-9895a38d67be&session_id=OpaqueRef%3Ae5d922e1-ad47-4141-a3ab-51fc0b12b6a6&task_id=OpaqueRef%3Ab016656c-97d1-41a5-a855-69b9f2531cc7","task":{"uuid":"6c886135-4ecb-e723-d438-bda19a6a4568","name_label":"[XO] VM import","name_description":"","allowed_operations":[],"current_operations":{},"created":"20221017T11:02:32Z","finished":"20221017T11:12:55Z","status":"failure","resident_on":"OpaqueRef:ccfe0e03-5b7d-46ac-a8ef-35d63c9ba5e7","progress":1,"type":"<none/>","result":"","error_info":["IMPORT_ERROR","INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ]"],"other_config":{"object_creation":"complete"},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 1967))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"},"pool_master":{"uuid":"58e38fa8-c2d2-459d-8446-0cf0bbda626e","name_label":"Ryzen1-XCP-NG","name_description":"Don't worry, everything happens for a Ryzen...","memory_overhead":1763532800,"allowed_operations":["vm_migrate","provision","vm_resume","evacuate","vm_start"],"current_operations":{},"API_version_major":2,"API_version_minor":16,"API_version_vendor":"XenSource","API_version_vendor_implementation":{},"enabled":true,"software_version":{"product_version":"8.2.1","product_version_text":"8.2","product_version_text_short":"8.2","platform_name":"XCP","platform_version":"3.2.1","product_brand":"XCP-ng","build_number":"release/yangtze/master/58","hostname":"localhost","date":"2022-08-17","dbv":"0.0.1","xapi":"1.20","xen":"4.13.4-9.25.1","linux":"4.19.0+1","xencenter_min":"2.16","xencenter_max":"2.16","network_backend":"openvswitch","db_schema":"5.602"},"other_config":{"agent_start_time":"1665919758.","boot_time":"1665919723.","last_blob_sync_time":"1665228407.72","iscsi_iqn":"iqn.2022-09.com.example:166a7d3c","rpm_patch_installation_time":"1662506898.06"},"capabilities":["xen-3.0-x86_64","hvm-3.0-x86_32","hvm-3.0-x86_32p","hvm-3.0-x86_64",""],"cpu_configuration":{},"sched_policy":"credit","supported_bootloaders":["pygrub","eliloader"],"resident_VMs":["OpaqueRef:56141690-c36d-4ec0-9d99-bb5d8466f91f","OpaqueRef:8b312ac3-6180-4ba9-9c56-93d834df437a","OpaqueRef:8b3beb03-4a51-40f5-a851-efbabd9ff922","OpaqueRef:b95b1f28-95ee-4521-be90-37a5833fc032","OpaqueRef:fbe1523c-20a7-486d-93e6-b4a295373fd5","OpaqueRef:4d8a7b05-0451-4938-bdec-573b30118790","OpaqueRef:52e1c465-464c-4154-9ff2-21cb13539a7c"],"logging":{"syslog_destination":""},"PIFs":["OpaqueRef:f36e1f2e-c5df-4da9-86ab-ef5492f8f3c4","OpaqueRef:d918c7c8-94c4-4309-8932-011b6a852b2b","OpaqueRef:bc95e309-e2d8-45ba-9bef-e7d5ce6099b3","OpaqueRef:bc185fce-743f-4945-ab75-9e3662778240","OpaqueRef:ba11c9db-07a4-4ffb-bc7e-7bc73446d31e","OpaqueRef:affcdb28-84b9-4f0c-b209-e669b5033763","OpaqueRef:a1242f16-f538-429e-8024-11055c7d7c6f","OpaqueRef:a10ee21a-9f1a-4c26-874d-b2ff250e5779","OpaqueRef:9ef07c9b-112b-4f86-a333-83160d1e61c8","OpaqueRef:9d86120a-1cae-4202-b987-a1a739cc882a","OpaqueRef:926579cb-48f7-408b-8679-8cb54b138edf","OpaqueRef:8d460590-b160-4d7c-b93e-f61531d7baa5","OpaqueRef:7e16a323-60b6-4cb4-afa5-721b4562257c","OpaqueRef:69ba1431-1a93-4fec-809e-3ce616a175f9","OpaqueRef:67027b3d-5729-4f79-a167-d2c65b688e10","OpaqueRef:60928c09-5fc5-45d7-a449-d2055789d13f","OpaqueRef:5e9a179b-085b-4331-bcde-e4b9cc1a00cb","OpaqueRef:53e5135c-09fb-4611-a637-8b44b80da013","OpaqueRef:102ef69e-1635-4369-b5d6-cccec168d67a"],"suspend_image_sr":"OpaqueRef:8e796368-0836-4da3-80f3-99f1f8783fb9","crash_dump_sr":"OpaqueRef:8e796368-0836-4da3-80f3-99f1f8783fb9","crashdumps":["OpaqueRef:fc7e5999-d0b6-403c-92a1-c7db8edf7a79"],"patches":[],"updates":[],"PBDs":["OpaqueRef:f2430beb-2e71-4804-8eea-b2431a8161ec","OpaqueRef:cc9c8681-6418-4f58-a61e-c8faf4c77a48","OpaqueRef:baa91a3c-1437-462b-8a9b-5f5e267faadb","OpaqueRef:b13e92f2-1c64-4280-b246-b0a09d0bf946","OpaqueRef:977a5ce1-a1f5-460c-813f-b8cc022e3bae","OpaqueRef:02d876af-68c3-4bae-b28e-1e32f53ec73d"],"host_CPUs":["OpaqueRef:c0ff1496-07f3-4d18-904d-421ba87266c2","OpaqueRef:29a31390-4e22-46cb-ba3a-b6c18927154b","OpaqueRef:45e20ffd-e8ba-4d70-8f3d-e11901dd3ded","OpaqueRef:42294c04-e800-4f88-8a14-ad90810ff61d","OpaqueRef:b42d634c-82e5-4067-8faf-776cd8c0c732","OpaqueRef:e2baba85-4774-41be-8103-750d34e3c3e6","OpaqueRef:ad47e14b-d7b7-40d2-8975-9cc826253fc3","OpaqueRef:5df2c60a-30dc-4ca9-88ce-59a679795229","OpaqueRef:e8ea3968-a77c-4efd-99dd-037eae4b69e4","OpaqueRef:a92f9bae-db6b-4a0c-87ec-8c740515d51d","OpaqueRef:227bda7e-a0aa-43e9-b429-7a21286e6ebd","OpaqueRef:c1e92cce-27eb-4cdd-bac4-216956d03135","OpaqueRef:e12a945a-ac2b-476e-a728-b2055b14f468","OpaqueRef:2c6c7c76-6aeb-474d-a799-32f3a06f2063","OpaqueRef:9c4d92b3-1107-457d-bf95-d886714272bc","OpaqueRef:4211a3cc-d5b2-4f6c-8bca-99aedbb77769","OpaqueRef:3607079b-2f1e-4cbc-a59e-a8c79fc6a74e","OpaqueRef:30254f21-7962-4059-82e9-5802fb36c0e0","OpaqueRef:6f6b3ae0-c7d5-46d6-9df9-804c90706572","OpaqueRef:464c61a9-96aa-4602-80dc-bf67360f14cd","OpaqueRef:21bc1460-a25f-428d-a6a4-d85b8a635a32","OpaqueRef:e31f75a8-8b7f-4b10-945c-aa032527221a","OpaqueRef:98f497c1-1938-49bf-b8c6-62a9a481613b","OpaqueRef:52f42239-4d0f-4057-9fe8-86e58db662ab"],"cpu_info":{"cpu_count":"24","socket_count":"1","vendor":"AuthenticAMD","speed":"3693.074","modelname":"AMD Ryzen 9 5900X 12-Core Processor","family":"25","model":"33","stepping":"2","flags":"fpu de tsc msr pae mce cx8 apic mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy abm sse4a misalignsse 3dnowprefetch bpext ibpb vmmcall fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat vaes vpclmulqdq rdpid","features_pv":"1789c3f5-f6f83203-2bd1cbf5-040001e3-00000007-218c0329-00400600-00000000-20001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm":"178bfbff-f6f83203-2fd3fbff-040085f7-0000000f-219c07a9-0040060c-00000000-310ed005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm_host":"178bfbff-f6f83203-2e500800-040001f7-0000000f-219c07a9-0040060c-00000000-310ed005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_pv_host":"1789c3f5-f6f83203-2a500800-040001e3-00000007-218c0329-00400600-00000000-20001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"},"hostname":"ryzen1-xcp-ng","address":"192.168.3.94","metrics":"OpaqueRef:286451e9-e3a3-4185-92a4-21cc83e11a6c","license_params":{"restrict_vswitch_controller":"false","restrict_lab":"false","restrict_stage":"false","restrict_storagelink":"false","restrict_storagelink_site_recovery":"false","restrict_web_selfservice":"false","restrict_web_selfservice_manager":"false","restrict_hotfix_apply":"false","restrict_export_resource_data":"false","restrict_read_caching":"false","restrict_cifs":"false","restrict_health_check":"false","restrict_xcm":"false","restrict_vm_memory_introspection":"false","restrict_batch_hotfix_apply":"false","restrict_management_on_vlan":"false","restrict_ws_proxy":"false","restrict_vlan":"false","restrict_qos":"false","restrict_pool_attached_storage":"false","restrict_netapp":"false","restrict_equalogic":"false","restrict_pooling":"false","enable_xha":"true","restrict_marathon":"false","restrict_email_alerting":"false","restrict_historical_performance":"false","restrict_wlb":"false","restrict_rbac":"false","restrict_dmc":"false","restrict_checkpoint":"false","restrict_cpu_masking":"false","restrict_connection":"false","platform_filter":"false","regular_nag_dialog":"false","restrict_vmpr":"false","restrict_vmss":"false","restrict_intellicache":"false","restrict_gpu":"false","restrict_dr":"false","restrict_vif_locking":"false","restrict_storage_xen_motion":"false","restrict_vgpu":"false","restrict_integrated_gpu_passthrough":"false","restrict_vss":"false","restrict_guest_agent_auto_update":"false","restrict_pci_device_for_auto_update":"false","restrict_xen_motion":"false","restrict_guest_ip_setting":"false","restrict_ad":"false","restrict_nested_virt":"false","restrict_live_patching":"false","restrict_set_vcpus_number_live":"false","restrict_pvs_proxy":"false","restrict_igmp_snooping":"false","restrict_rpu":"false","restrict_pool_size":"false","restrict_cbt":"false","restrict_usb_passthrough":"false","restrict_network_sriov":"false","restrict_corosync":"true","restrict_zstd_export":"false","restrict_pool_secret_rotation":"false"},"ha_statefiles":[],"ha_network_peers":[],"blobs":{},"tags":[],"external_auth_type":"","external_auth_service_name":"","external_auth_configuration":{},"edition":"xcp-ng","license_server":{"address":"localhost","port":"27000"},"bios_strings":{"bios-vendor":"American Megatrends International, LLC.","bios-version":"T1.29b","system-manufacturer":"To Be Filled By O.E.M.","system-product-name":"X570D4U","system-version":"To Be Filled By O.E.M.","system-serial-number":"To Be Filled By O.E.M.","baseboard-manufacturer":"ASRockRack","baseboard-product-name":"X570D4U","baseboard-version":"","baseboard-serial-number":"T80-F3001300401","oem-1":"Xen","oem-2":"MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d","oem-3":"Default string","hp-rombios":""},"power_on_mode":"","power_on_config":{},"local_cache_sr":"OpaqueRef:8e796368-0836-4da3-80f3-99f1f8783fb9","chipset_info":{"iommu":"false"},"PCIs":["OpaqueRef:d57d7375-af85-4914-b62e-51223bb8506b","OpaqueRef:d1c33253-b62c-47b8-97ec-39ed7339d930","OpaqueRef:c8dfa8f6-26b1-4eef-89b6-9cbb5bef1df3","OpaqueRef:c78f15c1-69bf-46d1-b482-a08f7f1f039c","OpaqueRef:9db62e9d-fcc0-4abf-9f2e-b6e684f9b43e","OpaqueRef:56f44afe-87b4-4d10-a3dc-dc0e0e72cb32","OpaqueRef:3a5b38b2-5573-4f19-b3ce-649f5ccc968e","OpaqueRef:2fd2ac48-6334-4a70-b058-0e8f632e5aba","OpaqueRef:01302328-a19c-471c-9a1a-83f651a1662b"],"PGPUs":["OpaqueRef:3f0a5ebf-7c4a-4082-81ce-a2e375b3a411"],"PUSBs":[],"ssl_legacy":false,"guest_VCPUs_params":{},"display":"enabled","virtual_hardware_platform_versions":[0,1,2],"control_domain":"OpaqueRef:52e1c465-464c-4154-9ff2-21cb13539a7c","updates_requiring_reboot":[],"features":[],"iscsi_iqn":"iqn.2022-09.com.example:166a7d3c","multipathing":false,"uefi_certificates":"","certificates":[],"editions":["xcp-ng"]},"SR":{"uuid":"6ff9113c-3531-9e2a-66ba-8fc5be69eee7","name_label":"Ryzen2 Local Storage","name_description":"","allowed_operations":["vdi_enable_cbt","vdi_list_changed_blocks","unplug","plug","pbd_create","vdi_disable_cbt","update","pbd_destroy","vdi_resize","vdi_clone","vdi_data_destroy","scan","vdi_snapshot","vdi_mirror","vdi_create","vdi_destroy","vdi_set_on_boot"],"current_operations":{},"VDIs":["OpaqueRef:dc771123-6833-4c3b-9199-a42da7f5ff5b","OpaqueRef:d8a630ed-ee56-4109-b9bd-cbe3c1b5f405","OpaqueRef:c4defeab-e0c5-4e3d-8cdb-08d71ac0acb6","OpaqueRef:78fe3d0b-effd-4620-abfd-162877c8b428"],"PBDs":["OpaqueRef:e6d92534-41c9-4d62-9dc7-63897568deba"],"virtual_allocation":144955146240,"physical_utilisation":49125265408,"physical_size":1924991709184,"type":"ext","content_type":"user","shared":false,"other_config":{"i18n-original-value-name_label":"Local storage","i18n-key":"local-storage"},"tags":[],"sm_config":{"devserial":""},"blobs":{},"local_cache_enabled":true,"introduced_by":"OpaqueRef:NULL","clustered":false,"is_tools_sr":false},"message":"IMPORT_ERROR(INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ])","name":"XapiError","stack":"XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ])\n    at Function.wrap (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/_XapiError.js:16:12)\n    at _default (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/_getTaskResult.js:11:29)\n    at Xapi._addRecordToCache (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:954:24)\n    at forEach (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:988:14)\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:978:12)\n    at Xapi._watchEvents (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:1144:14)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:442:9)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"}
      
      florentF 1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Ping florent on this 🙂

        1 Reply Last reply Reply Quote 1
        • florentF Offline
          florent Vates 🪐 XO Team @lawrencesystems
          last edited by florent

          hi lawrencesystems,

          is there a way to give me access to your XO and look at the files on the samba ?

          Full backups ( as in the *.xva files ) are untouched by vhd directory.

          Regards,

          Florent

          lawrencesystemsL 1 Reply Last reply Reply Quote 0
          • lawrencesystemsL Offline
            lawrencesystems Ambassador @florent
            last edited by

            I am going to be out of town until next week, but I can keep the backups so we can arrange a time to test it.

            I have done a few more tests, I setup a Synology using SMB and it backs up and restores just fine. I also moved the existing backups over to an NFS share and that previously created backup gives the same Stream_vdi.Invalid_checksum error with the same block number on restore.

            florentF 1 Reply Last reply Reply Quote 0
            • florentF Offline
              florent Vates 🪐 XO Team @lawrencesystems
              last edited by

              lawrencesystems we don't do any magic on the xva (yet) . It's probably an error coming from the host, and I wuld like to investigate if possible. It's great it you can keep the files.

              Do you use encryption ?

              lawrencesystemsL 1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Yeah, sounds like a somehow corrupted XVA that can't be imported. The checksum computation fails on the host. So likely the XVA is corrupted, alternatively a transfer issue corrupting the file, but I'm less sure about this 🤔

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  What could be interesting: importing the XVA with xe CLI and see if the result is the same.

                  1 Reply Last reply Reply Quote 1
                  • lawrencesystemsL Offline
                    lawrencesystems Ambassador @florent
                    last edited by

                    florent

                    No encryption on this remote. And if I have time before I leave I will test importing the XVA with CLI and see what error I get.

                    lawrencesystemsL 1 Reply Last reply Reply Quote 1
                    • lawrencesystemsL Offline
                      lawrencesystems Ambassador @lawrencesystems
                      last edited by

                      I really think there is some bug in the TrueNAS Core SMB stack that I am fighting here because this issue
                      https://xcp-ng.org/forum/topic/6148/delta-backup-showing-success-no-delta-saved?_=1666089968101
                      I had reported before was only happening on TrueNAS Core via SMB. Also this is solved sine XO that extra integrity checking feature was added to the delta backups.

                      florentF 1 Reply Last reply Reply Quote 0
                      • florentF Offline
                        florent Vates 🪐 XO Team @lawrencesystems
                        last edited by

                        lawrencesystems We added the healthcheck for full backup (here : https://github.com/vatesfr/xen-orchestra/pull/6401 )
                        Also we removed our js smb implementation ( https://github.com/vatesfr/xen-orchestra/commit/b50e95802c6a876d4e6ffd0c62c209a187b1bfb2 )

                        to try to circumvent the strange samba comportment

                        fbeauchamp opened this pull request in vatesfr/xen-orchestra

                        closed fix(backups): add healthcheck to full backup #6401

                        0 julien-f committed to vatesfr/xen-orchestra
                        feat(fs): remove JS based SMB handler
                        
                        It's not well tested nor maintained.
                        1 Reply Last reply Reply Quote 1
                        • ajpri1998A ajpri1998 referenced this topic on
                        • First post
                          Last post