Restoring Full Backup Error: Stream_vdi.Invalid_checksum

lawrencesystems

I am running xo-server 5.103.1 xo-web 5.104.0 built from sources on Debian GNU/Linux bookworm/sid with all packages up to date. The XO instance has 6GB of memory and 16 CPU's assigned.

I have several "Full Backup" jobs going to a TrueNAS server via an SMB remote and they are backing up successfully without any errors. I am not using the "Store backup as multiple data blocks" for this remote. While I was doing a test restore of one of the larger VM's which is about 220 GiB it failed and gave me the error below.

I keep a few versions and the older backup worked so I was assuming that backup was somehow corrupted. I ran the "Full Backup" job again and it completed successfully then tried to restore and same error message but different "Block Ref:" number in the error.

I have backed up and restored this VM from a NFS remote on the TrueNAS which is using the "Store backup as multiple data blocks" options and it is working just fine.

I know lots of people are using SMB for the remotes so I am looking for some insights as where to look to figure out if this is an issue with using SMB for the remote with larger VM's or if this is just something broken in my XO instance.

{"code":"IMPORT_ERROR","params":["INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ]"],"url":"https://192.168.3.95/import/?sr_id=OpaqueRef%3A8e6025c8-30e6-4744-a1da-9895a38d67be&session_id=OpaqueRef%3Ae5d922e1-ad47-4141-a3ab-51fc0b12b6a6&task_id=OpaqueRef%3Ab016656c-97d1-41a5-a855-69b9f2531cc7","task":{"uuid":"6c886135-4ecb-e723-d438-bda19a6a4568","name_label":"[XO] VM import","name_description":"","allowed_operations":[],"current_operations":{},"created":"20221017T11:02:32Z","finished":"20221017T11:12:55Z","status":"failure","resident_on":"OpaqueRef:ccfe0e03-5b7d-46ac-a8ef-35d63c9ba5e7","progress":1,"type":"<none/>","result":"","error_info":["IMPORT_ERROR","INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ]"],"other_config":{"object_creation":"complete"},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 1967))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"},"pool_master":{"uuid":"58e38fa8-c2d2-459d-8446-0cf0bbda626e","name_label":"Ryzen1-XCP-NG","name_description":"Don't worry, everything happens for a Ryzen...","memory_overhead":1763532800,"allowed_operations":["vm_migrate","provision","vm_resume","evacuate","vm_start"],"current_operations":{},"API_version_major":2,"API_version_minor":16,"API_version_vendor":"XenSource","API_version_vendor_implementation":{},"enabled":true,"software_version":{"product_version":"8.2.1","product_version_text":"8.2","product_version_text_short":"8.2","platform_name":"XCP","platform_version":"3.2.1","product_brand":"XCP-ng","build_number":"release/yangtze/master/58","hostname":"localhost","date":"2022-08-17","dbv":"0.0.1","xapi":"1.20","xen":"4.13.4-9.25.1","linux":"4.19.0+1","xencenter_min":"2.16","xencenter_max":"2.16","network_backend":"openvswitch","db_schema":"5.602"},"other_config":{"agent_start_time":"1665919758.","boot_time":"1665919723.","last_blob_sync_time":"1665228407.72","iscsi_iqn":"iqn.2022-09.com.example:166a7d3c","rpm_patch_installation_time":"1662506898.06"},"capabilities":["xen-3.0-x86_64","hvm-3.0-x86_32","hvm-3.0-x86_32p","hvm-3.0-x86_64",""],"cpu_configuration":{},"sched_policy":"credit","supported_bootloaders":["pygrub","eliloader"],"resident_VMs":["OpaqueRef:56141690-c36d-4ec0-9d99-bb5d8466f91f","OpaqueRef:8b312ac3-6180-4ba9-9c56-93d834df437a","OpaqueRef:8b3beb03-4a51-40f5-a851-efbabd9ff922","OpaqueRef:b95b1f28-95ee-4521-be90-37a5833fc032","OpaqueRef:fbe1523c-20a7-486d-93e6-b4a295373fd5","OpaqueRef:4d8a7b05-0451-4938-bdec-573b30118790","OpaqueRef:52e1c465-464c-4154-9ff2-21cb13539a7c"],"logging":{"syslog_destination":""},"PIFs":["OpaqueRef:f36e1f2e-c5df-4da9-86ab-ef5492f8f3c4","OpaqueRef:d918c7c8-94c4-4309-8932-011b6a852b2b","OpaqueRef:bc95e309-e2d8-45ba-9bef-e7d5ce6099b3","OpaqueRef:bc185fce-743f-4945-ab75-9e3662778240","OpaqueRef:ba11c9db-07a4-4ffb-bc7e-7bc73446d31e","OpaqueRef:affcdb28-84b9-4f0c-b209-e669b5033763","OpaqueRef:a1242f16-f538-429e-8024-11055c7d7c6f","OpaqueRef:a10ee21a-9f1a-4c26-874d-b2ff250e5779","OpaqueRef:9ef07c9b-112b-4f86-a333-83160d1e61c8","OpaqueRef:9d86120a-1cae-4202-b987-a1a739cc882a","OpaqueRef:926579cb-48f7-408b-8679-8cb54b138edf","OpaqueRef:8d460590-b160-4d7c-b93e-f61531d7baa5","OpaqueRef:7e16a323-60b6-4cb4-afa5-721b4562257c","OpaqueRef:69ba1431-1a93-4fec-809e-3ce616a175f9","OpaqueRef:67027b3d-5729-4f79-a167-d2c65b688e10","OpaqueRef:60928c09-5fc5-45d7-a449-d2055789d13f","OpaqueRef:5e9a179b-085b-4331-bcde-e4b9cc1a00cb","OpaqueRef:53e5135c-09fb-4611-a637-8b44b80da013","OpaqueRef:102ef69e-1635-4369-b5d6-cccec168d67a"],"suspend_image_sr":"OpaqueRef:8e796368-0836-4da3-80f3-99f1f8783fb9","crash_dump_sr":"OpaqueRef:8e796368-0836-4da3-80f3-99f1f8783fb9","crashdumps":["OpaqueRef:fc7e5999-d0b6-403c-92a1-c7db8edf7a79"],"patches":[],"updates":[],"PBDs":["OpaqueRef:f2430beb-2e71-4804-8eea-b2431a8161ec","OpaqueRef:cc9c8681-6418-4f58-a61e-c8faf4c77a48","OpaqueRef:baa91a3c-1437-462b-8a9b-5f5e267faadb","OpaqueRef:b13e92f2-1c64-4280-b246-b0a09d0bf946","OpaqueRef:977a5ce1-a1f5-460c-813f-b8cc022e3bae","OpaqueRef:02d876af-68c3-4bae-b28e-1e32f53ec73d"],"host_CPUs":["OpaqueRef:c0ff1496-07f3-4d18-904d-421ba87266c2","OpaqueRef:29a31390-4e22-46cb-ba3a-b6c18927154b","OpaqueRef:45e20ffd-e8ba-4d70-8f3d-e11901dd3ded","OpaqueRef:42294c04-e800-4f88-8a14-ad90810ff61d","OpaqueRef:b42d634c-82e5-4067-8faf-776cd8c0c732","OpaqueRef:e2baba85-4774-41be-8103-750d34e3c3e6","OpaqueRef:ad47e14b-d7b7-40d2-8975-9cc826253fc3","OpaqueRef:5df2c60a-30dc-4ca9-88ce-59a679795229","OpaqueRef:e8ea3968-a77c-4efd-99dd-037eae4b69e4","OpaqueRef:a92f9bae-db6b-4a0c-87ec-8c740515d51d","OpaqueRef:227bda7e-a0aa-43e9-b429-7a21286e6ebd","OpaqueRef:c1e92cce-27eb-4cdd-bac4-216956d03135","OpaqueRef:e12a945a-ac2b-476e-a728-b2055b14f468","OpaqueRef:2c6c7c76-6aeb-474d-a799-32f3a06f2063","OpaqueRef:9c4d92b3-1107-457d-bf95-d886714272bc","OpaqueRef:4211a3cc-d5b2-4f6c-8bca-99aedbb77769","OpaqueRef:3607079b-2f1e-4cbc-a59e-a8c79fc6a74e","OpaqueRef:30254f21-7962-4059-82e9-5802fb36c0e0","OpaqueRef:6f6b3ae0-c7d5-46d6-9df9-804c90706572","OpaqueRef:464c61a9-96aa-4602-80dc-bf67360f14cd","OpaqueRef:21bc1460-a25f-428d-a6a4-d85b8a635a32","OpaqueRef:e31f75a8-8b7f-4b10-945c-aa032527221a","OpaqueRef:98f497c1-1938-49bf-b8c6-62a9a481613b","OpaqueRef:52f42239-4d0f-4057-9fe8-86e58db662ab"],"cpu_info":{"cpu_count":"24","socket_count":"1","vendor":"AuthenticAMD","speed":"3693.074","modelname":"AMD Ryzen 9 5900X 12-Core Processor","family":"25","model":"33","stepping":"2","flags":"fpu de tsc msr pae mce cx8 apic mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy abm sse4a misalignsse 3dnowprefetch bpext ibpb vmmcall fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat vaes vpclmulqdq rdpid","features_pv":"1789c3f5-f6f83203-2bd1cbf5-040001e3-00000007-218c0329-00400600-00000000-20001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm":"178bfbff-f6f83203-2fd3fbff-040085f7-0000000f-219c07a9-0040060c-00000000-310ed005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm_host":"178bfbff-f6f83203-2e500800-040001f7-0000000f-219c07a9-0040060c-00000000-310ed005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_pv_host":"1789c3f5-f6f83203-2a500800-040001e3-00000007-218c0329-00400600-00000000-20001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"},"hostname":"ryzen1-xcp-ng","address":"192.168.3.94","metrics":"OpaqueRef:286451e9-e3a3-4185-92a4-21cc83e11a6c","license_params":{"restrict_vswitch_controller":"false","restrict_lab":"false","restrict_stage":"false","restrict_storagelink":"false","restrict_storagelink_site_recovery":"false","restrict_web_selfservice":"false","restrict_web_selfservice_manager":"false","restrict_hotfix_apply":"false","restrict_export_resource_data":"false","restrict_read_caching":"false","restrict_cifs":"false","restrict_health_check":"false","restrict_xcm":"false","restrict_vm_memory_introspection":"false","restrict_batch_hotfix_apply":"false","restrict_management_on_vlan":"false","restrict_ws_proxy":"false","restrict_vlan":"false","restrict_qos":"false","restrict_pool_attached_storage":"false","restrict_netapp":"false","restrict_equalogic":"false","restrict_pooling":"false","enable_xha":"true","restrict_marathon":"false","restrict_email_alerting":"false","restrict_historical_performance":"false","restrict_wlb":"false","restrict_rbac":"false","restrict_dmc":"false","restrict_checkpoint":"false","restrict_cpu_masking":"false","restrict_connection":"false","platform_filter":"false","regular_nag_dialog":"false","restrict_vmpr":"false","restrict_vmss":"false","restrict_intellicache":"false","restrict_gpu":"false","restrict_dr":"false","restrict_vif_locking":"false","restrict_storage_xen_motion":"false","restrict_vgpu":"false","restrict_integrated_gpu_passthrough":"false","restrict_vss":"false","restrict_guest_agent_auto_update":"false","restrict_pci_device_for_auto_update":"false","restrict_xen_motion":"false","restrict_guest_ip_setting":"false","restrict_ad":"false","restrict_nested_virt":"false","restrict_live_patching":"false","restrict_set_vcpus_number_live":"false","restrict_pvs_proxy":"false","restrict_igmp_snooping":"false","restrict_rpu":"false","restrict_pool_size":"false","restrict_cbt":"false","restrict_usb_passthrough":"false","restrict_network_sriov":"false","restrict_corosync":"true","restrict_zstd_export":"false","restrict_pool_secret_rotation":"false"},"ha_statefiles":[],"ha_network_peers":[],"blobs":{},"tags":[],"external_auth_type":"","external_auth_service_name":"","external_auth_configuration":{},"edition":"xcp-ng","license_server":{"address":"localhost","port":"27000"},"bios_strings":{"bios-vendor":"American Megatrends International, LLC.","bios-version":"T1.29b","system-manufacturer":"To Be Filled By O.E.M.","system-product-name":"X570D4U","system-version":"To Be Filled By O.E.M.","system-serial-number":"To Be Filled By O.E.M.","baseboard-manufacturer":"ASRockRack","baseboard-product-name":"X570D4U","baseboard-version":"","baseboard-serial-number":"T80-F3001300401","oem-1":"Xen","oem-2":"MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d","oem-3":"Default string","hp-rombios":""},"power_on_mode":"","power_on_config":{},"local_cache_sr":"OpaqueRef:8e796368-0836-4da3-80f3-99f1f8783fb9","chipset_info":{"iommu":"false"},"PCIs":["OpaqueRef:d57d7375-af85-4914-b62e-51223bb8506b","OpaqueRef:d1c33253-b62c-47b8-97ec-39ed7339d930","OpaqueRef:c8dfa8f6-26b1-4eef-89b6-9cbb5bef1df3","OpaqueRef:c78f15c1-69bf-46d1-b482-a08f7f1f039c","OpaqueRef:9db62e9d-fcc0-4abf-9f2e-b6e684f9b43e","OpaqueRef:56f44afe-87b4-4d10-a3dc-dc0e0e72cb32","OpaqueRef:3a5b38b2-5573-4f19-b3ce-649f5ccc968e","OpaqueRef:2fd2ac48-6334-4a70-b058-0e8f632e5aba","OpaqueRef:01302328-a19c-471c-9a1a-83f651a1662b"],"PGPUs":["OpaqueRef:3f0a5ebf-7c4a-4082-81ce-a2e375b3a411"],"PUSBs":[],"ssl_legacy":false,"guest_VCPUs_params":{},"display":"enabled","virtual_hardware_platform_versions":[0,1,2],"control_domain":"OpaqueRef:52e1c465-464c-4154-9ff2-21cb13539a7c","updates_requiring_reboot":[],"features":[],"iscsi_iqn":"iqn.2022-09.com.example:166a7d3c","multipathing":false,"uefi_certificates":"","certificates":[],"editions":["xcp-ng"]},"SR":{"uuid":"6ff9113c-3531-9e2a-66ba-8fc5be69eee7","name_label":"Ryzen2 Local Storage","name_description":"","allowed_operations":["vdi_enable_cbt","vdi_list_changed_blocks","unplug","plug","pbd_create","vdi_disable_cbt","update","pbd_destroy","vdi_resize","vdi_clone","vdi_data_destroy","scan","vdi_snapshot","vdi_mirror","vdi_create","vdi_destroy","vdi_set_on_boot"],"current_operations":{},"VDIs":["OpaqueRef:dc771123-6833-4c3b-9199-a42da7f5ff5b","OpaqueRef:d8a630ed-ee56-4109-b9bd-cbe3c1b5f405","OpaqueRef:c4defeab-e0c5-4e3d-8cdb-08d71ac0acb6","OpaqueRef:78fe3d0b-effd-4620-abfd-162877c8b428"],"PBDs":["OpaqueRef:e6d92534-41c9-4d62-9dc7-63897568deba"],"virtual_allocation":144955146240,"physical_utilisation":49125265408,"physical_size":1924991709184,"type":"ext","content_type":"user","shared":false,"other_config":{"i18n-original-value-name_label":"Local storage","i18n-key":"local-storage"},"tags":[],"sm_config":{"devserial":""},"blobs":{},"local_cache_enabled":true,"introduced_by":"OpaqueRef:NULL","clustered":false,"is_tools_sr":false},"message":"IMPORT_ERROR(INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ])","name":"XapiError","stack":"XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Stream_vdi.Invalid_checksum(\"Block Ref:69/00156509 checksum failed: original = 36C730AC922C3EBB; recomputed = D7D6772B5F65D4E1\") ])\n    at Function.wrap (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/_XapiError.js:16:12)\n    at _default (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/_getTaskResult.js:11:29)\n    at Xapi._addRecordToCache (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:954:24)\n    at forEach (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:988:14)\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:978:12)\n    at Xapi._watchEvents (/etc/xo/xo-builds/xen-orchestra-202209302029/packages/xen-api/src/index.js:1144:14)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:442:9)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"}

olivierlambert

Ping @florent on this

florent

hi @lawrencesystems,

is there a way to give me access to your XO and look at the files on the samba ?

Full backups ( as in the *.xva files ) are untouched by vhd directory.

Regards,

Florent

lawrencesystems

I am going to be out of town until next week, but I can keep the backups so we can arrange a time to test it.

I have done a few more tests, I setup a Synology using SMB and it backs up and restores just fine. I also moved the existing backups over to an NFS share and that previously created backup gives the same Stream_vdi.Invalid_checksum error with the same block number on restore.

florent

@lawrencesystems we don't do any magic on the xva (yet) . It's probably an error coming from the host, and I wuld like to investigate if possible. It's great it you can keep the files.

Do you use encryption ?

olivierlambert

Yeah, sounds like a somehow corrupted XVA that can't be imported. The checksum computation fails on the host. So likely the XVA is corrupted, alternatively a transfer issue corrupting the file, but I'm less sure about this

olivierlambert

What could be interesting: importing the XVA with xe CLI and see if the result is the same.

lawrencesystems

@florent

No encryption on this remote. And if I have time before I leave I will test importing the XVA with CLI and see what error I get.

lawrencesystems

I really think there is some bug in the TrueNAS Core SMB stack that I am fighting here because this issue
https://xcp-ng.org/forum/topic/6148/delta-backup-showing-success-no-delta-saved?_=1666089968101
I had reported before was only happening on TrueNAS Core via SMB. Also this is solved sine XO that extra integrity checking feature was added to the delta backups.

florent

@lawrencesystems We added the healthcheck for full backup (here : https://github.com/vatesfr/xen-orchestra/pull/6401 )
Also we removed our js smb implementation ( https://github.com/vatesfr/xen-orchestra/commit/b50e95802c6a876d4e6ffd0c62c209a187b1bfb2 )

to try to circumvent the strange samba comportment

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(backups): add healthcheck to full backup #6401

0

julien-f committed to vatesfr/xen-orchestra

feat(fs): remove JS based SMB handler

It's not well tested nor maintained.