SSL routines:ssl3_get_record:decryption failed or bad record when runing Full Backup

[nicolasvi]

Hello, Ive been reciving this error with full backups.

1 Failure
Windows 10
UUID: aa11ae17-55b9-e2d0-767b-4bca7d4017d5
Start time: Thursday, June 22nd 2023, 2:00:01 am
End time: Thursday, June 22nd 2023, 2:00:47 am
Duration: a minute
Error: C0171460C87F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:

I have 1 server with 2 VM, backing up every night to a Truenas SMB share.
XOA Comunity, fully updated

I originaly thought it could be related to a faulty disk that was giving me some errors, but error keeps apearing after replacement.

Sometimes the task completes without error, sometimes i get this error. Last night 1VM backed up OK, the other i get this error. Sometimes I get the error within 1 minute, sometimes after 30min.

What can i try to fix it?

Thanks, Nicolas

[nicolasvi]

Yesterday I made a new ubuntu VM with a fresh Comunity install, imported the config file.
Same result

1 Failure
Windows 10
UUID: aa11ae17-55b9-e2d0-767b-4bca7d4017d5
Start time: Tuesday, June 27th 2023, 2:00:01 am
End time: Tuesday, June 27th 2023, 2:00:33 am
Duration: a few seconds
Error: C077A451D87F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:

I will try with a new remote to see if anything changes.

[nicolasvi]

New clean dataset, on truenas core backup server, with new remote on XOA failed the same way

im out of ideas

 Windows 2
 Snapshot 
Start: Jun 27, 2023, 10:37:18 PM
End: Jun 27, 2023, 10:37:20 PM
 XOA2 SMB 
 transfer
Start: Jun 27, 2023, 10:37:20 PM
End: Jun 27, 2023, 11:03:23 PM
Duration: 26 minutes
Error: C0D7C267AD7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
Start: Jun 27, 2023, 10:37:20 PM
End: Jun 27, 2023, 11:03:23 PM
Duration: 26 minutes
Error: C0D7C267AD7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
Start: Jun 27, 2023, 10:37:18 PM
End: Jun 27, 2023, 11:03:33 PM
Duration: 26 minutes
Error: C0D7C267AD7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
Type: full

[florent]

@nicolasvi If it is a XOA , can you open a ticket and a support tunnel ?

I would like to look into it .

Also , do you use remote encryption ?

[nicolasvi]

@florent

Thanks for your reply

Im using Xen Orchestra from source, I cant open a ticket.
I could provide you access to the server

Im NOT using Encryption

Im going to try installing a new NAS to discard any problem on the remote side.

Thanks, Nicolas

[nicolasvi]

I tried with an external HDD with network conection, and i get the same error

So i dont think the problem is in the remote side.

Could a network switch cause this type of errors?
Its a non-admin tp-link switch

[nicolasvi]

Well, i tried something else.

I installed a new xcpng server on a new pc, I conected it to Xen Orchestra, and tried to clone a VM into the new SR in the new server

This is the error im getting, also a SSL error.

vm.copy
{
  "vm": "aa11ae17-55b9-e2d0-767b-4bca7d4017d5",
  "sr": "68e37620-53f2-e8cd-1f41-fc7ed0662be6",
  "name": "Windows 10_NVME (20230125T020002Z)_COPY"
}
{
  "library": "SSL routines",
  "reason": "decryption failed or bad record mac",
  "code": "ERR_SSL_DECRYPTION_FAILED_OR_BAD_RECORD_MAC",
  "pool_master": {
    "uuid": "2026fd16-d9fa-4829-a796-5a1d054934d1",
    "name_label": "xcp-ng-4600G",
    "name_description": "Default install",
    "memory_overhead": 614526976,
    "allowed_operations": [
      "vm_migrate",
      "provision",
      "vm_resume",
      "evacuate",
      "vm_start"
    ],
    "current_operations": {},
    "API_version_major": 2,
    "API_version_minor": 16,
    "API_version_vendor": "XenSource",
    "API_version_vendor_implementation": {},
    "enabled": true,
    "software_version": {
      "product_version": "8.2.1",
      "product_version_text": "8.2",
      "product_version_text_short": "8.2",
      "platform_name": "XCP",
      "platform_version": "3.2.1",
      "product_brand": "XCP-ng",
      "build_number": "release/yangtze/master/58",
      "hostname": "localhost",
      "date": "2023-04-27",
      "dbv": "0.0.1",
      "xapi": "1.20",
      "xen": "4.13.5-9.31",
      "linux": "4.19.0+1",
      "xencenter_min": "2.16",
      "xencenter_max": "2.16",
      "network_backend": "openvswitch",
      "db_schema": "5.602"
    },
    "other_config": {
      "agent_start_time": "1688937974.",
      "boot_time": "1688937927.",
      "rpm_patch_installation_time": "1688929690.783",
      "iscsi_iqn": "iqn.2023-07.com.example:26f2c293"
    },
    "capabilities": [
      "xen-3.0-x86_64",
      "hvm-3.0-x86_32",
      "hvm-3.0-x86_32p",
      "hvm-3.0-x86_64",
      ""
    ],
    "cpu_configuration": {},
    "sched_policy": "credit",
    "supported_bootloaders": [
      "pygrub",
      "eliloader"
    ],
    "resident_VMs": [
      "OpaqueRef:279db818-b68c-41b0-878c-2643cba2e33a"
    ],
    "logging": {},
    "PIFs": [
      "OpaqueRef:8c8486b3-a011-4d27-8673-bfa503243668"
    ],
    "suspend_image_sr": "OpaqueRef:330796d1-72df-4b74-a47c-a489e6732d07",
    "crash_dump_sr": "OpaqueRef:330796d1-72df-4b74-a47c-a489e6732d07",
    "crashdumps": [],
    "patches": [],
    "updates": [],
    "PBDs": [
      "OpaqueRef:de390e31-f8bb-445b-ad6a-82665befd0f0",
      "OpaqueRef:c661d408-4d7e-4671-8671-f0681eba510e",
      "OpaqueRef:9004aa40-0320-43dd-9432-a101a3975bb2",
      "OpaqueRef:857dcc3b-09b9-4be1-99f8-89a746386c91",
      "OpaqueRef:5958f84e-ea7d-4f1d-9cd8-5843afa36e03"
    ],
    "host_CPUs": [
      "OpaqueRef:440f4cb7-d6ae-4555-bf45-412855401c97",
      "OpaqueRef:fd0e6e97-d636-4815-b1af-15e8c69e7df6",
      "OpaqueRef:82bc36f9-de38-4148-8f28-4378e985c189",
      "OpaqueRef:1b170109-47f7-44ce-a4cd-0ed833fe45c1",
      "OpaqueRef:25b48891-b3cc-49d0-862a-468152a51daa",
      "OpaqueRef:9fa6a436-6edf-4289-a854-9f337dea15f7",
      "OpaqueRef:c6356d64-f186-4758-8efa-6799ea1fa00e",
      "OpaqueRef:e0111a81-e63e-4cb7-9cca-973eb88e9f32",
      "OpaqueRef:5b33fbee-7180-4f43-a846-d5b1c90f6b1e",
      "OpaqueRef:6ada684d-a0c3-4922-9ede-284c99112054",
      "OpaqueRef:3b96a7be-52af-412a-92d5-2e47d55ed69d",
      "OpaqueRef:fdca4c61-4ce6-45e4-b7ec-bfe0aaa9a529"
    ],
    "cpu_info": {
      "cpu_count": "12",
      "socket_count": "1",
      "vendor": "AuthenticAMD",
      "speed": "3693.014",
      "modelname": "AMD Ryzen 5 4600G with Radeon Graphics",
      "family": "23",
      "model": "96",
      "stepping": "1",
      "flags": "fpu de tsc msr pae mce cx8 apic mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy abm sse4a misalignsse 3dnowprefetch bpext ssbd ibpb vmmcall fsgsbase bmi1 avx2 bmi2 rdseed adx clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat rdpid",
      "features_pv": "1789c3f5-f6f83203-2bd1cbf5-040001e3-00000007-218c0129-00400000-00000000-00001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
      "features_hvm": "178bfbff-f6f83203-2fd3fbff-040085f7-0000000f-219c01a9-00400004-00000000-010cd005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
      "features_hvm_host": "178bfbff-f6f83203-2e500800-040001f7-0000000f-219c01a9-00400004-00000000-010cd005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
      "features_pv_host": "1789c3f5-f6f83203-2a500800-040001e3-00000007-218c0129-00400000-00000000-00001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"
    },
    "hostname": "xcp-ng-4600G",
    "address": "192.168.1.154",
    "metrics": "OpaqueRef:7b37d2fe-c3d5-497d-83e1-6317571f9e8e",
    "license_params": {
      "restrict_vswitch_controller": "false",
      "restrict_lab": "false",
      "restrict_stage": "false",
      "restrict_storagelink": "false",
      "restrict_storagelink_site_recovery": "false",
      "restrict_web_selfservice": "false",
      "restrict_web_selfservice_manager": "false",
      "restrict_hotfix_apply": "false",
      "restrict_export_resource_data": "false",
      "restrict_read_caching": "false",
      "restrict_cifs": "false",
      "restrict_health_check": "false",
      "restrict_xcm": "false",
      "restrict_vm_memory_introspection": "false",
      "restrict_batch_hotfix_apply": "false",
      "restrict_management_on_vlan": "false",
      "restrict_ws_proxy": "false",
      "restrict_vlan": "false",
      "restrict_qos": "false",
      "restrict_pool_attached_storage": "false",
      "restrict_netapp": "false",
      "restrict_equalogic": "false",
      "restrict_pooling": "false",
      "enable_xha": "true",
      "restrict_marathon": "false",
      "restrict_email_alerting": "false",
      "restrict_historical_performance": "false",
      "restrict_wlb": "false",
      "restrict_rbac": "false",
      "restrict_dmc": "false",
      "restrict_checkpoint": "false",
      "restrict_cpu_masking": "false",
      "restrict_connection": "false",
      "platform_filter": "false",
      "regular_nag_dialog": "false",
      "restrict_vmpr": "false",
      "restrict_vmss": "false",
      "restrict_intellicache": "false",
      "restrict_gpu": "false",
      "restrict_dr": "false",
      "restrict_vif_locking": "false",
      "restrict_storage_xen_motion": "false",
      "restrict_vgpu": "false",
      "restrict_integrated_gpu_passthrough": "false",
      "restrict_vss": "false",
      "restrict_guest_agent_auto_update": "false",
      "restrict_pci_device_for_auto_update": "false",
      "restrict_xen_motion": "false",
      "restrict_guest_ip_setting": "false",
      "restrict_ad": "false",
      "restrict_nested_virt": "false",
      "restrict_live_patching": "false",
      "restrict_set_vcpus_number_live": "false",
      "restrict_pvs_proxy": "false",
      "restrict_igmp_snooping": "false",
      "restrict_rpu": "false",
      "restrict_pool_size": "false",
      "restrict_cbt": "false",
      "restrict_usb_passthrough": "false",
      "restrict_network_sriov": "false",
      "restrict_corosync": "true",
      "restrict_zstd_export": "false",
      "restrict_pool_secret_rotation": "false"
    },
    "ha_statefiles": [],
    "ha_network_peers": [],
    "blobs": {},
    "tags": [],
    "external_auth_type": "",
    "external_auth_service_name": "",
    "external_auth_configuration": {},
    "edition": "xcp-ng",
    "license_server": {
      "address": "localhost",
      "port": "27000"
    },
    "bios_strings": {
      "bios-vendor": "American Megatrends International, LLC.",
      "bios-version": "F14",
      "system-manufacturer": "Gigabyte Technology Co., Ltd.",
      "system-product-name": "A520M H",
      "system-version": "Default string",
      "system-serial-number": "Default string",
      "baseboard-manufacturer": "Gigabyte Technology Co., Ltd.",
      "baseboard-product-name": "A520M H",
      "baseboard-version": "x.x",
      "baseboard-serial-number": "Default string",
      "oem-1": "Xen",
      "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d",
      "oem-3": "Default string",
      "hp-rombios": ""
    },
    "power_on_mode": "",
    "power_on_config": {},
    "local_cache_sr": "OpaqueRef:330796d1-72df-4b74-a47c-a489e6732d07",
    "chipset_info": {
      "iommu": "true"
    },
    "PCIs": [
      "OpaqueRef:fc55df6b-3712-429d-8f4c-56d6d59480a6",
      "OpaqueRef:f1ea652c-fa1e-4338-9dc5-ad5a47863b78",
      "OpaqueRef:eeac8fe3-3bef-4c12-9c67-aa1b85baf20f",
      "OpaqueRef:dd225072-30af-4ea7-b438-1e10f74c23c4",
      "OpaqueRef:aa328fa7-7093-47bd-9bca-f837792ebcdd",
      "OpaqueRef:969843b3-cb0d-42f6-90c4-387058190d48",
      "OpaqueRef:96601e05-552b-4eec-8271-94a7598fb7a9",
      "OpaqueRef:94696449-b738-4568-863c-b4f3b84627dc",
      "OpaqueRef:7181688e-a157-431b-a3fc-4226557b71a5",
      "OpaqueRef:547c2159-2b7c-4384-8e75-bc52cf7ae80c",
      "OpaqueRef:31bb3c35-5bd1-4cb4-bf18-49a05fbf26c2"
    ],
    "PGPUs": [
      "OpaqueRef:a1c7436f-0269-4f2b-99bf-71095cd10a1d"
    ],
    "PUSBs": [
      "OpaqueRef:d930bc92-6c6f-4659-a1f2-3a37cc09e5fb"
    ],
    "ssl_legacy": false,
    "guest_VCPUs_params": {},
    "display": "enabled",
    "virtual_hardware_platform_versions": [
      0,
      1,
      2
    ],
    "control_domain": "OpaqueRef:279db818-b68c-41b0-878c-2643cba2e33a",
    "updates_requiring_reboot": [],
    "features": [],
    "iscsi_iqn": "iqn.2023-07.com.example:26f2c293",
    "multipathing": false,
    "uefi_certificates": "",
    "certificates": [],
    "editions": [
      "xcp-ng"
    ]
  },
  "SR": {
    "uuid": "68e37620-53f2-e8cd-1f41-fc7ed0662be6",
    "name_label": "Local storage",
    "name_description": "",
    "allowed_operations": [
      "vdi_enable_cbt",
      "vdi_list_changed_blocks",
      "unplug",
      "plug",
      "pbd_create",
      "vdi_disable_cbt",
      "update",
      "pbd_destroy",
      "vdi_resize",
      "vdi_clone",
      "vdi_data_destroy",
      "scan",
      "vdi_snapshot",
      "vdi_mirror",
      "vdi_create",
      "vdi_destroy",
      "vdi_set_on_boot"
    ],
    "current_operations": {},
    "VDIs": [],
    "PBDs": [
      "OpaqueRef:c661d408-4d7e-4671-8671-f0681eba510e"
    ],
    "virtual_allocation": 0,
    "physical_utilisation": 62939136,
    "physical_size": 202124111872,
    "type": "ext",
    "content_type": "user",
    "shared": false,
    "other_config": {
      "i18n-original-value-name_label": "Local storage",
      "i18n-key": "local-storage"
    },
    "tags": [],
    "sm_config": {
      "devserial": ""
    },
    "blobs": {},
    "local_cache_enabled": true,
    "introduced_by": "OpaqueRef:NULL",
    "clustered": false,
    "is_tools_sr": false
  },
  "message": "C0D7F6C4AE7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
",
  "name": "Error",
  "stack": "Error: C0D7F6C4AE7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
"

I also tried to export the VM to a local PC, the export went to 70% then completelly hang the server. Not even a PING response. I manually rebooted and it boot ok.

Any ideas?

[nicolasvi]

Update: SOLVED

I installed a temporal server, and copied all vms (it took a few tries, but eventually finished ok)

With the main server out of production, I took it to my office, and found out that the memory was making some errors. Only 3 or 4 errors with each Memtest Pass.

Replaced memory, and 1 NVME SSD that died, clean 8.2.1 install, and its working OK with no errors during backup

Thanks!!!!

[olivierlambert]

Ah great news Thanks for the feedback!