Update: SOLVED
I installed a temporal server, and copied all vms (it took a few tries, but eventually finished ok)
With the main server out of production, I took it to my office, and found out that the memory was making some errors. Only 3 or 4 errors with each Memtest Pass.
Replaced memory, and 1 NVME SSD that died, clean 8.2.1 install, and its working OK with no errors during backup
Thanks!!!!
Update: SOLVED
I installed a temporal server, and copied all vms (it took a few tries, but eventually finished ok)
With the main server out of production, I took it to my office, and found out that the memory was making some errors. Only 3 or 4 errors with each Memtest Pass.
Replaced memory, and 1 NVME SSD that died, clean 8.2.1 install, and its working OK with no errors during backup
Thanks!!!!
Well, i tried something else.
I installed a new xcpng server on a new pc, I conected it to Xen Orchestra, and tried to clone a VM into the new SR in the new server
This is the error im getting, also a SSL error.
vm.copy
{
"vm": "aa11ae17-55b9-e2d0-767b-4bca7d4017d5",
"sr": "68e37620-53f2-e8cd-1f41-fc7ed0662be6",
"name": "Windows 10_NVME (20230125T020002Z)_COPY"
}
{
"library": "SSL routines",
"reason": "decryption failed or bad record mac",
"code": "ERR_SSL_DECRYPTION_FAILED_OR_BAD_RECORD_MAC",
"pool_master": {
"uuid": "2026fd16-d9fa-4829-a796-5a1d054934d1",
"name_label": "xcp-ng-4600G",
"name_description": "Default install",
"memory_overhead": 614526976,
"allowed_operations": [
"vm_migrate",
"provision",
"vm_resume",
"evacuate",
"vm_start"
],
"current_operations": {},
"API_version_major": 2,
"API_version_minor": 16,
"API_version_vendor": "XenSource",
"API_version_vendor_implementation": {},
"enabled": true,
"software_version": {
"product_version": "8.2.1",
"product_version_text": "8.2",
"product_version_text_short": "8.2",
"platform_name": "XCP",
"platform_version": "3.2.1",
"product_brand": "XCP-ng",
"build_number": "release/yangtze/master/58",
"hostname": "localhost",
"date": "2023-04-27",
"dbv": "0.0.1",
"xapi": "1.20",
"xen": "4.13.5-9.31",
"linux": "4.19.0+1",
"xencenter_min": "2.16",
"xencenter_max": "2.16",
"network_backend": "openvswitch",
"db_schema": "5.602"
},
"other_config": {
"agent_start_time": "1688937974.",
"boot_time": "1688937927.",
"rpm_patch_installation_time": "1688929690.783",
"iscsi_iqn": "iqn.2023-07.com.example:26f2c293"
},
"capabilities": [
"xen-3.0-x86_64",
"hvm-3.0-x86_32",
"hvm-3.0-x86_32p",
"hvm-3.0-x86_64",
""
],
"cpu_configuration": {},
"sched_policy": "credit",
"supported_bootloaders": [
"pygrub",
"eliloader"
],
"resident_VMs": [
"OpaqueRef:279db818-b68c-41b0-878c-2643cba2e33a"
],
"logging": {},
"PIFs": [
"OpaqueRef:8c8486b3-a011-4d27-8673-bfa503243668"
],
"suspend_image_sr": "OpaqueRef:330796d1-72df-4b74-a47c-a489e6732d07",
"crash_dump_sr": "OpaqueRef:330796d1-72df-4b74-a47c-a489e6732d07",
"crashdumps": [],
"patches": [],
"updates": [],
"PBDs": [
"OpaqueRef:de390e31-f8bb-445b-ad6a-82665befd0f0",
"OpaqueRef:c661d408-4d7e-4671-8671-f0681eba510e",
"OpaqueRef:9004aa40-0320-43dd-9432-a101a3975bb2",
"OpaqueRef:857dcc3b-09b9-4be1-99f8-89a746386c91",
"OpaqueRef:5958f84e-ea7d-4f1d-9cd8-5843afa36e03"
],
"host_CPUs": [
"OpaqueRef:440f4cb7-d6ae-4555-bf45-412855401c97",
"OpaqueRef:fd0e6e97-d636-4815-b1af-15e8c69e7df6",
"OpaqueRef:82bc36f9-de38-4148-8f28-4378e985c189",
"OpaqueRef:1b170109-47f7-44ce-a4cd-0ed833fe45c1",
"OpaqueRef:25b48891-b3cc-49d0-862a-468152a51daa",
"OpaqueRef:9fa6a436-6edf-4289-a854-9f337dea15f7",
"OpaqueRef:c6356d64-f186-4758-8efa-6799ea1fa00e",
"OpaqueRef:e0111a81-e63e-4cb7-9cca-973eb88e9f32",
"OpaqueRef:5b33fbee-7180-4f43-a846-d5b1c90f6b1e",
"OpaqueRef:6ada684d-a0c3-4922-9ede-284c99112054",
"OpaqueRef:3b96a7be-52af-412a-92d5-2e47d55ed69d",
"OpaqueRef:fdca4c61-4ce6-45e4-b7ec-bfe0aaa9a529"
],
"cpu_info": {
"cpu_count": "12",
"socket_count": "1",
"vendor": "AuthenticAMD",
"speed": "3693.014",
"modelname": "AMD Ryzen 5 4600G with Radeon Graphics",
"family": "23",
"model": "96",
"stepping": "1",
"flags": "fpu de tsc msr pae mce cx8 apic mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy abm sse4a misalignsse 3dnowprefetch bpext ssbd ibpb vmmcall fsgsbase bmi1 avx2 bmi2 rdseed adx clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat rdpid",
"features_pv": "1789c3f5-f6f83203-2bd1cbf5-040001e3-00000007-218c0129-00400000-00000000-00001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
"features_hvm": "178bfbff-f6f83203-2fd3fbff-040085f7-0000000f-219c01a9-00400004-00000000-010cd005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
"features_hvm_host": "178bfbff-f6f83203-2e500800-040001f7-0000000f-219c01a9-00400004-00000000-010cd005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
"features_pv_host": "1789c3f5-f6f83203-2a500800-040001e3-00000007-218c0129-00400000-00000000-00001005-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"
},
"hostname": "xcp-ng-4600G",
"address": "192.168.1.154",
"metrics": "OpaqueRef:7b37d2fe-c3d5-497d-83e1-6317571f9e8e",
"license_params": {
"restrict_vswitch_controller": "false",
"restrict_lab": "false",
"restrict_stage": "false",
"restrict_storagelink": "false",
"restrict_storagelink_site_recovery": "false",
"restrict_web_selfservice": "false",
"restrict_web_selfservice_manager": "false",
"restrict_hotfix_apply": "false",
"restrict_export_resource_data": "false",
"restrict_read_caching": "false",
"restrict_cifs": "false",
"restrict_health_check": "false",
"restrict_xcm": "false",
"restrict_vm_memory_introspection": "false",
"restrict_batch_hotfix_apply": "false",
"restrict_management_on_vlan": "false",
"restrict_ws_proxy": "false",
"restrict_vlan": "false",
"restrict_qos": "false",
"restrict_pool_attached_storage": "false",
"restrict_netapp": "false",
"restrict_equalogic": "false",
"restrict_pooling": "false",
"enable_xha": "true",
"restrict_marathon": "false",
"restrict_email_alerting": "false",
"restrict_historical_performance": "false",
"restrict_wlb": "false",
"restrict_rbac": "false",
"restrict_dmc": "false",
"restrict_checkpoint": "false",
"restrict_cpu_masking": "false",
"restrict_connection": "false",
"platform_filter": "false",
"regular_nag_dialog": "false",
"restrict_vmpr": "false",
"restrict_vmss": "false",
"restrict_intellicache": "false",
"restrict_gpu": "false",
"restrict_dr": "false",
"restrict_vif_locking": "false",
"restrict_storage_xen_motion": "false",
"restrict_vgpu": "false",
"restrict_integrated_gpu_passthrough": "false",
"restrict_vss": "false",
"restrict_guest_agent_auto_update": "false",
"restrict_pci_device_for_auto_update": "false",
"restrict_xen_motion": "false",
"restrict_guest_ip_setting": "false",
"restrict_ad": "false",
"restrict_nested_virt": "false",
"restrict_live_patching": "false",
"restrict_set_vcpus_number_live": "false",
"restrict_pvs_proxy": "false",
"restrict_igmp_snooping": "false",
"restrict_rpu": "false",
"restrict_pool_size": "false",
"restrict_cbt": "false",
"restrict_usb_passthrough": "false",
"restrict_network_sriov": "false",
"restrict_corosync": "true",
"restrict_zstd_export": "false",
"restrict_pool_secret_rotation": "false"
},
"ha_statefiles": [],
"ha_network_peers": [],
"blobs": {},
"tags": [],
"external_auth_type": "",
"external_auth_service_name": "",
"external_auth_configuration": {},
"edition": "xcp-ng",
"license_server": {
"address": "localhost",
"port": "27000"
},
"bios_strings": {
"bios-vendor": "American Megatrends International, LLC.",
"bios-version": "F14",
"system-manufacturer": "Gigabyte Technology Co., Ltd.",
"system-product-name": "A520M H",
"system-version": "Default string",
"system-serial-number": "Default string",
"baseboard-manufacturer": "Gigabyte Technology Co., Ltd.",
"baseboard-product-name": "A520M H",
"baseboard-version": "x.x",
"baseboard-serial-number": "Default string",
"oem-1": "Xen",
"oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d",
"oem-3": "Default string",
"hp-rombios": ""
},
"power_on_mode": "",
"power_on_config": {},
"local_cache_sr": "OpaqueRef:330796d1-72df-4b74-a47c-a489e6732d07",
"chipset_info": {
"iommu": "true"
},
"PCIs": [
"OpaqueRef:fc55df6b-3712-429d-8f4c-56d6d59480a6",
"OpaqueRef:f1ea652c-fa1e-4338-9dc5-ad5a47863b78",
"OpaqueRef:eeac8fe3-3bef-4c12-9c67-aa1b85baf20f",
"OpaqueRef:dd225072-30af-4ea7-b438-1e10f74c23c4",
"OpaqueRef:aa328fa7-7093-47bd-9bca-f837792ebcdd",
"OpaqueRef:969843b3-cb0d-42f6-90c4-387058190d48",
"OpaqueRef:96601e05-552b-4eec-8271-94a7598fb7a9",
"OpaqueRef:94696449-b738-4568-863c-b4f3b84627dc",
"OpaqueRef:7181688e-a157-431b-a3fc-4226557b71a5",
"OpaqueRef:547c2159-2b7c-4384-8e75-bc52cf7ae80c",
"OpaqueRef:31bb3c35-5bd1-4cb4-bf18-49a05fbf26c2"
],
"PGPUs": [
"OpaqueRef:a1c7436f-0269-4f2b-99bf-71095cd10a1d"
],
"PUSBs": [
"OpaqueRef:d930bc92-6c6f-4659-a1f2-3a37cc09e5fb"
],
"ssl_legacy": false,
"guest_VCPUs_params": {},
"display": "enabled",
"virtual_hardware_platform_versions": [
0,
1,
2
],
"control_domain": "OpaqueRef:279db818-b68c-41b0-878c-2643cba2e33a",
"updates_requiring_reboot": [],
"features": [],
"iscsi_iqn": "iqn.2023-07.com.example:26f2c293",
"multipathing": false,
"uefi_certificates": "",
"certificates": [],
"editions": [
"xcp-ng"
]
},
"SR": {
"uuid": "68e37620-53f2-e8cd-1f41-fc7ed0662be6",
"name_label": "Local storage",
"name_description": "",
"allowed_operations": [
"vdi_enable_cbt",
"vdi_list_changed_blocks",
"unplug",
"plug",
"pbd_create",
"vdi_disable_cbt",
"update",
"pbd_destroy",
"vdi_resize",
"vdi_clone",
"vdi_data_destroy",
"scan",
"vdi_snapshot",
"vdi_mirror",
"vdi_create",
"vdi_destroy",
"vdi_set_on_boot"
],
"current_operations": {},
"VDIs": [],
"PBDs": [
"OpaqueRef:c661d408-4d7e-4671-8671-f0681eba510e"
],
"virtual_allocation": 0,
"physical_utilisation": 62939136,
"physical_size": 202124111872,
"type": "ext",
"content_type": "user",
"shared": false,
"other_config": {
"i18n-original-value-name_label": "Local storage",
"i18n-key": "local-storage"
},
"tags": [],
"sm_config": {
"devserial": ""
},
"blobs": {},
"local_cache_enabled": true,
"introduced_by": "OpaqueRef:NULL",
"clustered": false,
"is_tools_sr": false
},
"message": "C0D7F6C4AE7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
",
"name": "Error",
"stack": "Error: C0D7F6C4AE7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
"
I also tried to export the VM to a local PC, the export went to 70% then completelly hang the server. Not even a PING response. I manually rebooted and it boot ok.
Any ideas?
I tried with an external HDD with network conection, and i get the same error
So i dont think the problem is in the remote side.
Could a network switch cause this type of errors?
Its a non-admin tp-link switch
Thanks for your reply
Im using Xen Orchestra from source, I cant open a ticket.
I could provide you access to the server
Im NOT using Encryption
Im going to try installing a new NAS to discard any problem on the remote side.
Thanks, Nicolas
New clean dataset, on truenas core backup server, with new remote on XOA failed the same way
im out of ideas
Windows 2
Snapshot
Start: Jun 27, 2023, 10:37:18 PM
End: Jun 27, 2023, 10:37:20 PM
XOA2 SMB
transfer
Start: Jun 27, 2023, 10:37:20 PM
End: Jun 27, 2023, 11:03:23 PM
Duration: 26 minutes
Error: C0D7C267AD7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
Start: Jun 27, 2023, 10:37:20 PM
End: Jun 27, 2023, 11:03:23 PM
Duration: 26 minutes
Error: C0D7C267AD7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
Start: Jun 27, 2023, 10:37:18 PM
End: Jun 27, 2023, 11:03:33 PM
Duration: 26 minutes
Error: C0D7C267AD7F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
Type: full
Yesterday I made a new ubuntu VM with a fresh Comunity install, imported the config file.
Same result
1 Failure
Windows 10
UUID: aa11ae17-55b9-e2d0-767b-4bca7d4017d5
Start time: Tuesday, June 27th 2023, 2:00:01 am
End time: Tuesday, June 27th 2023, 2:00:33 am
Duration: a few seconds
Error: C077A451D87F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
I will try with a new remote to see if anything changes.
Hello, Ive been reciving this error with full backups.
1 Failure
Windows 10
UUID: aa11ae17-55b9-e2d0-767b-4bca7d4017d5
Start time: Thursday, June 22nd 2023, 2:00:01 am
End time: Thursday, June 22nd 2023, 2:00:47 am
Duration: a minute
Error: C0171460C87F0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:560:
I have 1 server with 2 VM, backing up every night to a Truenas SMB share.
XOA Comunity, fully updated
I originaly thought it could be related to a faulty disk that was giving me some errors, but error keeps apearing after replacement.
Sometimes the task completes without error, sometimes i get this error. Last night 1VM backed up OK, the other i get this error. Sometimes I get the error within 1 minute, sometimes after 30min.
What can i try to fix it?
Thanks, Nicolas
@olivierlambert Dont know why the screenshots were not loading, posting that again
With no snapshots and coalesce completed, very very little space used.
@olivierlambert with just 1 snapshot the space used jumps to 300GB
@olivierlambert im calculating the 40gb by the space used in the windows VM for both drives
This is the only VM using with this windows in this machine, i cant remember cloning it.
I recovered this machine from an external backup when i built this server (previous server died from natural causes)