Restoring from backup error: self-signed certificate
-
Hello,
We are testing XCP-ng at my workplace, and so far are very happy with the performance and the backup capabilities which seem promising. The performance seems great even on an old machine we use for testing stuff.
I scheduled a full backup of a VM to a QNAP NAS and it worked perfectly. However, in testing the restore function in XO, I am immediately receiving the below error. The task ([XO] VM import (on xcp-test)) has been stuck at 0% for an hour or so. It can't be cancelled and it doesn't seem to fail either.
Obviously it is related to a self-signed certificate, but I'm not understanding where. I am not using any certificates on the XCP-ng server itself, or on the XO VM.
So I have two questions:
- How can I cancel a hung backup/restore in progress without rebooting the whole server?
- What troubleshooting steps can I take to resolve the certificate error?
Thanks so much in advance.
-KevinbackupNg.importVmBackup { "id": "02d195d0-3179-4e08-afc2-3559d717eabc//xo-vm-backups/de2be8e2-e63e-0397-89a9-82ab281bfe4c/20240315T130001Z.json", "settings": { "mapVdisSrs": {}, "newMacAddresses": false, "useDifferentialRestore": false }, "sr": "51e66f10-b830-134c-6a4d-ccbaca176b8d" } { "code": "DEPTH_ZERO_SELF_SIGNED_CERT", "originalUrl": "https://10.0.90.1/import/?sr_id=OpaqueRef%3A8759a452-9cb9-4691-877e-6286cf6da7dd&session_id=OpaqueRef%3Ae21a579e-ecae-4955-86d0-bd7864551fd1", "url": "https://10.0.90.1/import/?sr_id=OpaqueRef%3A8759a452-9cb9-4691-877e-6286cf6da7dd&session_id=OpaqueRef%3Ae21a579e-ecae-4955-86d0-bd7864551fd1", "pool_master": { "uuid": "e54323a8-a0a6-4d0f-bdfc-f5591c49a82e", "name_label": "xcp-test", "name_description": "Default install", "memory_overhead": 377720832, "allowed_operations": [ "vm_migrate", "provision", "vm_resume", "evacuate", "vm_start" ], "current_operations": {}, "API_version_major": 2, "API_version_minor": 16, "API_version_vendor": "XenSource", "API_version_vendor_implementation": {}, "enabled": true, "software_version": { "product_version": "8.2.1", "product_version_text": "8.2", "product_version_text_short": "8.2", "platform_name": "XCP", "platform_version": "3.2.1", "product_brand": "XCP-ng", "build_number": "release/yangtze/master/58", "hostname": "localhost", "date": "2023-10-18", "dbv": "0.0.1", "xapi": "1.20", "xen": "4.13.5-9.38", "linux": "4.19.0+1", "xencenter_min": "2.16", "xencenter_max": "2.16", "network_backend": "openvswitch", "db_schema": "5.603" }, "other_config": { "agent_start_time": "1710512795.", "boot_time": "1710512755.", "iscsi_iqn": "iqn.2024-03.com.corp.local:b12e63a4" }, "capabilities": [ "xen-3.0-x86_64", "hvm-3.0-x86_32", "hvm-3.0-x86_32p", "hvm-3.0-x86_64", "" ], "cpu_configuration": {}, "sched_policy": "credit", "supported_bootloaders": [ "pygrub", "eliloader" ], "resident_VMs": [ "OpaqueRef:ee315ea0-e710-46e5-950a-311033fcddf4", "OpaqueRef:f18eb629-2860-42f2-b34d-93a47948c53d", "OpaqueRef:c79581dd-17fa-4943-8f4c-a211d98a391d" ], "logging": {}, "PIFs": [ "OpaqueRef:862e510d-d4e2-4f47-b65b-76bd9f2cfe41" ], "suspend_image_sr": "OpaqueRef:7b14f084-c96d-4d26-9aa9-3ec3e507fe84", "crash_dump_sr": "OpaqueRef:7b14f084-c96d-4d26-9aa9-3ec3e507fe84", "crashdumps": [], "patches": [], "updates": [], "PBDs": [ "OpaqueRef:f5147bb0-e879-4851-81d9-f65b3f577cfc", "OpaqueRef:ad047568-7aa9-4057-a2bb-2234561fa9f6", "OpaqueRef:9c91f198-b6e7-4b44-b7b9-5fb0d10e9f71", "OpaqueRef:945c52ae-b81c-419e-9a71-e62226a78ec2", "OpaqueRef:82ffd4f2-38d5-4d5e-a138-4e781f89c9ca", "OpaqueRef:7d1d173b-77ce-435f-b7ce-6b456ef087da" ], "host_CPUs": [ "OpaqueRef:a9c6f849-ed27-4695-a857-3260b570675a", "OpaqueRef:1c599843-0aa5-45ae-b643-791d86923ac0", "OpaqueRef:024392dc-8e86-42e5-a18f-d80d966df940", "OpaqueRef:7f92d0dc-9169-4615-9fc2-58d45db5ad47", "OpaqueRef:6b4fc6f8-dec2-4737-91d0-2cd52ce13dfb", "OpaqueRef:be47d328-7de1-4984-a83c-09051113032d", "OpaqueRef:f8074f6a-7d4f-412c-b5d0-88843a193a09", "OpaqueRef:7e551f5f-0c09-4c20-9b6b-0c1705bde5e1" ], "cpu_info": { "cpu_count": "8", "socket_count": "1", "vendor": "GenuineIntel", "speed": "3997.373", "modelname": "Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz", "family": "6", "model": "60", "stepping": "3", "flags": "fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms xsaveopt", "features_pv": "1fc9cbf5-f6f83203-2991cbf5-00000023-00000001-00000329-00000000-00000000-00001000-ac000400-00000000-00000000-00000000-00000000-00000000-00000000-00080004-00000000-00000000-00000000-00000000-00000000", "features_hvm": "1fcbfbff-f7fa3223-2d93fbff-00000423-00000001-000007ab-00000000-00000000-00001000-bc000400-00000000-00000000-00000000-00000000-00000000-00000000-00080004-00000000-00000000-00000000-00000000-00000000", "features_hvm_host": "1fcbfbff-f7fa3223-2c100800-00000021-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_pv_host": "1fc9cbf5-f6f83203-28100800-00000021-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000" }, "hostname": "xcp-test", "address": "10.0.90.1", "metrics": "OpaqueRef:07c2bcd0-11e2-40c0-9fe7-fc8d6e433b60", "license_params": { "restrict_vswitch_controller": "false", "restrict_lab": "false", "restrict_stage": "false", "restrict_storagelink": "false", "restrict_storagelink_site_recovery": "false", "restrict_web_selfservice": "false", "restrict_web_selfservice_manager": "false", "restrict_hotfix_apply": "false", "restrict_export_resource_data": "false", "restrict_read_caching": "false", "restrict_cifs": "false", "restrict_health_check": "false", "restrict_xcm": "false", "restrict_vm_memory_introspection": "false", "restrict_batch_hotfix_apply": "false", "restrict_management_on_vlan": "false", "restrict_ws_proxy": "false", "restrict_vlan": "false", "restrict_qos": "false", "restrict_pool_attached_storage": "false", "restrict_netapp": "false", "restrict_equalogic": "false", "restrict_pooling": "false", "enable_xha": "true", "restrict_marathon": "false", "restrict_email_alerting": "false", "restrict_historical_performance": "false", "restrict_wlb": "false", "restrict_rbac": "false", "restrict_dmc": "false", "restrict_checkpoint": "false", "restrict_cpu_masking": "false", "restrict_connection": "false", "platform_filter": "false", "regular_nag_dialog": "false", "restrict_vmpr": "false", "restrict_vmss": "false", "restrict_intellicache": "false", "restrict_gpu": "false", "restrict_dr": "false", "restrict_vif_locking": "false", "restrict_storage_xen_motion": "false", "restrict_vgpu": "false", "restrict_integrated_gpu_passthrough": "false", "restrict_vss": "false", "restrict_guest_agent_auto_update": "false", "restrict_pci_device_for_auto_update": "false", "restrict_xen_motion": "false", "restrict_guest_ip_setting": "false", "restrict_ad": "false", "restrict_nested_virt": "false", "restrict_live_patching": "false", "restrict_set_vcpus_number_live": "false", "restrict_pvs_proxy": "false", "restrict_igmp_snooping": "false", "restrict_rpu": "false", "restrict_pool_size": "false", "restrict_cbt": "false", "restrict_usb_passthrough": "false", "restrict_network_sriov": "false", "restrict_corosync": "true", "restrict_zstd_export": "false", "restrict_pool_secret_rotation": "false" }, "ha_statefiles": [], "ha_network_peers": [], "blobs": {}, "tags": [], "external_auth_type": "", "external_auth_service_name": "", "external_auth_configuration": {}, "edition": "xcp-ng", "license_server": { "address": "localhost", "port": "27000" }, "bios_strings": { "bios-vendor": "American Megatrends Inc.", "bios-version": "3503", "system-manufacturer": "ASUS", "system-product-name": "All Series", "system-version": "System Version", "system-serial-number": "System Serial Number", "baseboard-manufacturer": "ASUSTeK COMPUTER INC.", "baseboard-product-name": "Z97-A-USB31", "baseboard-version": "Rev 1.xx", "baseboard-serial-number": "150850429900542", "oem-1": "Xen", "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d", "oem-3": "To Be Filled By O.E.M.", "oem-4": "To Be Filled By O.E.M.", "oem-5": "Ferrari", "oem-6": "To Be Filled By O.E.M.", "hp-rombios": "" }, "power_on_mode": "", "power_on_config": {}, "local_cache_sr": "OpaqueRef:NULL", "chipset_info": { "iommu": "false" }, "PCIs": [ "OpaqueRef:f9b54698-d8eb-4b45-896b-60a3ec4ba337", "OpaqueRef:e50556e8-0e1d-4e90-a8a2-7bf19a394bcc", "OpaqueRef:dfd23ced-a2fa-4beb-812a-5f7b16e78d92", "OpaqueRef:ca19e106-2d87-4bba-994e-0106fc349cf5", "OpaqueRef:45670663-bd71-4da0-923c-1f49f1f6d29e", "OpaqueRef:36425a8d-285f-439e-b591-af8eb8838fbb" ], "PGPUs": [ "OpaqueRef:6f6e4fc9-7b67-4f57-856d-4bed9c1633ac" ], "PUSBs": [], "ssl_legacy": false, "guest_VCPUs_params": {}, "display": "enabled", "virtual_hardware_platform_versions": [ 0, 1, 2 ], "control_domain": "OpaqueRef:c79581dd-17fa-4943-8f4c-a211d98a391d", "updates_requiring_reboot": [], "features": [], "iscsi_iqn": "iqn.2024-03.com.corp.local:b12e63a4", "multipathing": false, "uefi_certificates": "", "certificates": [], "editions": [ "xcp-ng" ], "https_only": false }, "SR": { "uuid": "51e66f10-b830-134c-6a4d-ccbaca176b8d", "name_label": "C5-JOBS-XCP", "name_description": "C5-JOBS-XCP", "allowed_operations": [ "vdi_enable_cbt", "vdi_list_changed_blocks", "unplug", "plug", "pbd_create", "vdi_disable_cbt", "update", "pbd_destroy", "vdi_resize", "vdi_clone", "vdi_data_destroy", "scan", "vdi_snapshot", "vdi_mirror", "vdi_create", "vdi_destroy", "vdi_set_on_boot" ], "current_operations": {}, "VDIs": [ "OpaqueRef:ff085f7b-df4c-45f9-930a-a13d13d2f113" ], "PBDs": [ "OpaqueRef:82ffd4f2-38d5-4d5e-a138-4e781f89c9ca" ], "virtual_allocation": 136365211648, "physical_utilisation": 12272683450368, "physical_size": 28316617670656, "type": "nfs", "content_type": "user", "shared": true, "other_config": { "auto-scan": "true" }, "tags": [], "sm_config": {}, "blobs": {}, "local_cache_enabled": false, "introduced_by": "OpaqueRef:NULL", "clustered": false, "is_tools_sr": false }, "message": "self-signed certificate", "name": "Error", "stack": "Error: self-signed certificate at TLSSocket.onConnectSecure (node:_tls_wrap:1659:34) at TLSSocket.emit (node:events:517:28) at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202403150735/@xen-orchestra/log/configure.js:52:17) at TLSSocket._finishInit (node:_tls_wrap:1070:8) at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:856:12) at TLSWrap.callbackTrampoline (node:internal/async_hooks:128:17)" } -
@KS Have you checked the QNAP NAS on its certificates for any self signed ones? As the NAS could be using them for its web management interface and/or any of its other network services.
Also the XCP-ng host utilises a self-signed certificate by default when it is first installed.
It's likely the case also for the XO VM as well (if XOA) so would need changing from self-signed to a proper cross signed certificate.
-
Thanks for your reply.
I have checked the QNAP; it does have a self-signed certificate however I do not believe this is causing the issue as I have zero issues with this unit with any other device, or even backing up from XCP-ng to the QNAP.
I believe the issue lies with the XCP-ng host and it's self-signed certificate. I will look into that. Thanks.
-
@KS said in Restoring from backup error: self-signed certificate:
Thanks for your reply.
I have checked the QNAP; it does have a self-signed certificate however I do not believe this is causing the issue as I have zero issues with this unit with any other device, or even backing up from XCP-ng to the QNAP.
I believe the issue lies with the XCP-ng host and it's self-signed certificate. I will look into that. Thanks.
You don't think that it's certificate can interfere but the SMB and/or NFS protocol support digital signing and/or encryption of the connection when configured. So depending on its settings and/or default settings of that QNAP NAS it may be encrypting the connection. Thus putting into place its self signed certificate!
Also have you installed (added - trusted) the QNAP NAS's self signed certificate on any other device at any point in time?
@olivierlambert Do you know of a way from the shell to cancel a running backup task which has hanged and can't be cancelled from Xen Orchestra?
-
@john-c said in Restoring from backup error: self-signed certificate:
You don't think that it's certificate can interfere but the SMB and/or NFS protocol support digital signing and/or encryption of the connection when configured. So depending on its settings and/or default settings of that QNAP NAS it may be encrypting the connection. Thus putting into place its self signed certificate!
I'm sure that's a possibility, I'm just not sure that's the reason because I haven't had issues with that QNAP - we use Veeam for backup and restore to that QNAP with no issues at all. Also, the log I posted above makes no mention of the QNAP server, by name or by IP, at all.
Those two reasons are why I'm leaning more towards it being a certificate issue with the XCP-ng server and/or the XOA VM.
-
@KS said in Restoring from backup error: self-signed certificate:
@john-c said in Restoring from backup error: self-signed certificate:
You don't think that it's certificate can interfere but the SMB and/or NFS protocol support digital signing and/or encryption of the connection when configured. So depending on its settings and/or default settings of that QNAP NAS it may be encrypting the connection. Thus putting into place its self signed certificate!
I'm sure that's a possibility, I'm just not sure that's the reason because I haven't had issues with that QNAP - we use Veeam for backup and restore to that QNAP with no issues at all. Also, the log I posted above makes no mention of the QNAP server, by name or by IP, at all.
Those two reasons are why I'm leaning more towards it being a certificate issue with the XCP-ng server and/or the XOA VM.
Does Veeam or any other devices have the self-signed certificate added in a way that it would be trusted or has an certificate error exception been added to any device which connects to Veeam or that QNAP NAS?
Just something to consider as this will cause the certificate error to not appear when connecting to that address. Thus if this was done by someone other than you and it wasn't noted down anywhere then the error won't appear and you won't even know.
-
Hello All,
I'm having a similar issue after upgrading from 8.2.1 to 8.3 Beta 2 in my home lab.
Steps performed prior to backup ..
- Full VM backup to TrueNAS and exported XO config json.
- Upgrade XCP-ng host using the 8.3 Beta 2 ISO
- Updated XO to latest version and imported XO config json.
Tried a VM restore from TrueNAS backup and I'm getting the
"self-signed certificate" error. -
@Ismail said in Restoring from backup error: self-signed certificate:
Just an update, the restore worked on the older version of XO which was about 21 commits behind. Current XO version ...
-
Hey @Ismail
Could you please tell me what version of XO the restore worked in? I'm currently on 6c160 which it says is 4 commits behind. I would like to try that out to see if it solves my issue as well.
Thanks
-
Hello @KS unfortunately I deleted the older XO vm, before making a note of the version, and proceeded to upgrade to the latest.
-
@Ismail No worries! Follow up question then, does the restore still work after you've updated to the latest version?
-
@KS Nope, I did one successful restore and then assumed all was good and proceeded to upgrade XO. By the way I'm using Ronivay's XO vm image from https://github.com/ronivay/XenOrchestraInstallerUpdater
-
Same issue on my host, with commit 8e5d9.
I tried the health check, and it is stuck.
I do not see any error message, the log just says pending.I tried a restore check with XOA 5.91.2, no issue on that side, only XO from source seems to be not working.
-
Thanks for your reply. Seems like this might be a bug with the latest version of XO from sources.
I'm trying to use the XO script by Ronivay to install an older branch to see if that solves this issue. In the xo-install.cfg file there is a "BRANCH" variable and it says you can specify a commit number, but that doesn't seem to working for me. I'm sure I'm formatting it incorrectly. Thoughts?
Tried both:
BRANCH="6056a61" BRANCH="6056a618c3e503fcc0de4fe19007574e40bfddd5"Thanks.
-
Same issue here with the latest XO
I tested restore from Qnap with NFS, and there were no errors it's just stuck in tasks with no movement and no way to cancel.
If it helps everything is with default self-signed certificates.
Backups run with no issues, just something with the restore. -
Same issues with restoring on the current commit, but rolled back 21 commits (6056a) and it works again.
-
Same here. I was able to install the same Commit and it works now with the pre-installed certificates.
This appears to be a bug in newer commits.
-
Hi @KS I use Ronivay's precompiled VM which only deploys the latest build. I will have to look into this install script to compile a previous build of the vm. Will let you know if I get it working.
-
Same here, its definately a bug because I was able to do a backup and restore just last week.
I upgraded to the latest 2 days ago and now backups are broken, same version here:
-
I can confirm that in the community edition, something between 0ccfb4b and 8e5d9 breaks restoring to hosts with self signed certs.
I had to try and restore a couple of VMs after the upgrade to 8e5d9 and despite a couple of attempts to compile previous commits and see if they would work, nothing did until I reached 0ccfb4b.
Once I had rolled it back that far, the restores were no longer stuck at 0% and there were no problems restoring any of the backups that I tied.
