Potential bug with Windows VM backup: "Body Timeout Error"
-
@poddingue My colleage managed to workaround the issue by re-installing the lab on other hardware, I think it is now Cisco UCS-servers. It was HP ProLiant-servers before.
Maybe it was an issue with some of the NIC or firmware, im not really sure, but it works now.
-
I get this error with each metadata backup run, but every time it fail on different pool. Only 1 failed pool at once.
-
@Tristis-Oris Hi,
Can you upload your complete logs from one of these jobs, please? You can DM me if you prefer
-
@poddingue I have asked our engineers for feedback. We have left one big VM with 1TB empty disk - and full backups are going without errors for one week.
So it seems issue is resolved.
I have to mention that during this time we have migrated to XCP-NG 8.3. and are using one of the latest XO builds. -
That's excellent news, thanks a lot for the feedback!
-
commit 63f8d, (Ronivay script)
Got this error on my test ofTWINSTOR: next gen 2 nodes HCI
First I ran the job manually (metadata and config), OK
Then the scheduled job ran with this error on the metadata.
{ reportWhen: "failure" } id "1783045813184" jobId "e664efb6-9514-4e6b-8303-f236ad4dbfb9" jobName "twin-conf" message "backup" scheduleId "d343fdd7-9dde-4695-adec-ba2eb43f23d2" start 1783045813184 status "failure" tasks [ {…}, {…} ] end 1783046114017 result { message: "backup task failed with undefined error", name: "Error", stack: "Error: backup task failed with undefined error\n at forwardResult (file:///opt/xo/xo-builds/xen-orchestra-202607020951/pa….runJobSequence (file:///opt/xo/xo-builds/xen-orchestra-202607020951/packages/xo-server/src/xo-mixins/jobs/index.mjs:339:7)" }Config seems to be OK
-
Hi,
Would be nice to have the entire log
Could it be possible that the master or the pool was not reachable at the time of the metadata backup?
-
@olivierlambert
Removed the uefi_certificates{ "data": { "reportWhen": "failure" }, "id": "1783045813184", "jobId": "e664efb6-9514-4e6b-8303-f236ad4dbfb9", "jobName": "twin-conf", "message": "backup", "scheduleId": "d343fdd7-9dde-4695-adec-ba2eb43f23d2", "start": 1783045813184, "status": "failure", "tasks": [ { "id": "0mr4bfumi-dzqwb7948p", "start": 1783045813194, "status": "success", "tasks": [ { "id": "0mr4bfumi-9kdhhby2p5", "start": 1783045813194, "status": "success", "end": 1783045813258, "message": "Starting XO metadata backup for the remote (da08058c-c031-499d-b423-52b1e76380ed). (e664efb6-9514-4e6b-8303-f236ad4dbfb9)", "data": { "id": "da08058c-c031-499d-b423-52b1e76380ed", "type": "remote" } } ], "end": 1783045813259, "message": "Starting XO metadata backup. (e664efb6-9514-4e6b-8303-f236ad4dbfb9)", "data": { "type": "xo" } }, { "id": "0mr4bfuml-bqp15noo59i", "start": 1783045813197, "status": "failure", "tasks": [ { "id": "0mr4bfun2-d0l56rvwtg", "start": 1783045813214, "status": "failure", "end": 1783046113969, "result": { "name": "BodyTimeoutError", "code": "UND_ERR_BODY_TIMEOUT", "message": "Body Timeout Error", "stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607020951/node_modules/undici/lib/dispatcher/client-h1.js:688:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607020951/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:605:17)\n at processTimers (node:internal/timers:541:7)" }, "message": "Starting metadata backup for the pool (7e461b17-b34e-5e10-277b-84677bb207bf) for the remote (da08058c-c031-499d-b423-52b1e76380ed). (e664efb6-9514-4e6b-8303-f236ad4dbfb9)", "data": { "id": "da08058c-c031-499d-b423-52b1e76380ed", "type": "remote", "progress": 0 } } ], "end": 1783046113971, "message": "Starting metadata backup for the pool (7e461b17-b34e-5e10-277b-84677bb207bf). (e664efb6-9514-4e6b-8303-f236ad4dbfb9)", "data": { "id": "7e461b17-b34e-5e10-277b-84677bb207bf", "pool": { "uuid": "7e461b17-b34e-5e10-277b-84677bb207bf", "name_label": "Middle-earth", "name_description": "", "master": "OpaqueRef:5ceac8f1-744c-13b0-0652-75fdff79f6c9", "default_SR": "OpaqueRef:NULL", "suspend_image_SR": "OpaqueRef:NULL", "crash_dump_SR": "OpaqueRef:NULL", "other_config": { "xo:clientInfo:zffyon3svem": "{\"lastConnected\":1783034682565,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.104\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"192.168.11.104/24\"},{\"address\":\"fe80::a47c:87ff:fe67:5e41\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"fe80::a47c:87ff:fe67:5e41/64\",\"scopeid\":2}]}}", "xo:clientInfo:c725c94e-296d-61d8-23a0-f07c63520f65": "{\"lastConnected\":1783013072625,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.101\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"e2:7f:90:57:75:a6\",\"internal\":false,\"cidr\":\"192.168.11.101/24\"},{\"address\":\"fe80::e07f:90ff:fe57:75a6\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"e2:7f:90:57:75:a6\",\"internal\":false,\"cidr\":\"fe80::e07f:90ff:fe57:75a6/64\",\"scopeid\":2}]}}", "xo:clientInfo:9kul7pgojmd": "{\"lastConnected\":1783002005305,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.104\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"192.168.11.104/24\"},{\"address\":\"fe80::a47c:87ff:fe67:5e41\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"fe80::a47c:87ff:fe67:5e41/64\",\"scopeid\":2}]}}", "xo:clientInfo:0rwhmux9ihqm": "{\"lastConnected\":1782989396232,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.104\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"192.168.11.104/24\"},{\"address\":\"fe80::a47c:87ff:fe67:5e41\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"fe80::a47c:87ff:fe67:5e41/64\",\"scopeid\":2}]}}", "xo:clientInfo:4gtha9oducv": "{\"lastConnected\":1782983029680,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.104\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"192.168.11.104/24\"},{\"address\":\"fe80::a47c:87ff:fe67:5e41\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"a6:7c:87:67:5e:41\",\"internal\":false,\"cidr\":\"fe80::a47c:87ff:fe67:5e41/64\",\"scopeid\":2}]}}", "auto_poweron": "true", "xo:clientInfo:7f8ngv9i6st": "{\"lastConnected\":1782946813358,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.102\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"ae:f5:9b:89:01:07\",\"internal\":false,\"cidr\":\"192.168.11.102/24\"},{\"address\":\"fe80::acf5:9bff:fe89:107\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"ae:f5:9b:89:01:07\",\"internal\":false,\"cidr\":\"fe80::acf5:9bff:fe89:107/64\",\"scopeid\":2}]}}", "xo:clientInfo:1hfj1o99bag": "{\"lastConnected\":1782945346185,\"networkInterfaces\":{\"enX0\":[{\"address\":\"192.168.11.102\",\"netmask\":\"255.255.255.0\",\"family\":\"IPv4\",\"mac\":\"ae:f5:9b:89:01:07\",\"internal\":false,\"cidr\":\"192.168.11.102/24\"},{\"address\":\"fe80::acf5:9bff:fe89:107\",\"netmask\":\"ffff:ffff:ffff:ffff::\",\"family\":\"IPv6\",\"mac\":\"ae:f5:9b:89:01:07\",\"internal\":false,\"cidr\":\"fe80::acf5:9bff:fe89:107/64\",\"scopeid\":2}]}}", "memory-ratio-hvm": "0.25", "memory-ratio-pv": "0.25" }, "ha_enabled": true, "ha_configuration": {}, "ha_statefiles": [ "OpaqueRef:1088109f-4634-2d30-6d7d-4b120dd09690" ], "ha_host_failures_to_tolerate": 1, "ha_plan_exists_for": 1, "ha_allow_overcommit": false, "ha_overcommitted": false, "blobs": {}, "tags": [], "gui_config": {}, "health_check_config": {}, "wlb_url": "", "wlb_username": "", "wlb_enabled": false, "wlb_verify_cert": true, "redo_log_enabled": false, "redo_log_vdi": "OpaqueRef:NULL", "vswitch_controller": "", "restrictions": { "restrict_vswitch_controller": "false", "restrict_lab": "false", "restrict_stage": "false", "restrict_storagelink": "false", "restrict_storagelink_site_recovery": "false", "restrict_web_selfservice": "false", "restrict_web_selfservice_manager": "false", "restrict_hotfix_apply": "false", "restrict_export_resource_data": "false", "restrict_read_caching": "false", "restrict_cifs": "false", "restrict_health_check": "false", "restrict_xcm": "false", "restrict_vm_memory_introspection": "false", "restrict_batch_hotfix_apply": "false", "restrict_management_on_vlan": "false", "restrict_ws_proxy": "false", "restrict_cloud_management": "false", "restrict_nrpe": "false", "restrict_vlan": "false", "restrict_qos": "false", "restrict_pool_attached_storage": "false", "restrict_netapp": "false", "restrict_equalogic": "false", "restrict_pooling": "false", "enable_xha": "true", "restrict_marathon": "false", "restrict_email_alerting": "false", "restrict_historical_performance": "false", "restrict_wlb": "false", "restrict_rbac": "false", "restrict_dmc": "false", "restrict_checkpoint": "false", "restrict_cpu_masking": "false", "restrict_connection": "false", "platform_filter": "false", "regular_nag_dialog": "false", "restrict_vmpr": "false", "restrict_vmss": "false", "restrict_intellicache": "false", "restrict_gpu": "false", "restrict_dr": "false", "restrict_vif_locking": "false", "restrict_storage_xen_motion": "false", "restrict_vgpu": "false", "restrict_integrated_gpu_passthrough": "false", "restrict_vss": "false", "restrict_guest_agent_auto_update": "false", "restrict_pci_device_for_auto_update": "false", "restrict_xen_motion": "false", "restrict_guest_ip_setting": "false", "restrict_ad": "false", "restrict_nested_virt": "false", "restrict_live_patching": "false", "restrict_set_vcpus_number_live": "false", "restrict_pvs_proxy": "false", "restrict_igmp_snooping": "false", "restrict_rpu": "false", "restrict_pool_size": "false", "restrict_cbt": "false", "restrict_usb_passthrough": "false", "restrict_network_sriov": "false", "restrict_corosync": "true", "restrict_cluster_address": "false", "restrict_zstd_export": "false", "restrict_pool_secret_rotation": "false", "restrict_certificate_verification": "false", "restrict_updates": "false", "restrict_internal_repo_access": "false", "restrict_vtpm": "false", "restrict_vm_groups": "false", "restrict_vm_start": "false", "restrict_vm_appliance_start": "false" }, "metadata_VDIs": [], "ha_cluster_stack": "xhad", "allowed_operations": [ "ha_disable", "designate_new_master", "configure_repositories", "sync_updates", "sync_bundle", "apply_updates", "cert_refresh", "exchange_certificates_on_join", "exchange_ca_certificates_on_join", "copy_primary_host_certs", "eject", "get_updates" ], "current_operations": {}, "guest_agent_config": {}, "cpu_info": { "features_hvm_host": "1fcbfbff-f7fa3203-2c100800-00000021-00000001-000007ab-00000000-00000000-00101000-bc000400-00000000-00000000-00000000-00000000-00000000-00000000-0c000004-40000000-00000000-00000000-00000000-00000000-00000000", "features_pv_host": "1fc9cbf5-f6d83203-28100800-00000021-00000001-00000329-00000000-00000000-00001000-ac000400-00000000-00000000-00000000-00000000-00000000-00000000-0c000004-40000000-00000000-00000000-00000000-00000000-00000000", "socket_count": "2", "cpu_count": "16", "vendor": "GenuineIntel" }, "policy_no_vendor_device": false, "live_patching_disabled": false, "igmp_snooping_enabled": false, "uefi_certificates": # removed # "custom_uefi_certificates": "", "is_psr_pending": false, "tls_verification_enabled": true, "repositories": [], "client_certificate_auth_enabled": false, "client_certificate_auth_name": "", "repository_proxy_url": "", "repository_proxy_username": "", "repository_proxy_password": "OpaqueRef:NULL", "migration_compression": false, "coordinator_bias": true, "local_auth_max_threads": 8, "ext_auth_max_threads": 1, "ext_auth_cache_enabled": false, "ext_auth_cache_size": 50, "ext_auth_cache_expiry": 300, "telemetry_uuid": "OpaqueRef:b19f5323-f179-a7b7-3e45-0cbd78992b0e", "telemetry_frequency": "weekly", "telemetry_next_collection": "20260702T23:44:39Z", "last_update_sync": "19700101T00:00:00Z", "update_sync_frequency": "weekly", "update_sync_day": 0, "update_sync_enabled": false, "recommendations": { "max-vm-anti-affinity-groups": "5" }, "license_server": {}, "ha_reboot_vm_on_internal_shutdown": true, "limit_console_sessions": false, "vm_console_idle_timeout": 0 }, "poolMaster": { "uuid": "aae1f2ab-1f2f-4a38-af90-97d850a2caaf", "name_label": "saruman", "name_description": "Default install", "memory_overhead": 463126528, "allowed_operations": [ "vm_migrate", "provision", "vm_resume", "enable", "evacuate", "vm_start" ], "current_operations": {}, "API_version_major": 2, "API_version_minor": 21, "API_version_vendor": "XenSource", "API_version_vendor_implementation": {}, "enabled": true, "software_version": { "product_version": "8.3.0", "product_version_text": "8.3", "product_version_text_short": "8.3", "platform_name": "XCP", "platform_version": "3.4.0", "product_brand": "XCP-ng", "xapi": "26.1", "build_number": "8.3.0", "git_id": "11", "hostname": "localhost", "date": "20260618T12:39:30Z", "dbv": "2026.0202", "xapi_build": "26.1.11", "xen": "4.17.6-9", "linux": "4.19.0+1", "xencenter_min": "2.21", "xencenter_max": "2.21", "network_backend": "openvswitch", "db_schema": "5.795" }, "other_config": { "agent_start_time": "1783013042.", "boot_time": "1783012910.", "multipathhandle": "dmp", "multipathing": "true", "last_blob_sync_time": "1782941656.94", "iscsi_iqn": "iqn.2026-07.com.example:9a839973" }, "capabilities": [ "xen-3.0-x86_64", "hvm-3.0-x86_32", "hvm-3.0-x86_32p", "hvm-3.0-x86_64", "" ], "cpu_configuration": {}, "sched_policy": "credit", "supported_bootloaders": [ "pygrub", "eliloader" ], "resident_VMs": [ "OpaqueRef:b4cb1c1e-8c91-178b-74be-7528a4c2dcc0", "OpaqueRef:4a3ee368-0fb7-dba6-eb3e-6cf4456f9e84" ], "logging": {}, "PIFs": [ "OpaqueRef:b9de3d0c-a8c6-6b96-9248-4b8ee7de8e86" ], "suspend_image_sr": "OpaqueRef:NULL", "crash_dump_sr": "OpaqueRef:NULL", "crashdumps": [], "patches": [], "updates": [], "PBDs": [ "OpaqueRef:d435ddeb-af36-ae51-67b5-2f711ba4c2a9", "OpaqueRef:c5b01445-3d66-d589-124f-d8edaf3389e8", "OpaqueRef:735a54e2-505d-afaa-3522-b99621ffdb44", "OpaqueRef:5c175bce-c41d-d97d-d108-fcbd16a601d4", "OpaqueRef:528cbc67-88d9-465c-0e99-2a11602a4ff9" ], "host_CPUs": [ "OpaqueRef:543628ec-dab6-a811-c4ac-94a878c4fd0b", "OpaqueRef:ed0d5baf-537f-03a4-ed99-9d4c21321a5b", "OpaqueRef:0ac07111-529a-6fc9-16e2-bd444fbca1e1", "OpaqueRef:9188a89f-112b-8121-227d-fb88322d21d0", "OpaqueRef:669b605f-9c0f-1577-7d2a-cb64dc936f05", "OpaqueRef:826404a3-6afd-0458-aaf6-f7220a376c0d", "OpaqueRef:afd77111-fbb5-0d75-9be8-0d3ecd5cfbf6", "OpaqueRef:81eff8c4-8dd4-0b54-d8c8-b247d6c0acac" ], "cpu_info": { "cpu_count": "8", "socket_count": "1", "threads_per_core": "2", "nr_nodes": "1", "vendor": "GenuineIntel", "speed": "1800.005", "modelname": "Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz", "family": "6", "model": "142", "stepping": "10", "flags": "fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt xsaveopt xsavec xgetbv1 arch_capabilities", "features_pv": "1fc9cbf5-f6f83203-2991cbf5-00000123-00000007-008c0329-00000000-00000100-00001000-ac000400-00000000-00000000-00000000-00000000-00000000-00000000-1c020004-40000000-00000000-00000000-00000000-00000000-00000000", "features_hvm": "1fcbfbff-f7fa3223-2d93fbff-00000523-0000000f-009c47ab-00000000-00000100-00101000-bc000400-00000000-00000000-00000000-00000000-00000000-00000000-1c020004-40000000-00000000-00000000-00000000-00000000-00000000", "features_hvm_host": "1fcbfbff-f7fa3203-2c100800-00000121-0000000f-009c07ab-00000000-00000000-00101000-bc000400-00000000-00000000-00000000-00000000-00000000-00000000-0c000004-40000000-00000000-00000000-00000000-00000000-00000000", "features_pv_host": "1fc9cbf5-f6d83203-28100800-00000121-00000007-008c0329-00000000-00000000-00001000-ac000400-00000000-00000000-00000000-00000000-00000000-00000000-0c000004-40000000-00000000-00000000-00000000-00000000-00000000" }, "hostname": "saruman", "address": "192.168.11.23", "metrics": "OpaqueRef:5ba864f1-c495-0787-d239-947f0d6f99d4", "license_params": { "restrict_vswitch_controller": "false", "restrict_lab": "false", "restrict_stage": "false", "restrict_storagelink": "false", "restrict_storagelink_site_recovery": "false", "restrict_web_selfservice": "false", "restrict_web_selfservice_manager": "false", "restrict_hotfix_apply": "false", "restrict_export_resource_data": "false", "restrict_read_caching": "false", "restrict_cifs": "false", "restrict_health_check": "false", "restrict_xcm": "false", "restrict_vm_memory_introspection": "false", "restrict_batch_hotfix_apply": "false", "restrict_management_on_vlan": "false", "restrict_ws_proxy": "false", "restrict_cloud_management": "false", "restrict_nrpe": "false", "restrict_vlan": "false", "restrict_qos": "false", "restrict_pool_attached_storage": "false", "restrict_netapp": "false", "restrict_equalogic": "false", "restrict_pooling": "false", "enable_xha": "true", "restrict_marathon": "false", "restrict_email_alerting": "false", "restrict_historical_performance": "false", "restrict_wlb": "false", "restrict_rbac": "false", "restrict_dmc": "false", "restrict_checkpoint": "false", "restrict_cpu_masking": "false", "restrict_connection": "false", "platform_filter": "false", "regular_nag_dialog": "false", "restrict_vmpr": "false", "restrict_vmss": "false", "restrict_intellicache": "false", "restrict_gpu": "false", "restrict_dr": "false", "restrict_vif_locking": "false", "restrict_storage_xen_motion": "false", "restrict_vgpu": "false", "restrict_integrated_gpu_passthrough": "false", "restrict_vss": "false", "restrict_guest_agent_auto_update": "false", "restrict_pci_device_for_auto_update": "false", "restrict_xen_motion": "false", "restrict_guest_ip_setting": "false", "restrict_ad": "false", "restrict_nested_virt": "false", "restrict_live_patching": "false", "restrict_set_vcpus_number_live": "false", "restrict_pvs_proxy": "false", "restrict_igmp_snooping": "false", "restrict_rpu": "false", "restrict_pool_size": "false", "restrict_cbt": "false", "restrict_usb_passthrough": "false", "restrict_network_sriov": "false", "restrict_corosync": "true", "restrict_cluster_address": "false", "restrict_zstd_export": "false", "restrict_pool_secret_rotation": "false", "restrict_certificate_verification": "false", "restrict_updates": "false", "restrict_internal_repo_access": "false", "restrict_vtpm": "false", "restrict_vm_groups": "false", "restrict_vm_start": "false", "restrict_vm_appliance_start": "false" }, "ha_statefiles": [ "OpaqueRef:1088109f-4634-2d30-6d7d-4b120dd09690" ], "ha_network_peers": [ "8c67881d-e100-4a02-9e8d-98aac410e6b0", "aae1f2ab-1f2f-4a38-af90-97d850a2caaf" ], "blobs": {}, "tags": [], "external_auth_type": "", "external_auth_service_name": "", "external_auth_configuration": {}, "edition": "xcp-ng", "license_server": { "address": "localhost", "port": "27000" }, "bios_strings": { "bios-vendor": "HP", "bios-version": "Q83 Ver. 01.31.00", "system-manufacturer": "HP", "system-product-name": "HP ProBook 640 G4", "system-version": "SBKPF", "system-serial-number": "5CG83432QF", "baseboard-manufacturer": "HP", "baseboard-product-name": "83D2", "baseboard-version": "KBC Version 05.4E.00", "baseboard-serial-number": "PGWKH00WBB71HJ", "oem-1": "Xen", "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d", "oem-3": "FBYTE#3X476J6S6b7B7H7M7Q7W7m7saBaEapaqauawbUbhcAdQdUdpdqfAgdhk.Dt;", "oem-4": "BUILDID#17WWC6BT601#SAK8#DAK8;", "oem-5": "EDK2_1", "oem-6": "Buff=2", "oem-7": "HRDWFEATS=VTX:1;VTD:1;SGX:2;NONHPBATDET:1", "hp-rombios": "" }, "power_on_mode": "", "power_on_config": {}, "local_cache_sr": "OpaqueRef:NULL", "chipset_info": { "iommu": "true" }, "PCIs": [ "OpaqueRef:d69ea10d-4a28-305b-bc77-c5b325b5efca", "OpaqueRef:d3b82ebd-72be-7b4d-aeb0-45286ba30820", "OpaqueRef:963b2435-0607-a303-c312-f841673e644d", "OpaqueRef:91af9331-a1a4-3025-073b-dd347fb9b0cf", "OpaqueRef:6326bf67-6291-e5b5-a028-990f642bb113", "OpaqueRef:3d4982c9-e712-610a-01d8-570f8cabd276", "OpaqueRef:1a62277c-19dd-df1b-d9e0-5d9991ee031c", "OpaqueRef:04ce1fdb-f128-5e1d-6567-dc35ab79204a" ], "PGPUs": [ "OpaqueRef:8f35fe69-4c6d-c0c1-3958-26a98932d691" ], "PUSBs": [], "ssl_legacy": false, "guest_VCPUs_params": {}, "display": "enabled", "virtual_hardware_platform_versions": [ 0, 1, 2 ], "control_domain": "OpaqueRef:4a3ee368-0fb7-dba6-eb3e-6cf4456f9e84", "updates_requiring_reboot": [], "features": [], "iscsi_iqn": "iqn.2026-07.com.example:9a839973", "multipathing": true, "uefi_certificates": # removed # "certificates": [ "OpaqueRef:f3d58304-b037-5739-b57f-3d301b18a7ff", "OpaqueRef:0dc9c6bd-0fb7-de23-70d2-4bc2bef46c17" ], "editions": [ "xcp-ng" ], "pending_guidances": [], "tls_verification_enabled": true, "last_software_update": "19700101T00:00:00Z", "https_only": false, "latest_synced_updates_applied": "unknown", "numa_affinity_policy": "default_policy", "pending_guidances_recommended": [], "pending_guidances_full": [], "last_update_hash": "", "ssh_enabled": true, "ssh_enabled_timeout": 0, "ssh_expiry": "19700101T00:00:00Z", "console_idle_timeout": 0, "ssh_auto_mode": false, "max_cstate": "", "secure_boot": false, "ntp_mode": "DHCP", "ntp_custom_servers": [], "timezone": "Europe/Stockholm" }, "type": "pool" } } ], "end": 1783046114017, "result": { "message": "backup task failed with undefined error", "name": "Error", "stack": "Error: backup task failed with undefined error\n at forwardResult (file:///opt/xo/xo-builds/xen-orchestra-202607020951/packages/xo-server/src/_handleBackupLog.mjs:37:25)\n at handleBackupLog (file:///opt/xo/xo-builds/xen-orchestra-202607020951/packages/xo-server/src/_handleBackupLog.mjs:68:12)\n at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202607020951/packages/xo-server/src/xo-mixins/metadata-backups.mjs:133:13)\n at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202607020951/@xen-orchestra/mixins/Tasks.mjs:205:23)\n at Task.onProgress [as _onProgress] (/opt/xo/xo-builds/xen-orchestra-202607020951/@vates/task/combineEvents.js:61:5)\n at Task.#emit (/opt/xo/xo-builds/xen-orchestra-202607020951/@vates/task/index.js:157:10)\n at Task.#end (/opt/xo/xo-builds/xen-orchestra-202607020951/@vates/task/index.js:168:15)\n at Task.run (/opt/xo/xo-builds/xen-orchestra-202607020951/@vates/task/index.js:190:16)\n at Jobs.runJob (file:///opt/xo/xo-builds/xen-orchestra-202607020951/packages/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n at Jobs.runJobSequence (file:///opt/xo/xo-builds/xen-orchestra-202607020951/packages/xo-server/src/xo-mixins/jobs/index.mjs:339:7)" } }Could it be possible that the master or the pool was not reachable at the time of the metadata backup?
Don't think so.
The config ran OK at 04:30 but metadata did not, see above
And the backup job 5 min later ran OK -
Thanks for the logs @ph7, that helps narrow it down.
Some context on what this error actually means: XO's XAPI client kills the connection if it receives nothing from the pool master for 5 minutes during the transfer. So the
BodyTimeoutErroron a metadata backup means the/pool/xmldbdumpresponse from your master went silent. Either xapi stalled while producing the dump, or the connection died silently somewhere in between (a NAT/firewall/VPN dropping the connection state would look exactly like this). Note that your XO config backup succeeding right before doesn't rule out a network issue: the config backup only talks to the remote, it never touches the pool master.To figure out which side is failing, could you check a few things?
- On your backup remote, look at the file left behind by the failed run:
xo-pool-metadata-backups/<scheduleId>/<poolUuid>/<date>/data. What size is it, compared to the same file from a successful run? If it's empty or tiny, the master stalled. If it looks complete, the data actually arrived and only the end-of-connection signal got lost, which would point at the network path. - On the pool master, grab
/var/log/xensource.logaround the failure time. There should be a task named "Export pool metadata" and the xmldbdump HTTP handler. We'd like to know if it completed on the xapi side. - Is there anything between your XO and the pool master, like NAT, a firewall or a VPN?
- Does it fail every run, or intermittently?
@Tristis-Oris the same questions apply to your setup. The fact that it hits different pools one after the other makes me suspect something shared in the network path.
- On your backup remote, look at the file left behind by the failed run:
-
@olivierlambert for now it solved after toolstack restart everywhere.
-
Is there anything between your XO and the pool master, like NAT, a firewall or a VPN?
No, XO is running on master (twinstor)
Both hosts connected to the same switch
the remote is 2 swithes away
same switches as my "production" poolDoes it fail every run, or intermittently?
Only run scheduled once, manually once.
I will increase the schedule to more than once a dayMore answers to come later.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login