VDI_IO_ERROR Continuous Replication on clean install.
-
@yomono NodeJS yes. In Node 18, they made a breaking change to put by default a 5 minutes timeout.
We fixed that by adding a specific config to get a longer timeout, see https://github.com/vatesfr/xen-orchestra/commit/f6fd1db1ef12633cc5bb8ec8ab5bc84682dd3fe7
Without this piece of config, you'll end any HTTP stream after exactly 5 minutes.
-
@olivierlambert Well, I checked the config.toml and, indeed, I can confirm the fix is there.
What's really odd here is that I'm using the exact same installation script for XO that I've used in the past.
So, why now this value is correct but it wasn't when executing the exact same script in a VM? (Not an actual question for you, but just wondering out loud)
So @EddieCh08666741 and @Tristis-Oris you were having an exact 5 minutes timeout too so you might want to check this config as well
Thanks for the support, Olivier -
my config is fine, but problem still here for this one pool.
-
@Tristis-Oris i double check and the codes mentioned is updated. The issue is the same for doing the restore from delta backups.
-
{"code":"VDI_IO_ERROR","params":["Device I/O errors"],"url":"https://192.168.0.100/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Af4a60bad-36da-46cb-a555-8e76ff2f3dd6&session_id=OpaqueRef%3Aba2fcb1d-bc46-445b-9540-b0e9e51236ee&task_id=OpaqueRef%3A8ca30f05-cc13-4c04-907f-6b4468a6ace2","task":{"uuid":"fa99f516-2352-3b92-7419-4950559458d8","name_label":"[XO] Importing content into VDI 0","name_description":"","allowed_operations":[],"current_operations":{},"created":"20230204T06:49:36Z","finished":"20230204T06:54:36Z","status":"failure","resident_on":"OpaqueRef:dba25009-ad21-4aad-a7d6-0b03af29a906","progress":1,"type":"<none/>","result":"","error_info":["VDI_IO_ERROR","Device I/O errors"],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename ocaml/xapi/vhd_tool_wrapper.ml)(line 77))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/import_raw_vdi.ml)(line 170)))"},"pool_master":{"uuid":"e16ba86e-9915-446d-824e-f2246a2711ad","name_label":"blade7.quape.com","name_description":"Default install","memory_overhead":2050928640,"allowed_operations":["vm_migrate","provision","vm_resume","evacuate","vm_start"],"current_operations":{},"API_version_major":2,"API_version_minor":16,"API_version_vendor":"XenSource","API_version_vendor_implementation":{},"enabled":true,"software_version":{"product_version":"8.2.1","product_version_text":"8.2","product_version_text_short":"8.2","platform_name":"XCP","platform_version":"3.2.1","product_brand":"XCP-ng","build_number":"release/yangtze/master/58","hostname":"localhost","date":"2022-10-12","dbv":"0.0.1","xapi":"1.20","xen":"4.13.4-9.19.1","linux":"4.19.0+1","xencenter_min":"2.16","xencenter_max":"2.16","network_backend":"openvswitch","db_schema":"5.602"},"other_config":{"agent_start_time":"1670522053.","boot_time":"1670465798.","rpm_patch_installation_time":"1670522042.976","iscsi_iqn":"iqn.2022-10.com.quape:88df7156"},"capabilities":["xen-3.0-x86_64","hvm-3.0-x86_32","hvm-3.0-x86_32p","hvm-3.0-x86_64",""],"cpu_configuration":{},"sched_policy":"credit","supported_bootloaders":["pygrub","eliloader"],"resident_VMs":["OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003"],"logging":{},"PIFs":["OpaqueRef:bb16676d-1c45-4363-9550-ffc6f2323239","OpaqueRef:ae69076b-b4c6-4711-9ab2-95b135daf2d7","OpaqueRef:171c0df8-5b44-459e-8756-01b70115586d","OpaqueRef:12f18eda-5b70-4735-b95e-316fc901d635"],"suspend_image_sr":"OpaqueRef:NULL","crash_dump_sr":"OpaqueRef:NULL","crashdumps":[],"patches":[],"updates":[],"PBDs":["OpaqueRef:08cf61cc-234c-4f62-b006-05fc9f14d27b","OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3","OpaqueRef:064440da-c058-491d-85b6-e9faa1166878"],"host_CPUs":["OpaqueRef:060a9fa7-302b-4f3e-9125-c51c649cc080","OpaqueRef:13839264-af8f-47dc-b235-533aa28810c7","OpaqueRef:bea9bc59-c762-447c-8145-961af6bf404b","OpaqueRef:ee0ab5bc-3d46-4edc-84b2-cf6f33ce060d","OpaqueRef:8b7690a2-7ac0-4b6e-955c-ede13892942a","OpaqueRef:ed322770-a47f-499e-8c6f-68dbca0627cf","OpaqueRef:4e02a573-5038-4667-b89f-aeef10f64b57","OpaqueRef:f86550ec-15f0-4d85-96e9-e2da1f9fdde3","OpaqueRef:282ccd71-14f2-49ed-b5e3-ad0ad2ea4182","OpaqueRef:02cc7f55-a453-40e9-8285-45a8eb97c4eb","OpaqueRef:a4af195f-2530-42a9-a684-8882097558ab","OpaqueRef:b1ec191e-a995-4c1b-8bac-dde94e355bb9","OpaqueRef:e4a5b5b9-4263-4061-b1a6-2241ff9df33e","OpaqueRef:15a31b91-5daa-4fdd-a128-64222be18599","OpaqueRef:a63dbbc2-43d2-4a17-81de-deef98c4ff70","OpaqueRef:865c00c1-ab16-4c36-83e0-26058bfc9d6f","OpaqueRef:f6c5abf4-b772-427c-b533-9e1dac402dd7","OpaqueRef:e4870fbc-9263-41a4-84fc-bcf747ce6326","OpaqueRef:ac916ff7-e936-4ae4-9841-9e07743881f9","OpaqueRef:029a2790-2795-444f-b0cf-3db6bd0195c2","OpaqueRef:33f83ad1-d849-48e5-995c-7f9b4a696ccd","OpaqueRef:deb97b8b-fff4-4192-ad8f-dfb26f1f8c88","OpaqueRef:dd32e3ce-e147-479b-b0ac-487d25db1384","OpaqueRef:1b9d353c-77bd-4caf-9ae0-64eb2887c06a","OpaqueRef:97fe2a88-fa69-48f6-9f52-a6b3eb39ba10","OpaqueRef:fa1ec86c-2f3a-49d5-b1b0-52787894153e","OpaqueRef:5d5e4dbe-a11f-4d10-aac6-34f806b52317","OpaqueRef:f65eccc8-5fd2-4439-8817-5736ef0c7c08","OpaqueRef:20755a4b-2ce8-4f29-bce8-5e5f058ddc71","OpaqueRef:da6f5842-1388-435c-a741-df424fe3216d","OpaqueRef:504cce7a-66a7-49bd-8c2a-76d093aa5328","OpaqueRef:3801bf53-3bc3-4de9-bdcc-51142bbfcd41"],"cpu_info":{"cpu_count":"32","socket_count":"2","vendor":"GenuineIntel","speed":"3200.329","modelname":"Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz","family":"6","model":"63","stepping":"2","flags":"fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms xsaveopt","features_pv":"1fc9cbf5-f6f83203-2991cbf5-00000023-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm":"1fcbfbff-f7fa3223-2d93fbff-00000423-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm_host":"1fcbfbff-f7fa3223-2c100800-00000021-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_pv_host":"1fc9cbf5-f6f83203-28100800-00000021-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"},"hostname":"blade7.quape.com","address":"","metrics":"OpaqueRef:1434bd31-37b0-416e-93f1-bdcbddafd324","license_params":{"restrict_vswitch_controller":"false","restrict_lab":"false","restrict_stage":"false","restrict_storagelink":"false","restrict_storagelink_site_recovery":"false","restrict_web_selfservice":"false","restrict_web_selfservice_manager":"false","restrict_hotfix_apply":"false","restrict_export_resource_data":"false","restrict_read_caching":"false","restrict_cifs":"false","restrict_health_check":"false","restrict_xcm":"false","restrict_vm_memory_introspection":"false","restrict_batch_hotfix_apply":"false","restrict_management_on_vlan":"false","restrict_ws_proxy":"false","restrict_vlan":"false","restrict_qos":"false","restrict_pool_attached_storage":"false","restrict_netapp":"false","restrict_equalogic":"false","restrict_pooling":"false","enable_xha":"true","restrict_marathon":"false","restrict_email_alerting":"false","restrict_historical_performance":"false","restrict_wlb":"false","restrict_rbac":"false","restrict_dmc":"false","restrict_checkpoint":"false","restrict_cpu_masking":"false","restrict_connection":"false","platform_filter":"false","regular_nag_dialog":"false","restrict_vmpr":"false","restrict_vmss":"false","restrict_intellicache":"false","restrict_gpu":"false","restrict_dr":"false","restrict_vif_locking":"false","restrict_storage_xen_motion":"false","restrict_vgpu":"false","restrict_integrated_gpu_passthrough":"false","restrict_vss":"false","restrict_guest_agent_auto_update":"false","restrict_pci_device_for_auto_update":"false","restrict_xen_motion":"false","restrict_guest_ip_setting":"false","restrict_ad":"false","restrict_nested_virt":"false","restrict_live_patching":"false","restrict_set_vcpus_number_live":"false","restrict_pvs_proxy":"false","restrict_igmp_snooping":"false","restrict_rpu":"false","restrict_pool_size":"false","restrict_cbt":"false","restrict_usb_passthrough":"false","restrict_network_sriov":"false","restrict_corosync":"true","restrict_zstd_export":"false","restrict_pool_secret_rotation":"false"},"ha_statefiles":[],"ha_network_peers":[],"blobs":{},"tags":[],"external_auth_type":"","external_auth_service_name":"","external_auth_configuration":{},"edition":"xcp-ng","license_server":{"address":"localhost","port":"27000"},"bios_strings":{"bios-vendor":"Dell Inc.","bios-version":"2.15.0","system-manufacturer":"Dell Inc.","system-product-name":"PowerEdge FC630","system-version":"","system-serial-number":"F94H182","baseboard-manufacturer":"Dell Inc.","baseboard-product-name":"0JXJPT","baseboard-version":"A02","baseboard-serial-number":".F94H182.CN7016354O001T.03.","oem-1":"Xen","oem-2":"MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d","oem-3":"Dell System","oem-4":"5[0000]","oem-5":"14[1]","oem-6":"17[6D0BB6C5CA6B4566]","oem-7":"17[6D0C10A7A9419261]","oem-8":"18[0]","oem-9":"19[1]","oem-10":"19[1]","hp-rombios":""},"power_on_mode":"","power_on_config":{},"local_cache_sr":"OpaqueRef:NULL","chipset_info":{"iommu":"true"},"PCIs":["OpaqueRef:d0f241a8-4daa-4501-b3cf-adb0892cc89b","OpaqueRef:a04776fb-5068-4ac7-8bd2-9d5771e1f852","OpaqueRef:92c5c165-8350-4b59-abf5-faed53c18098","OpaqueRef:8aec8abe-10c7-4927-804b-6cbb34e36b0b","OpaqueRef:6b676c66-16f3-484d-a091-ea0dee4bf8a8","OpaqueRef:644d2d29-5dbf-40e7-8416-73005db02084","OpaqueRef:34d72682-8f23-4236-a2c4-d4b2b5d32f26","OpaqueRef:29a4690d-bda1-4d5c-b4d1-17b62bb93df3","OpaqueRef:0779a461-8555-4f99-b967-b2edb5384afd"],"PGPUs":["OpaqueRef:d4dfdda0-964f-4421-9612-8019572b5e80"],"PUSBs":["OpaqueRef:4a3f6f0c-1ba1-4bc2-9d10-eb058a68268f"],"ssl_legacy":false,"guest_VCPUs_params":{},"display":"enabled","virtual_hardware_platform_versions":[0,1,2],"control_domain":"OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003","updates_requiring_reboot":[],"features":[],"iscsi_iqn":"iqn.2022-10.com.quape:88df7156","multipathing":false,"uefi_certificates":"","certificates":[],"editions":["xcp-ng"]},"SR":{"uuid":"73bf933d-050d-453c-5b43-1f464221678c","name_label":"SSD1","name_description":"","allowed_operations":["vdi_enable_cbt","vdi_list_changed_blocks","unplug","plug","pbd_create","vdi_disable_cbt","update","pbd_destroy","vdi_resize","vdi_clone","vdi_data_destroy","scan","vdi_snapshot","vdi_mirror","vdi_create","vdi_destroy","vdi_set_on_boot"],"current_operations":{},"VDIs":["OpaqueRef:f4a60bad-36da-46cb-a555-8e76ff2f3dd6"],"PBDs":["OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3"],"virtual_allocation":322122547200,"physical_utilisation":80355328,"physical_size":1007984996352,"type":"ext","content_type":"user","shared":false,"other_config":{},"tags":[],"sm_config":{"devserial":"scsi-357c35481e852d688"},"blobs":{},"local_cache_enabled":false,"introduced_by":"OpaqueRef:NULL","clustered":false,"is_tools_sr":false},"VDI":{"uuid":"ab00cbc9-7da2-443e-b145-b8a907abed3e","name_label":"quape.net 0","name_description":"Created by template provisioner","allowed_operations":["generate_config","update","forget","destroy","snapshot","resize","copy","clone"],"current_operations":{},"SR":"OpaqueRef:fed4ee9c-d397-4087-a4f8-902da574d30c","VBDs":["OpaqueRef:6ef02723-5995-4699-993e-2e0c8bbff868"],"crash_dumps":[],"virtual_size":322122547200,"physical_utilisation":636416,"type":"system","sharable":false,"read_only":false,"other_config":{"content_id":"d111595a-6f0d-53fa-07a9-97ac599cc436","xo:copy_of":"f8004138-6815-4c17-b31c-c00083542337"},"storage_lock":false,"location":"ab00cbc9-7da2-443e-b145-b8a907abed3e","managed":true,"missing":false,"parent":"OpaqueRef:NULL","xenstore_data":{},"sm_config":{},"is_a_snapshot":false,"snapshot_of":"OpaqueRef:NULL","snapshots":[],"snapshot_time":"19700101T00:00:00Z","tags":[],"allow_caching":false,"on_boot":"persist","metadata_of_pool":"","metadata_latest":false,"is_tools_iso":false,"cbt_enabled":false},"message":"VDI_IO_ERROR(Device I/O errors)","name":"XapiError","stack":"XapiError: VDI_IO_ERROR(Device I/O errors)\n at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)\n at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)\n at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:954:37)\n at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:988:14)\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:978:12)\n at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1144:14)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at processImmediate (node:internal/timers:447:9)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)"} -
I did quite a number of test and these are all fresh install and the codes for node js fix is there. I'm using the restore function to local SR ( ext4 ) Is there some other codes that needs to be updated ? Anyone have similar issue ?
-
@EddieCh08666741 Hope my above logs can help team to see if there's a bug in the script.
-
I want to share my findings. I did a clean install XOCE on several distros ubuntu,debian and the 5mins restore still happens. I've tried both commits one with the requesttimeout=0 and the other requesttimeout=86xxxx running on node 18. I've installed XOA stable version which is running lower version of node runs perfectly fine always 100% on every restore.
-
want to share that I fix this issue by upgrading NPM to latest stable version.
-
Okay thanks for your feedback. Do you remember the previous npm version and what's your current one?
-
seems i spoke too soon lol. Tested another VM and got this same 5mins A "socket" was not created for HTTP request before 300000ms
-
So. theare is a fix, i have this issue also, clean install
-
I got the same error "Error: A "socket" was not created for HTTP request before 300000ms". My two XCP-ng hosts are 8.2.1 with a clean install & applied all patches. I still don't know how to fix it.
-
i tryed eveything
1 downgrade orchestra nodejs to v16 - same
2 reinstaled xcp-ng clean - same
-
I've been testing with many variations all clean install from XO from sources all stopped at 5mins. installed 3-4 fresh copy. The only one which work consistently is the node 16 xoa stable version.
-
@EddieCh08666741
what branch in git ? -
@arsenieciprian the official supported XOA.
-
Hello,
I'm having the same issue in Delta Backups.
Opened here a topic here https://xcp-ng.org/forum/topic/7007/a-socket-was-not-created-for-http-request-before-300000ms -
@BrunoMorais I had the same error yesterday, I rolled back to a previous commit and it started working again. I don't know the exact commit right now, sorry
(Note that this is not the same error as "vdi_io_error" and the cause it's probably also not the same) -
Are you sure you are on the latest commit on
masterand did a rebuild?
