VDI_IO_ERROR Continuous Replication on clean install.
-
if i have no smlog - xen\dom0 not related with backup task. right?
smlog that usualy i got during this 5min have no any errors anyway, only some locking operations.
And it always takes 5min, some hardcoded timings?don't forget that problem also happens with FC connection, so it may concern any block based storage types.
-
I don't understand your sentence, can you take time to re-read it or rephrase it, because I doesn't make sense to me, sorry
What do you mean by "if i have no smlog - xen\dom0 not related with backup task. right?"?
-
i mean it could be XO issue, since it not communicate with xen. Otherwise it should write some logs.
-
I don't see the logical connection with XO, since it works on some SR and not on others. XO has no idea (or doesn't care) about the underlying storage.
-
well, i'm just made some tests and got some result. Have no idea how it should work)
-
i don't understand what happens.
Reinstalled xen to 8.2.0, CR was succeed for few times, but now i got this error again.Tried few tests - 2-3 fails in row then it succeed again.
Only way to never use this pool for CR. -
I also have some fresh installation 8.2.1 with similar error at 5 mins 2seconds 5 min 1 sec
-
On my side, yesterday I did the only test I haven't done so far: Installing XenOrchestra in a NON xcp-ng server.
Basically, since always, I had a separated XCP-NG server with just a single VM inside: The XO VM (Just in case, that VM was Ubuntu, Centos, Debian over time, so the base OS has nothing to do with this).
My solution for this was simple: Bare metal Linux. So the problem wasn't the XCP version on the source server, nor the destination server. It was the host server of the XO VM itself
Why? I have no idea, but it's definitely working now since I started a CR task yesterday, of a 1TB VM, with a destination server over internet, and is still exporting after 14 hours without any issues:
-
@yomono And to clarify, for this XO host, I tested 8.2.0, 8.2.1, and 8.3.0 fresh installs and all failed at exactly 5 minutes
-
This means your Node version was still using the default timeout.
-
@olivierlambert when you say "node" you mean node.js? How that timeout can be changed? Thanks
-
changing the node will fix this vdi error ?
-
@EddieCh08666741 The one which works is the fresh install without any updates.
-
@yomono NodeJS yes. In Node 18, they made a breaking change to put by default a 5 minutes timeout.
We fixed that by adding a specific config to get a longer timeout, see https://github.com/vatesfr/xen-orchestra/commit/f6fd1db1ef12633cc5bb8ec8ab5bc84682dd3fe7
Without this piece of config, you'll end any HTTP stream after exactly 5 minutes.
-
@olivierlambert Well, I checked the config.toml and, indeed, I can confirm the fix is there.
What's really odd here is that I'm using the exact same installation script for XO that I've used in the past.
So, why now this value is correct but it wasn't when executing the exact same script in a VM? (Not an actual question for you, but just wondering out loud)
So @EddieCh08666741 and @Tristis-Oris you were having an exact 5 minutes timeout too so you might want to check this config as well
Thanks for the support, Olivier -
my config is fine, but problem still here for this one pool.
-
@Tristis-Oris i double check and the codes mentioned is updated. The issue is the same for doing the restore from delta backups.
-
{"code":"VDI_IO_ERROR","params":["Device I/O errors"],"url":"https://192.168.0.100/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Af4a60bad-36da-46cb-a555-8e76ff2f3dd6&session_id=OpaqueRef%3Aba2fcb1d-bc46-445b-9540-b0e9e51236ee&task_id=OpaqueRef%3A8ca30f05-cc13-4c04-907f-6b4468a6ace2","task":{"uuid":"fa99f516-2352-3b92-7419-4950559458d8","name_label":"[XO] Importing content into VDI 0","name_description":"","allowed_operations":[],"current_operations":{},"created":"20230204T06:49:36Z","finished":"20230204T06:54:36Z","status":"failure","resident_on":"OpaqueRef:dba25009-ad21-4aad-a7d6-0b03af29a906","progress":1,"type":"<none/>","result":"","error_info":["VDI_IO_ERROR","Device I/O errors"],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename ocaml/xapi/vhd_tool_wrapper.ml)(line 77))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/import_raw_vdi.ml)(line 170)))"},"pool_master":{"uuid":"e16ba86e-9915-446d-824e-f2246a2711ad","name_label":"blade7.quape.com","name_description":"Default install","memory_overhead":2050928640,"allowed_operations":["vm_migrate","provision","vm_resume","evacuate","vm_start"],"current_operations":{},"API_version_major":2,"API_version_minor":16,"API_version_vendor":"XenSource","API_version_vendor_implementation":{},"enabled":true,"software_version":{"product_version":"8.2.1","product_version_text":"8.2","product_version_text_short":"8.2","platform_name":"XCP","platform_version":"3.2.1","product_brand":"XCP-ng","build_number":"release/yangtze/master/58","hostname":"localhost","date":"2022-10-12","dbv":"0.0.1","xapi":"1.20","xen":"4.13.4-9.19.1","linux":"4.19.0+1","xencenter_min":"2.16","xencenter_max":"2.16","network_backend":"openvswitch","db_schema":"5.602"},"other_config":{"agent_start_time":"1670522053.","boot_time":"1670465798.","rpm_patch_installation_time":"1670522042.976","iscsi_iqn":"iqn.2022-10.com.quape:88df7156"},"capabilities":["xen-3.0-x86_64","hvm-3.0-x86_32","hvm-3.0-x86_32p","hvm-3.0-x86_64",""],"cpu_configuration":{},"sched_policy":"credit","supported_bootloaders":["pygrub","eliloader"],"resident_VMs":["OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003"],"logging":{},"PIFs":["OpaqueRef:bb16676d-1c45-4363-9550-ffc6f2323239","OpaqueRef:ae69076b-b4c6-4711-9ab2-95b135daf2d7","OpaqueRef:171c0df8-5b44-459e-8756-01b70115586d","OpaqueRef:12f18eda-5b70-4735-b95e-316fc901d635"],"suspend_image_sr":"OpaqueRef:NULL","crash_dump_sr":"OpaqueRef:NULL","crashdumps":[],"patches":[],"updates":[],"PBDs":["OpaqueRef:08cf61cc-234c-4f62-b006-05fc9f14d27b","OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3","OpaqueRef:064440da-c058-491d-85b6-e9faa1166878"],"host_CPUs":["OpaqueRef:060a9fa7-302b-4f3e-9125-c51c649cc080","OpaqueRef:13839264-af8f-47dc-b235-533aa28810c7","OpaqueRef:bea9bc59-c762-447c-8145-961af6bf404b","OpaqueRef:ee0ab5bc-3d46-4edc-84b2-cf6f33ce060d","OpaqueRef:8b7690a2-7ac0-4b6e-955c-ede13892942a","OpaqueRef:ed322770-a47f-499e-8c6f-68dbca0627cf","OpaqueRef:4e02a573-5038-4667-b89f-aeef10f64b57","OpaqueRef:f86550ec-15f0-4d85-96e9-e2da1f9fdde3","OpaqueRef:282ccd71-14f2-49ed-b5e3-ad0ad2ea4182","OpaqueRef:02cc7f55-a453-40e9-8285-45a8eb97c4eb","OpaqueRef:a4af195f-2530-42a9-a684-8882097558ab","OpaqueRef:b1ec191e-a995-4c1b-8bac-dde94e355bb9","OpaqueRef:e4a5b5b9-4263-4061-b1a6-2241ff9df33e","OpaqueRef:15a31b91-5daa-4fdd-a128-64222be18599","OpaqueRef:a63dbbc2-43d2-4a17-81de-deef98c4ff70","OpaqueRef:865c00c1-ab16-4c36-83e0-26058bfc9d6f","OpaqueRef:f6c5abf4-b772-427c-b533-9e1dac402dd7","OpaqueRef:e4870fbc-9263-41a4-84fc-bcf747ce6326","OpaqueRef:ac916ff7-e936-4ae4-9841-9e07743881f9","OpaqueRef:029a2790-2795-444f-b0cf-3db6bd0195c2","OpaqueRef:33f83ad1-d849-48e5-995c-7f9b4a696ccd","OpaqueRef:deb97b8b-fff4-4192-ad8f-dfb26f1f8c88","OpaqueRef:dd32e3ce-e147-479b-b0ac-487d25db1384","OpaqueRef:1b9d353c-77bd-4caf-9ae0-64eb2887c06a","OpaqueRef:97fe2a88-fa69-48f6-9f52-a6b3eb39ba10","OpaqueRef:fa1ec86c-2f3a-49d5-b1b0-52787894153e","OpaqueRef:5d5e4dbe-a11f-4d10-aac6-34f806b52317","OpaqueRef:f65eccc8-5fd2-4439-8817-5736ef0c7c08","OpaqueRef:20755a4b-2ce8-4f29-bce8-5e5f058ddc71","OpaqueRef:da6f5842-1388-435c-a741-df424fe3216d","OpaqueRef:504cce7a-66a7-49bd-8c2a-76d093aa5328","OpaqueRef:3801bf53-3bc3-4de9-bdcc-51142bbfcd41"],"cpu_info":{"cpu_count":"32","socket_count":"2","vendor":"GenuineIntel","speed":"3200.329","modelname":"Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz","family":"6","model":"63","stepping":"2","flags":"fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms xsaveopt","features_pv":"1fc9cbf5-f6f83203-2991cbf5-00000023-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm":"1fcbfbff-f7fa3223-2d93fbff-00000423-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm_host":"1fcbfbff-f7fa3223-2c100800-00000021-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_pv_host":"1fc9cbf5-f6f83203-28100800-00000021-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"},"hostname":"blade7.quape.com","address":"","metrics":"OpaqueRef:1434bd31-37b0-416e-93f1-bdcbddafd324","license_params":{"restrict_vswitch_controller":"false","restrict_lab":"false","restrict_stage":"false","restrict_storagelink":"false","restrict_storagelink_site_recovery":"false","restrict_web_selfservice":"false","restrict_web_selfservice_manager":"false","restrict_hotfix_apply":"false","restrict_export_resource_data":"false","restrict_read_caching":"false","restrict_cifs":"false","restrict_health_check":"false","restrict_xcm":"false","restrict_vm_memory_introspection":"false","restrict_batch_hotfix_apply":"false","restrict_management_on_vlan":"false","restrict_ws_proxy":"false","restrict_vlan":"false","restrict_qos":"false","restrict_pool_attached_storage":"false","restrict_netapp":"false","restrict_equalogic":"false","restrict_pooling":"false","enable_xha":"true","restrict_marathon":"false","restrict_email_alerting":"false","restrict_historical_performance":"false","restrict_wlb":"false","restrict_rbac":"false","restrict_dmc":"false","restrict_checkpoint":"false","restrict_cpu_masking":"false","restrict_connection":"false","platform_filter":"false","regular_nag_dialog":"false","restrict_vmpr":"false","restrict_vmss":"false","restrict_intellicache":"false","restrict_gpu":"false","restrict_dr":"false","restrict_vif_locking":"false","restrict_storage_xen_motion":"false","restrict_vgpu":"false","restrict_integrated_gpu_passthrough":"false","restrict_vss":"false","restrict_guest_agent_auto_update":"false","restrict_pci_device_for_auto_update":"false","restrict_xen_motion":"false","restrict_guest_ip_setting":"false","restrict_ad":"false","restrict_nested_virt":"false","restrict_live_patching":"false","restrict_set_vcpus_number_live":"false","restrict_pvs_proxy":"false","restrict_igmp_snooping":"false","restrict_rpu":"false","restrict_pool_size":"false","restrict_cbt":"false","restrict_usb_passthrough":"false","restrict_network_sriov":"false","restrict_corosync":"true","restrict_zstd_export":"false","restrict_pool_secret_rotation":"false"},"ha_statefiles":[],"ha_network_peers":[],"blobs":{},"tags":[],"external_auth_type":"","external_auth_service_name":"","external_auth_configuration":{},"edition":"xcp-ng","license_server":{"address":"localhost","port":"27000"},"bios_strings":{"bios-vendor":"Dell Inc.","bios-version":"2.15.0","system-manufacturer":"Dell Inc.","system-product-name":"PowerEdge FC630","system-version":"","system-serial-number":"F94H182","baseboard-manufacturer":"Dell Inc.","baseboard-product-name":"0JXJPT","baseboard-version":"A02","baseboard-serial-number":".F94H182.CN7016354O001T.03.","oem-1":"Xen","oem-2":"MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d","oem-3":"Dell System","oem-4":"5[0000]","oem-5":"14[1]","oem-6":"17[6D0BB6C5CA6B4566]","oem-7":"17[6D0C10A7A9419261]","oem-8":"18[0]","oem-9":"19[1]","oem-10":"19[1]","hp-rombios":""},"power_on_mode":"","power_on_config":{},"local_cache_sr":"OpaqueRef:NULL","chipset_info":{"iommu":"true"},"PCIs":["OpaqueRef:d0f241a8-4daa-4501-b3cf-adb0892cc89b","OpaqueRef:a04776fb-5068-4ac7-8bd2-9d5771e1f852","OpaqueRef:92c5c165-8350-4b59-abf5-faed53c18098","OpaqueRef:8aec8abe-10c7-4927-804b-6cbb34e36b0b","OpaqueRef:6b676c66-16f3-484d-a091-ea0dee4bf8a8","OpaqueRef:644d2d29-5dbf-40e7-8416-73005db02084","OpaqueRef:34d72682-8f23-4236-a2c4-d4b2b5d32f26","OpaqueRef:29a4690d-bda1-4d5c-b4d1-17b62bb93df3","OpaqueRef:0779a461-8555-4f99-b967-b2edb5384afd"],"PGPUs":["OpaqueRef:d4dfdda0-964f-4421-9612-8019572b5e80"],"PUSBs":["OpaqueRef:4a3f6f0c-1ba1-4bc2-9d10-eb058a68268f"],"ssl_legacy":false,"guest_VCPUs_params":{},"display":"enabled","virtual_hardware_platform_versions":[0,1,2],"control_domain":"OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003","updates_requiring_reboot":[],"features":[],"iscsi_iqn":"iqn.2022-10.com.quape:88df7156","multipathing":false,"uefi_certificates":"","certificates":[],"editions":["xcp-ng"]},"SR":{"uuid":"73bf933d-050d-453c-5b43-1f464221678c","name_label":"SSD1","name_description":"","allowed_operations":["vdi_enable_cbt","vdi_list_changed_blocks","unplug","plug","pbd_create","vdi_disable_cbt","update","pbd_destroy","vdi_resize","vdi_clone","vdi_data_destroy","scan","vdi_snapshot","vdi_mirror","vdi_create","vdi_destroy","vdi_set_on_boot"],"current_operations":{},"VDIs":["OpaqueRef:f4a60bad-36da-46cb-a555-8e76ff2f3dd6"],"PBDs":["OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3"],"virtual_allocation":322122547200,"physical_utilisation":80355328,"physical_size":1007984996352,"type":"ext","content_type":"user","shared":false,"other_config":{},"tags":[],"sm_config":{"devserial":"scsi-357c35481e852d688"},"blobs":{},"local_cache_enabled":false,"introduced_by":"OpaqueRef:NULL","clustered":false,"is_tools_sr":false},"VDI":{"uuid":"ab00cbc9-7da2-443e-b145-b8a907abed3e","name_label":"quape.net 0","name_description":"Created by template provisioner","allowed_operations":["generate_config","update","forget","destroy","snapshot","resize","copy","clone"],"current_operations":{},"SR":"OpaqueRef:fed4ee9c-d397-4087-a4f8-902da574d30c","VBDs":["OpaqueRef:6ef02723-5995-4699-993e-2e0c8bbff868"],"crash_dumps":[],"virtual_size":322122547200,"physical_utilisation":636416,"type":"system","sharable":false,"read_only":false,"other_config":{"content_id":"d111595a-6f0d-53fa-07a9-97ac599cc436","xo:copy_of":"f8004138-6815-4c17-b31c-c00083542337"},"storage_lock":false,"location":"ab00cbc9-7da2-443e-b145-b8a907abed3e","managed":true,"missing":false,"parent":"OpaqueRef:NULL","xenstore_data":{},"sm_config":{},"is_a_snapshot":false,"snapshot_of":"OpaqueRef:NULL","snapshots":[],"snapshot_time":"19700101T00:00:00Z","tags":[],"allow_caching":false,"on_boot":"persist","metadata_of_pool":"","metadata_latest":false,"is_tools_iso":false,"cbt_enabled":false},"message":"VDI_IO_ERROR(Device I/O errors)","name":"XapiError","stack":"XapiError: VDI_IO_ERROR(Device I/O errors)\n at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)\n at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)\n at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:954:37)\n at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:988:14)\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:978:12)\n at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1144:14)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at processImmediate (node:internal/timers:447:9)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)"} -
I did quite a number of test and these are all fresh install and the codes for node js fix is there. I'm using the restore function to local SR ( ext4 ) Is there some other codes that needs to be updated ? Anyone have similar issue ?
-
@EddieCh08666741 Hope my above logs can help team to see if there's a bug in the script.
