VDI_IO_ERROR Continuous Replication on clean install.

Tristis Oris

@olivierlambert
sr_not_supported that not a error and not a reason. That because of default multipath Dell config for 3xxx series. Persist at 8.2.0 where CR working, so that just a warning.
As we have no any problems before, we never investigate to this setting. My bad again yay.

Replaced it to official for 4xxx and this warning gone. I see at 8.3 it already more universal for any generation.

        device {
                vendor "DellEMC"
                product "ME4"
                path_grouping_policy "group_by_prio"
                path_checker "tur"
                hardware_handler "1 alua"
                prio "alua"
                failback immediate
                path_selector "service-time 0"
        }

since it no default config for huawei, so we always used the official one.

        device {
                vendor                  "HUAWEI"
                product                 "XSG1"
                path_grouping_policy multibus
                path_checker            tur
                prio                    const
                path_selector           "round-robin 0"
                failback                immediate
                fast_io_fail_tmo        5
                dev_loss_tmo            30
        }

8.2.1:
CR not working:
both huawei, dell iscsi - multipath enabled
both huawei, dell iscsi - multipath disabled
working:
nfs vm disk
local thin\ext
local thick\lvm
8.3
working:
both huawei, dell iscsi - multipath enabled
local thick\lvm

and now interesting. After i solved this false warning, detach extra hosts from pool, detach all additional links (trunk, backup) to decrease comunications and log itself - it's no any SMlog generated during backup task.

MP enabled - with 2nd link for backup https://pastebin.com/URcnDckR
MP enabled - only Mng link, no SMlog generated https://pastebin.com/RHw40uzg

olivierlambert

I have the impression it's good news, but I'm not 100% sure to get it, can you rephrase a bit your conclusion?

Tristis Oris

if i have no smlog - xen\dom0 not related with backup task. right?
smlog that usualy i got during this 5min have no any errors anyway, only some locking operations.
And it always takes 5min, some hardcoded timings?

don't forget that problem also happens with FC connection, so it may concern any block based storage types.

olivierlambert

I don't understand your sentence, can you take time to re-read it or rephrase it, because I doesn't make sense to me, sorry

What do you mean by "if i have no smlog - xen\dom0 not related with backup task. right?"?

Tristis Oris

i mean it could be XO issue, since it not communicate with xen. Otherwise it should write some logs.

olivierlambert

I don't see the logical connection with XO, since it works on some SR and not on others. XO has no idea (or doesn't care) about the underlying storage.

Tristis Oris

well, i'm just made some tests and got some result. Have no idea how it should work)

Tristis Oris

i don't understand what happens.
Reinstalled xen to 8.2.0, CR was succeed for few times, but now i got this error again.

Tried few tests - 2-3 fails in row then it succeed again.
Only way to never use this pool for CR.

EddieCh08666741

I also have some fresh installation 8.2.1 with similar error at 5 mins 2seconds 5 min 1 sec

yomono

On my side, yesterday I did the only test I haven't done so far: Installing XenOrchestra in a NON xcp-ng server.
Basically, since always, I had a separated XCP-NG server with just a single VM inside: The XO VM (Just in case, that VM was Ubuntu, Centos, Debian over time, so the base OS has nothing to do with this).
My solution for this was simple: Bare metal Linux. So the problem wasn't the XCP version on the source server, nor the destination server. It was the host server of the XO VM itself
Why? I have no idea, but it's definitely working now since I started a CR task yesterday, of a 1TB VM, with a destination server over internet, and is still exporting after 14 hours without any issues:

yomono

@yomono And to clarify, for this XO host, I tested 8.2.0, 8.2.1, and 8.3.0 fresh installs and all failed at exactly 5 minutes

olivierlambert

This means your Node version was still using the default timeout.

yomono

@olivierlambert when you say "node" you mean node.js? How that timeout can be changed? Thanks

EddieCh08666741

changing the node will fix this vdi error ?

EddieCh08666741

@EddieCh08666741 The one which works is the fresh install without any updates.

olivierlambert

@yomono NodeJS yes. In Node 18, they made a breaking change to put by default a 5 minutes timeout.

We fixed that by adding a specific config to get a longer timeout, see https://github.com/vatesfr/xen-orchestra/commit/f6fd1db1ef12633cc5bb8ec8ab5bc84682dd3fe7

Without this piece of config, you'll end any HTTP stream after exactly 5 minutes.

0

julien-f committed to vatesfr/xen-orchestra

feat(xo-server): increase HTTP server request timeout to 1 day

Fixes #6590

yomono

@olivierlambert Well, I checked the config.toml and, indeed, I can confirm the fix is there.
What's really odd here is that I'm using the exact same installation script for XO that I've used in the past.
So, why now this value is correct but it wasn't when executing the exact same script in a VM? (Not an actual question for you, but just wondering out loud)
So @EddieCh08666741 and @Tristis-Oris you were having an exact 5 minutes timeout too so you might want to check this config as well
Thanks for the support, Olivier

Tristis Oris

my config is fine, but problem still here for this one pool.

EddieCh08666741

@Tristis-Oris i double check and the codes mentioned is updated. The issue is the same for doing the restore from delta backups.

EddieCh08666741

{"code":"VDI_IO_ERROR","params":["Device I/O errors"],"url":"https://192.168.0.100/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Af4a60bad-36da-46cb-a555-8e76ff2f3dd6&session_id=OpaqueRef%3Aba2fcb1d-bc46-445b-9540-b0e9e51236ee&task_id=OpaqueRef%3A8ca30f05-cc13-4c04-907f-6b4468a6ace2","task":{"uuid":"fa99f516-2352-3b92-7419-4950559458d8","name_label":"[XO] Importing content into VDI 0","name_description":"","allowed_operations":[],"current_operations":{},"created":"20230204T06:49:36Z","finished":"20230204T06:54:36Z","status":"failure","resident_on":"OpaqueRef:dba25009-ad21-4aad-a7d6-0b03af29a906","progress":1,"type":"<none/>","result":"","error_info":["VDI_IO_ERROR","Device I/O errors"],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename ocaml/xapi/vhd_tool_wrapper.ml)(line 77))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/import_raw_vdi.ml)(line 170)))"},"pool_master":{"uuid":"e16ba86e-9915-446d-824e-f2246a2711ad","name_label":"blade7.quape.com","name_description":"Default install","memory_overhead":2050928640,"allowed_operations":["vm_migrate","provision","vm_resume","evacuate","vm_start"],"current_operations":{},"API_version_major":2,"API_version_minor":16,"API_version_vendor":"XenSource","API_version_vendor_implementation":{},"enabled":true,"software_version":{"product_version":"8.2.1","product_version_text":"8.2","product_version_text_short":"8.2","platform_name":"XCP","platform_version":"3.2.1","product_brand":"XCP-ng","build_number":"release/yangtze/master/58","hostname":"localhost","date":"2022-10-12","dbv":"0.0.1","xapi":"1.20","xen":"4.13.4-9.19.1","linux":"4.19.0+1","xencenter_min":"2.16","xencenter_max":"2.16","network_backend":"openvswitch","db_schema":"5.602"},"other_config":{"agent_start_time":"1670522053.","boot_time":"1670465798.","rpm_patch_installation_time":"1670522042.976","iscsi_iqn":"iqn.2022-10.com.quape:88df7156"},"capabilities":["xen-3.0-x86_64","hvm-3.0-x86_32","hvm-3.0-x86_32p","hvm-3.0-x86_64",""],"cpu_configuration":{},"sched_policy":"credit","supported_bootloaders":["pygrub","eliloader"],"resident_VMs":["OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003"],"logging":{},"PIFs":["OpaqueRef:bb16676d-1c45-4363-9550-ffc6f2323239","OpaqueRef:ae69076b-b4c6-4711-9ab2-95b135daf2d7","OpaqueRef:171c0df8-5b44-459e-8756-01b70115586d","OpaqueRef:12f18eda-5b70-4735-b95e-316fc901d635"],"suspend_image_sr":"OpaqueRef:NULL","crash_dump_sr":"OpaqueRef:NULL","crashdumps":[],"patches":[],"updates":[],"PBDs":["OpaqueRef:08cf61cc-234c-4f62-b006-05fc9f14d27b","OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3","OpaqueRef:064440da-c058-491d-85b6-e9faa1166878"],"host_CPUs":["OpaqueRef:060a9fa7-302b-4f3e-9125-c51c649cc080","OpaqueRef:13839264-af8f-47dc-b235-533aa28810c7","OpaqueRef:bea9bc59-c762-447c-8145-961af6bf404b","OpaqueRef:ee0ab5bc-3d46-4edc-84b2-cf6f33ce060d","OpaqueRef:8b7690a2-7ac0-4b6e-955c-ede13892942a","OpaqueRef:ed322770-a47f-499e-8c6f-68dbca0627cf","OpaqueRef:4e02a573-5038-4667-b89f-aeef10f64b57","OpaqueRef:f86550ec-15f0-4d85-96e9-e2da1f9fdde3","OpaqueRef:282ccd71-14f2-49ed-b5e3-ad0ad2ea4182","OpaqueRef:02cc7f55-a453-40e9-8285-45a8eb97c4eb","OpaqueRef:a4af195f-2530-42a9-a684-8882097558ab","OpaqueRef:b1ec191e-a995-4c1b-8bac-dde94e355bb9","OpaqueRef:e4a5b5b9-4263-4061-b1a6-2241ff9df33e","OpaqueRef:15a31b91-5daa-4fdd-a128-64222be18599","OpaqueRef:a63dbbc2-43d2-4a17-81de-deef98c4ff70","OpaqueRef:865c00c1-ab16-4c36-83e0-26058bfc9d6f","OpaqueRef:f6c5abf4-b772-427c-b533-9e1dac402dd7","OpaqueRef:e4870fbc-9263-41a4-84fc-bcf747ce6326","OpaqueRef:ac916ff7-e936-4ae4-9841-9e07743881f9","OpaqueRef:029a2790-2795-444f-b0cf-3db6bd0195c2","OpaqueRef:33f83ad1-d849-48e5-995c-7f9b4a696ccd","OpaqueRef:deb97b8b-fff4-4192-ad8f-dfb26f1f8c88","OpaqueRef:dd32e3ce-e147-479b-b0ac-487d25db1384","OpaqueRef:1b9d353c-77bd-4caf-9ae0-64eb2887c06a","OpaqueRef:97fe2a88-fa69-48f6-9f52-a6b3eb39ba10","OpaqueRef:fa1ec86c-2f3a-49d5-b1b0-52787894153e","OpaqueRef:5d5e4dbe-a11f-4d10-aac6-34f806b52317","OpaqueRef:f65eccc8-5fd2-4439-8817-5736ef0c7c08","OpaqueRef:20755a4b-2ce8-4f29-bce8-5e5f058ddc71","OpaqueRef:da6f5842-1388-435c-a741-df424fe3216d","OpaqueRef:504cce7a-66a7-49bd-8c2a-76d093aa5328","OpaqueRef:3801bf53-3bc3-4de9-bdcc-51142bbfcd41"],"cpu_info":{"cpu_count":"32","socket_count":"2","vendor":"GenuineIntel","speed":"3200.329","modelname":"Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz","family":"6","model":"63","stepping":"2","flags":"fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms xsaveopt","features_pv":"1fc9cbf5-f6f83203-2991cbf5-00000023-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm":"1fcbfbff-f7fa3223-2d93fbff-00000423-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm_host":"1fcbfbff-f7fa3223-2c100800-00000021-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_pv_host":"1fc9cbf5-f6f83203-28100800-00000021-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"},"hostname":"blade7.quape.com","address":"","metrics":"OpaqueRef:1434bd31-37b0-416e-93f1-bdcbddafd324","license_params":{"restrict_vswitch_controller":"false","restrict_lab":"false","restrict_stage":"false","restrict_storagelink":"false","restrict_storagelink_site_recovery":"false","restrict_web_selfservice":"false","restrict_web_selfservice_manager":"false","restrict_hotfix_apply":"false","restrict_export_resource_data":"false","restrict_read_caching":"false","restrict_cifs":"false","restrict_health_check":"false","restrict_xcm":"false","restrict_vm_memory_introspection":"false","restrict_batch_hotfix_apply":"false","restrict_management_on_vlan":"false","restrict_ws_proxy":"false","restrict_vlan":"false","restrict_qos":"false","restrict_pool_attached_storage":"false","restrict_netapp":"false","restrict_equalogic":"false","restrict_pooling":"false","enable_xha":"true","restrict_marathon":"false","restrict_email_alerting":"false","restrict_historical_performance":"false","restrict_wlb":"false","restrict_rbac":"false","restrict_dmc":"false","restrict_checkpoint":"false","restrict_cpu_masking":"false","restrict_connection":"false","platform_filter":"false","regular_nag_dialog":"false","restrict_vmpr":"false","restrict_vmss":"false","restrict_intellicache":"false","restrict_gpu":"false","restrict_dr":"false","restrict_vif_locking":"false","restrict_storage_xen_motion":"false","restrict_vgpu":"false","restrict_integrated_gpu_passthrough":"false","restrict_vss":"false","restrict_guest_agent_auto_update":"false","restrict_pci_device_for_auto_update":"false","restrict_xen_motion":"false","restrict_guest_ip_setting":"false","restrict_ad":"false","restrict_nested_virt":"false","restrict_live_patching":"false","restrict_set_vcpus_number_live":"false","restrict_pvs_proxy":"false","restrict_igmp_snooping":"false","restrict_rpu":"false","restrict_pool_size":"false","restrict_cbt":"false","restrict_usb_passthrough":"false","restrict_network_sriov":"false","restrict_corosync":"true","restrict_zstd_export":"false","restrict_pool_secret_rotation":"false"},"ha_statefiles":[],"ha_network_peers":[],"blobs":{},"tags":[],"external_auth_type":"","external_auth_service_name":"","external_auth_configuration":{},"edition":"xcp-ng","license_server":{"address":"localhost","port":"27000"},"bios_strings":{"bios-vendor":"Dell Inc.","bios-version":"2.15.0","system-manufacturer":"Dell Inc.","system-product-name":"PowerEdge FC630","system-version":"","system-serial-number":"F94H182","baseboard-manufacturer":"Dell Inc.","baseboard-product-name":"0JXJPT","baseboard-version":"A02","baseboard-serial-number":".F94H182.CN7016354O001T.03.","oem-1":"Xen","oem-2":"MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d","oem-3":"Dell System","oem-4":"5[0000]","oem-5":"14[1]","oem-6":"17[6D0BB6C5CA6B4566]","oem-7":"17[6D0C10A7A9419261]","oem-8":"18[0]","oem-9":"19[1]","oem-10":"19[1]","hp-rombios":""},"power_on_mode":"","power_on_config":{},"local_cache_sr":"OpaqueRef:NULL","chipset_info":{"iommu":"true"},"PCIs":["OpaqueRef:d0f241a8-4daa-4501-b3cf-adb0892cc89b","OpaqueRef:a04776fb-5068-4ac7-8bd2-9d5771e1f852","OpaqueRef:92c5c165-8350-4b59-abf5-faed53c18098","OpaqueRef:8aec8abe-10c7-4927-804b-6cbb34e36b0b","OpaqueRef:6b676c66-16f3-484d-a091-ea0dee4bf8a8","OpaqueRef:644d2d29-5dbf-40e7-8416-73005db02084","OpaqueRef:34d72682-8f23-4236-a2c4-d4b2b5d32f26","OpaqueRef:29a4690d-bda1-4d5c-b4d1-17b62bb93df3","OpaqueRef:0779a461-8555-4f99-b967-b2edb5384afd"],"PGPUs":["OpaqueRef:d4dfdda0-964f-4421-9612-8019572b5e80"],"PUSBs":["OpaqueRef:4a3f6f0c-1ba1-4bc2-9d10-eb058a68268f"],"ssl_legacy":false,"guest_VCPUs_params":{},"display":"enabled","virtual_hardware_platform_versions":[0,1,2],"control_domain":"OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003","updates_requiring_reboot":[],"features":[],"iscsi_iqn":"iqn.2022-10.com.quape:88df7156","multipathing":false,"uefi_certificates":"","certificates":[],"editions":["xcp-ng"]},"SR":{"uuid":"73bf933d-050d-453c-5b43-1f464221678c","name_label":"SSD1","name_description":"","allowed_operations":["vdi_enable_cbt","vdi_list_changed_blocks","unplug","plug","pbd_create","vdi_disable_cbt","update","pbd_destroy","vdi_resize","vdi_clone","vdi_data_destroy","scan","vdi_snapshot","vdi_mirror","vdi_create","vdi_destroy","vdi_set_on_boot"],"current_operations":{},"VDIs":["OpaqueRef:f4a60bad-36da-46cb-a555-8e76ff2f3dd6"],"PBDs":["OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3"],"virtual_allocation":322122547200,"physical_utilisation":80355328,"physical_size":1007984996352,"type":"ext","content_type":"user","shared":false,"other_config":{},"tags":[],"sm_config":{"devserial":"scsi-357c35481e852d688"},"blobs":{},"local_cache_enabled":false,"introduced_by":"OpaqueRef:NULL","clustered":false,"is_tools_sr":false},"VDI":{"uuid":"ab00cbc9-7da2-443e-b145-b8a907abed3e","name_label":"quape.net 0","name_description":"Created by template provisioner","allowed_operations":["generate_config","update","forget","destroy","snapshot","resize","copy","clone"],"current_operations":{},"SR":"OpaqueRef:fed4ee9c-d397-4087-a4f8-902da574d30c","VBDs":["OpaqueRef:6ef02723-5995-4699-993e-2e0c8bbff868"],"crash_dumps":[],"virtual_size":322122547200,"physical_utilisation":636416,"type":"system","sharable":false,"read_only":false,"other_config":{"content_id":"d111595a-6f0d-53fa-07a9-97ac599cc436","xo:copy_of":"f8004138-6815-4c17-b31c-c00083542337"},"storage_lock":false,"location":"ab00cbc9-7da2-443e-b145-b8a907abed3e","managed":true,"missing":false,"parent":"OpaqueRef:NULL","xenstore_data":{},"sm_config":{},"is_a_snapshot":false,"snapshot_of":"OpaqueRef:NULL","snapshots":[],"snapshot_time":"19700101T00:00:00Z","tags":[],"allow_caching":false,"on_boot":"persist","metadata_of_pool":"","metadata_latest":false,"is_tools_iso":false,"cbt_enabled":false},"message":"VDI_IO_ERROR(Device I/O errors)","name":"XapiError","stack":"XapiError: VDI_IO_ERROR(Device I/O errors)\n    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)\n    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)\n    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:954:37)\n    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:988:14)\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:978:12)\n    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1144:14)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:447:9)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"}