XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    VDI_IO_ERROR Continuous Replication on clean install.

    Scheduled Pinned Locked Moved Solved Xen Orchestra
    66 Posts 7 Posters 7.0k Views 8 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Tristis OrisT Offline
      Tristis Oris Top contributor @olivierlambert
      last edited by

      if i have no smlog - xen\dom0 not related with backup task. right?
      smlog that usualy i got during this 5min have no any errors anyway, only some locking operations.
      And it always takes 5min, some hardcoded timings?

      don't forget that problem also happens with FC connection, so it may concern any block based storage types.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        I don't understand your sentence, can you take time to re-read it or rephrase it, because I doesn't make sense to me, sorry 😞

        What do you mean by "if i have no smlog - xen\dom0 not related with backup task. right?"?

        Tristis OrisT 1 Reply Last reply Reply Quote 0
        • Tristis OrisT Offline
          Tristis Oris Top contributor @olivierlambert
          last edited by

          i mean it could be XO issue, since it not communicate with xen. Otherwise it should write some logs.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            I don't see the logical connection with XO, since it works on some SR and not on others. XO has no idea (or doesn't care) about the underlying storage.

            1 Reply Last reply Reply Quote 0
            • Tristis OrisT Offline
              Tristis Oris Top contributor
              last edited by

              well, i'm just made some tests and got some result. Have no idea how it should work)

              1 Reply Last reply Reply Quote 0
              • Tristis OrisT Offline
                Tristis Oris Top contributor
                last edited by

                i don't understand what happens.
                Reinstalled xen to 8.2.0, CR was succeed for few times, but now i got this error again.

                Tried few tests - 2-3 fails in row then it succeed again.
                Only way to never use this pool for CR.

                1 Reply Last reply Reply Quote 0
                • EddieCh08666741E Offline
                  EddieCh08666741
                  last edited by

                  I also have some fresh installation 8.2.1 with similar error at 5 mins 2seconds 5 min 1 sec 😞

                  1 Reply Last reply Reply Quote 0
                  • Y Offline
                    yomono
                    last edited by

                    On my side, yesterday I did the only test I haven't done so far: Installing XenOrchestra in a NON xcp-ng server.
                    Basically, since always, I had a separated XCP-NG server with just a single VM inside: The XO VM (Just in case, that VM was Ubuntu, Centos, Debian over time, so the base OS has nothing to do with this).
                    My solution for this was simple: Bare metal Linux. So the problem wasn't the XCP version on the source server, nor the destination server. It was the host server of the XO VM itself
                    Why? I have no idea, but it's definitely working now since I started a CR task yesterday, of a 1TB VM, with a destination server over internet, and is still exporting after 14 hours without any issues:
                    7f769e0c-6880-4e21-8977-7e1688db24f5-imagen.png

                    Y 1 Reply Last reply Reply Quote 0
                    • Y Offline
                      yomono @yomono
                      last edited by

                      @yomono And to clarify, for this XO host, I tested 8.2.0, 8.2.1, and 8.3.0 fresh installs and all failed at exactly 5 minutes

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        This means your Node version was still using the default timeout.

                        Y 1 Reply Last reply Reply Quote 0
                        • Y Offline
                          yomono @olivierlambert
                          last edited by

                          @olivierlambert when you say "node" you mean node.js? How that timeout can be changed? Thanks

                          olivierlambertO 1 Reply Last reply Reply Quote 0
                          • EddieCh08666741E Offline
                            EddieCh08666741
                            last edited by

                            changing the node will fix this vdi error ?

                            e86eee8c-b1da-4d69-9776-11fd126348cd-image.png

                            EddieCh08666741E 1 Reply Last reply Reply Quote 0
                            • EddieCh08666741E Offline
                              EddieCh08666741 @EddieCh08666741
                              last edited by

                              @EddieCh08666741 The one which works is the fresh install without any updates.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO @yomono
                                last edited by

                                @yomono NodeJS yes. In Node 18, they made a breaking change to put by default a 5 minutes timeout.

                                We fixed that by adding a specific config to get a longer timeout, see https://github.com/vatesfr/xen-orchestra/commit/f6fd1db1ef12633cc5bb8ec8ab5bc84682dd3fe7

                                Without this piece of config, you'll end any HTTP stream after exactly 5 minutes.

                                0 julien-f committed to vatesfr/xen-orchestra
                                feat(xo-server): increase HTTP server request timeout to 1 day
                                
                                Fixes #6590
                                Y 1 Reply Last reply Reply Quote 0
                                • Y Offline
                                  yomono @olivierlambert
                                  last edited by

                                  @olivierlambert Well, I checked the config.toml and, indeed, I can confirm the fix is there.
                                  What's really odd here is that I'm using the exact same installation script for XO that I've used in the past.
                                  So, why now this value is correct but it wasn't when executing the exact same script in a VM? (Not an actual question for you, but just wondering out loud)
                                  So @EddieCh08666741 and @Tristis-Oris you were having an exact 5 minutes timeout too so you might want to check this config as well
                                  Thanks for the support, Olivier

                                  1 Reply Last reply Reply Quote 0
                                  • Tristis OrisT Offline
                                    Tristis Oris Top contributor
                                    last edited by

                                    my config is fine, but problem still here for this one pool.

                                    EddieCh08666741E 1 Reply Last reply Reply Quote 0
                                    • EddieCh08666741E Offline
                                      EddieCh08666741 @Tristis Oris
                                      last edited by

                                      @Tristis-Oris i double check and the codes mentioned is updated. The issue is the same for doing the restore from delta backups.

                                      1 Reply Last reply Reply Quote 0
                                      • EddieCh08666741E Offline
                                        EddieCh08666741
                                        last edited by Danp

                                        {"code":"VDI_IO_ERROR","params":["Device I/O errors"],"url":"https://192.168.0.100/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Af4a60bad-36da-46cb-a555-8e76ff2f3dd6&session_id=OpaqueRef%3Aba2fcb1d-bc46-445b-9540-b0e9e51236ee&task_id=OpaqueRef%3A8ca30f05-cc13-4c04-907f-6b4468a6ace2","task":{"uuid":"fa99f516-2352-3b92-7419-4950559458d8","name_label":"[XO] Importing content into VDI 0","name_description":"","allowed_operations":[],"current_operations":{},"created":"20230204T06:49:36Z","finished":"20230204T06:54:36Z","status":"failure","resident_on":"OpaqueRef:dba25009-ad21-4aad-a7d6-0b03af29a906","progress":1,"type":"<none/>","result":"","error_info":["VDI_IO_ERROR","Device I/O errors"],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename ocaml/xapi/vhd_tool_wrapper.ml)(line 77))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/import_raw_vdi.ml)(line 170)))"},"pool_master":{"uuid":"e16ba86e-9915-446d-824e-f2246a2711ad","name_label":"blade7.quape.com","name_description":"Default install","memory_overhead":2050928640,"allowed_operations":["vm_migrate","provision","vm_resume","evacuate","vm_start"],"current_operations":{},"API_version_major":2,"API_version_minor":16,"API_version_vendor":"XenSource","API_version_vendor_implementation":{},"enabled":true,"software_version":{"product_version":"8.2.1","product_version_text":"8.2","product_version_text_short":"8.2","platform_name":"XCP","platform_version":"3.2.1","product_brand":"XCP-ng","build_number":"release/yangtze/master/58","hostname":"localhost","date":"2022-10-12","dbv":"0.0.1","xapi":"1.20","xen":"4.13.4-9.19.1","linux":"4.19.0+1","xencenter_min":"2.16","xencenter_max":"2.16","network_backend":"openvswitch","db_schema":"5.602"},"other_config":{"agent_start_time":"1670522053.","boot_time":"1670465798.","rpm_patch_installation_time":"1670522042.976","iscsi_iqn":"iqn.2022-10.com.quape:88df7156"},"capabilities":["xen-3.0-x86_64","hvm-3.0-x86_32","hvm-3.0-x86_32p","hvm-3.0-x86_64",""],"cpu_configuration":{},"sched_policy":"credit","supported_bootloaders":["pygrub","eliloader"],"resident_VMs":["OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003"],"logging":{},"PIFs":["OpaqueRef:bb16676d-1c45-4363-9550-ffc6f2323239","OpaqueRef:ae69076b-b4c6-4711-9ab2-95b135daf2d7","OpaqueRef:171c0df8-5b44-459e-8756-01b70115586d","OpaqueRef:12f18eda-5b70-4735-b95e-316fc901d635"],"suspend_image_sr":"OpaqueRef:NULL","crash_dump_sr":"OpaqueRef:NULL","crashdumps":[],"patches":[],"updates":[],"PBDs":["OpaqueRef:08cf61cc-234c-4f62-b006-05fc9f14d27b","OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3","OpaqueRef:064440da-c058-491d-85b6-e9faa1166878"],"host_CPUs":["OpaqueRef:060a9fa7-302b-4f3e-9125-c51c649cc080","OpaqueRef:13839264-af8f-47dc-b235-533aa28810c7","OpaqueRef:bea9bc59-c762-447c-8145-961af6bf404b","OpaqueRef:ee0ab5bc-3d46-4edc-84b2-cf6f33ce060d","OpaqueRef:8b7690a2-7ac0-4b6e-955c-ede13892942a","OpaqueRef:ed322770-a47f-499e-8c6f-68dbca0627cf","OpaqueRef:4e02a573-5038-4667-b89f-aeef10f64b57","OpaqueRef:f86550ec-15f0-4d85-96e9-e2da1f9fdde3","OpaqueRef:282ccd71-14f2-49ed-b5e3-ad0ad2ea4182","OpaqueRef:02cc7f55-a453-40e9-8285-45a8eb97c4eb","OpaqueRef:a4af195f-2530-42a9-a684-8882097558ab","OpaqueRef:b1ec191e-a995-4c1b-8bac-dde94e355bb9","OpaqueRef:e4a5b5b9-4263-4061-b1a6-2241ff9df33e","OpaqueRef:15a31b91-5daa-4fdd-a128-64222be18599","OpaqueRef:a63dbbc2-43d2-4a17-81de-deef98c4ff70","OpaqueRef:865c00c1-ab16-4c36-83e0-26058bfc9d6f","OpaqueRef:f6c5abf4-b772-427c-b533-9e1dac402dd7","OpaqueRef:e4870fbc-9263-41a4-84fc-bcf747ce6326","OpaqueRef:ac916ff7-e936-4ae4-9841-9e07743881f9","OpaqueRef:029a2790-2795-444f-b0cf-3db6bd0195c2","OpaqueRef:33f83ad1-d849-48e5-995c-7f9b4a696ccd","OpaqueRef:deb97b8b-fff4-4192-ad8f-dfb26f1f8c88","OpaqueRef:dd32e3ce-e147-479b-b0ac-487d25db1384","OpaqueRef:1b9d353c-77bd-4caf-9ae0-64eb2887c06a","OpaqueRef:97fe2a88-fa69-48f6-9f52-a6b3eb39ba10","OpaqueRef:fa1ec86c-2f3a-49d5-b1b0-52787894153e","OpaqueRef:5d5e4dbe-a11f-4d10-aac6-34f806b52317","OpaqueRef:f65eccc8-5fd2-4439-8817-5736ef0c7c08","OpaqueRef:20755a4b-2ce8-4f29-bce8-5e5f058ddc71","OpaqueRef:da6f5842-1388-435c-a741-df424fe3216d","OpaqueRef:504cce7a-66a7-49bd-8c2a-76d093aa5328","OpaqueRef:3801bf53-3bc3-4de9-bdcc-51142bbfcd41"],"cpu_info":{"cpu_count":"32","socket_count":"2","vendor":"GenuineIntel","speed":"3200.329","modelname":"Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz","family":"6","model":"63","stepping":"2","flags":"fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms xsaveopt","features_pv":"1fc9cbf5-f6f83203-2991cbf5-00000023-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm":"1fcbfbff-f7fa3223-2d93fbff-00000423-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_hvm_host":"1fcbfbff-f7fa3223-2c100800-00000021-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000","features_pv_host":"1fc9cbf5-f6f83203-28100800-00000021-00000001-00000329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"},"hostname":"blade7.quape.com","address":"","metrics":"OpaqueRef:1434bd31-37b0-416e-93f1-bdcbddafd324","license_params":{"restrict_vswitch_controller":"false","restrict_lab":"false","restrict_stage":"false","restrict_storagelink":"false","restrict_storagelink_site_recovery":"false","restrict_web_selfservice":"false","restrict_web_selfservice_manager":"false","restrict_hotfix_apply":"false","restrict_export_resource_data":"false","restrict_read_caching":"false","restrict_cifs":"false","restrict_health_check":"false","restrict_xcm":"false","restrict_vm_memory_introspection":"false","restrict_batch_hotfix_apply":"false","restrict_management_on_vlan":"false","restrict_ws_proxy":"false","restrict_vlan":"false","restrict_qos":"false","restrict_pool_attached_storage":"false","restrict_netapp":"false","restrict_equalogic":"false","restrict_pooling":"false","enable_xha":"true","restrict_marathon":"false","restrict_email_alerting":"false","restrict_historical_performance":"false","restrict_wlb":"false","restrict_rbac":"false","restrict_dmc":"false","restrict_checkpoint":"false","restrict_cpu_masking":"false","restrict_connection":"false","platform_filter":"false","regular_nag_dialog":"false","restrict_vmpr":"false","restrict_vmss":"false","restrict_intellicache":"false","restrict_gpu":"false","restrict_dr":"false","restrict_vif_locking":"false","restrict_storage_xen_motion":"false","restrict_vgpu":"false","restrict_integrated_gpu_passthrough":"false","restrict_vss":"false","restrict_guest_agent_auto_update":"false","restrict_pci_device_for_auto_update":"false","restrict_xen_motion":"false","restrict_guest_ip_setting":"false","restrict_ad":"false","restrict_nested_virt":"false","restrict_live_patching":"false","restrict_set_vcpus_number_live":"false","restrict_pvs_proxy":"false","restrict_igmp_snooping":"false","restrict_rpu":"false","restrict_pool_size":"false","restrict_cbt":"false","restrict_usb_passthrough":"false","restrict_network_sriov":"false","restrict_corosync":"true","restrict_zstd_export":"false","restrict_pool_secret_rotation":"false"},"ha_statefiles":[],"ha_network_peers":[],"blobs":{},"tags":[],"external_auth_type":"","external_auth_service_name":"","external_auth_configuration":{},"edition":"xcp-ng","license_server":{"address":"localhost","port":"27000"},"bios_strings":{"bios-vendor":"Dell Inc.","bios-version":"2.15.0","system-manufacturer":"Dell Inc.","system-product-name":"PowerEdge FC630","system-version":"","system-serial-number":"F94H182","baseboard-manufacturer":"Dell Inc.","baseboard-product-name":"0JXJPT","baseboard-version":"A02","baseboard-serial-number":".F94H182.CN7016354O001T.03.","oem-1":"Xen","oem-2":"MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d","oem-3":"Dell System","oem-4":"5[0000]","oem-5":"14[1]","oem-6":"17[6D0BB6C5CA6B4566]","oem-7":"17[6D0C10A7A9419261]","oem-8":"18[0]","oem-9":"19[1]","oem-10":"19[1]","hp-rombios":""},"power_on_mode":"","power_on_config":{},"local_cache_sr":"OpaqueRef:NULL","chipset_info":{"iommu":"true"},"PCIs":["OpaqueRef:d0f241a8-4daa-4501-b3cf-adb0892cc89b","OpaqueRef:a04776fb-5068-4ac7-8bd2-9d5771e1f852","OpaqueRef:92c5c165-8350-4b59-abf5-faed53c18098","OpaqueRef:8aec8abe-10c7-4927-804b-6cbb34e36b0b","OpaqueRef:6b676c66-16f3-484d-a091-ea0dee4bf8a8","OpaqueRef:644d2d29-5dbf-40e7-8416-73005db02084","OpaqueRef:34d72682-8f23-4236-a2c4-d4b2b5d32f26","OpaqueRef:29a4690d-bda1-4d5c-b4d1-17b62bb93df3","OpaqueRef:0779a461-8555-4f99-b967-b2edb5384afd"],"PGPUs":["OpaqueRef:d4dfdda0-964f-4421-9612-8019572b5e80"],"PUSBs":["OpaqueRef:4a3f6f0c-1ba1-4bc2-9d10-eb058a68268f"],"ssl_legacy":false,"guest_VCPUs_params":{},"display":"enabled","virtual_hardware_platform_versions":[0,1,2],"control_domain":"OpaqueRef:64055ef1-c538-42e9-a880-e6803fe02003","updates_requiring_reboot":[],"features":[],"iscsi_iqn":"iqn.2022-10.com.quape:88df7156","multipathing":false,"uefi_certificates":"","certificates":[],"editions":["xcp-ng"]},"SR":{"uuid":"73bf933d-050d-453c-5b43-1f464221678c","name_label":"SSD1","name_description":"","allowed_operations":["vdi_enable_cbt","vdi_list_changed_blocks","unplug","plug","pbd_create","vdi_disable_cbt","update","pbd_destroy","vdi_resize","vdi_clone","vdi_data_destroy","scan","vdi_snapshot","vdi_mirror","vdi_create","vdi_destroy","vdi_set_on_boot"],"current_operations":{},"VDIs":["OpaqueRef:f4a60bad-36da-46cb-a555-8e76ff2f3dd6"],"PBDs":["OpaqueRef:4dae1d57-a359-40d2-a80d-08acb311c2d3"],"virtual_allocation":322122547200,"physical_utilisation":80355328,"physical_size":1007984996352,"type":"ext","content_type":"user","shared":false,"other_config":{},"tags":[],"sm_config":{"devserial":"scsi-357c35481e852d688"},"blobs":{},"local_cache_enabled":false,"introduced_by":"OpaqueRef:NULL","clustered":false,"is_tools_sr":false},"VDI":{"uuid":"ab00cbc9-7da2-443e-b145-b8a907abed3e","name_label":"quape.net 0","name_description":"Created by template provisioner","allowed_operations":["generate_config","update","forget","destroy","snapshot","resize","copy","clone"],"current_operations":{},"SR":"OpaqueRef:fed4ee9c-d397-4087-a4f8-902da574d30c","VBDs":["OpaqueRef:6ef02723-5995-4699-993e-2e0c8bbff868"],"crash_dumps":[],"virtual_size":322122547200,"physical_utilisation":636416,"type":"system","sharable":false,"read_only":false,"other_config":{"content_id":"d111595a-6f0d-53fa-07a9-97ac599cc436","xo:copy_of":"f8004138-6815-4c17-b31c-c00083542337"},"storage_lock":false,"location":"ab00cbc9-7da2-443e-b145-b8a907abed3e","managed":true,"missing":false,"parent":"OpaqueRef:NULL","xenstore_data":{},"sm_config":{},"is_a_snapshot":false,"snapshot_of":"OpaqueRef:NULL","snapshots":[],"snapshot_time":"19700101T00:00:00Z","tags":[],"allow_caching":false,"on_boot":"persist","metadata_of_pool":"","metadata_latest":false,"is_tools_iso":false,"cbt_enabled":false},"message":"VDI_IO_ERROR(Device I/O errors)","name":"XapiError","stack":"XapiError: VDI_IO_ERROR(Device I/O errors)\n    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)\n    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)\n    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:954:37)\n    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:988:14)\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:978:12)\n    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1144:14)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:447:9)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"}
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • EddieCh08666741E Offline
                                          EddieCh08666741
                                          last edited by

                                          76bc867e-a95d-4ca5-8553-c98b771af16c-image.png

                                          I did quite a number of test and these are all fresh install and the codes for node js fix is there. I'm using the restore function to local SR ( ext4 ) Is there some other codes that needs to be updated ? Anyone have similar issue ?

                                          EddieCh08666741E 1 Reply Last reply Reply Quote 0
                                          • EddieCh08666741E Offline
                                            EddieCh08666741 @EddieCh08666741
                                            last edited by

                                            @EddieCh08666741 Hope my above logs can help team to see if there's a bug in the script.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post