VDI_IO_ERROR(Device I/O errors) when you run scheduled backup

shorian

Source:

# dmesg | grep -i cut -C10
[    0.000000] Xen: [mem 0x0000000079a66000-0x0000000079f6efff] ACPI NVS
[    0.000000] Xen: [mem 0x0000000079f6f000-0x000000008fffffff] reserved
[    0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[    0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[    0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x0000000270f0efff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: Supermicro X10DRi/X10DRi, BIOS 2.1 09/13/2016
[    0.000000] Hypervisor detected: Xen PV
[    0.047663] tsc: Fast TSC calibration using PIT
[    0.047664] tsc: Detected 2399.968 MHz processor
[    0.047665] tsc: Detected 2400.010 MHz TSC
[    0.049570] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.049572] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.049578] last_pfn = 0x270f0f max_arch_pfn = 0x400000000
[    0.049579] Disabled

Target:

dmesg | grep -i cut -C10
[    0.000000] Xen: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[    0.000000] Xen: [mem 0x00000000fec18000-0x00000000fec18fff] reserved
[    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
[    0.000000] Xen: [mem 0x00000000fec28000-0x00000000fec28fff] reserved
[    0.000000] Xen: [mem 0x00000000fec30000-0x00000000fec30fff] reserved
[    0.000000] Xen: [mem 0x00000000fec38000-0x00000000fec38fff] reserved
[    0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed44fff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x00000004a52a5fff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] efi: EFI v2.50 by American Megatrends
[    0.000000] efi:  SMBIOS=0x6f0bc000  SMBIOS 3.0=0x6f0bb000  ACPI 2.0=0x6ca22000  ACPI=0x6ca22000  ESRT=0x64ba8f18
[    0.000000] SMBIOS 3.1.1 present.
[    0.000000] DMI: Supermicro SYS-6029P-TR/X11DPi-N, BIOS 2.0b 02/28/2018
[    0.000000] Hypervisor detected: Xen PV
[    0.000480] tsc: Detected 2200.030 MHz processor
[    0.002610] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.002613] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.002624] last_pfn = 0x4a52a6 max_arch_pfn = 0x400000000
[    0.002625] Disabled

olivierlambert

Hmm nothing obvious here…

tuxen

This got my attention:

Jan 15 19:17:40 xcp-ng-xen12-lon2 xapi: [error||623653 INET :::80||import] Caught exception in import handler: VDI_IO_ERROR: [ Device I/O errors ]
Jan 15 19:17:40 xcp-ng-xen12-lon2 xapi: [error||623653 INET :::80||backtrace] VDI.import D:378e6880299b failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
Jan 15 19:17:40 xcp-ng-xen12-lon2 xapi: [error||623653 INET :::80||backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")

This Unix.EPIPE error on the remote target means that the pipe stream is being closed before VDI.Import receives all the data. The outcome is a VDI I/O error due to a broken, partial sent/received VDI.

Since a remote-over-the-internet link can be more prone to latency/intermittency issues, it might be needed to adjust the remote NFS soft timeout/retries or mounting the target with hard option.

I would also check if the remote target is running out-of-space during the backup process.

shorian

@tuxen Target has plenty of space, over 5Tb free or about 20x the cumulative VM sizes. No NFS involved, it’s a locally mounted ext4 raid 1 array on the target box.

If same backup takes place behind the firewall it runs successfully 95% of the time, across the WAN it fails 95% of the time. Both over a 1gbps link.

Sometimes the failures clean themselves up, sometimes end up with a VM/disk marked [importing.....<backup name><VM name>] that need to be manually removed.

Any help hugely appreciated.

olivierlambert

Interesting. It's like the data stream is interrupted somehow for a bit and that's enough to trigger the issue.

EddieCh08666741

i have similar issue like this too. I'm using ext4 and it was perfectly fine when i'm using 7.6. After upgrading to ext4 and 8.2 fresh install. The CR dont work anymore.

EddieCh08666741

I just tried installing Xen Orchestra from the sources on Debian 11. The same CR works well.

My previous Xen Orchestra from the sources ubuntu 18 having issue with VDI ERROR. Will do more testing.

olivierlambert

Note: XOA is only the version distributed by Vates. Everything else is "Xen Orchestra from the sources"

EddieCh08666741

@olivierlambert Thanks Corrected. I'm so happy for XCP-NG. I'm one of the early backers in Kickstarter. Hang the shirt up in our office.

rauly94

Good Morning guys,

it looks like I'm having the same issue on a host that I'm doing replications. On the normal Delta Backups I'm not getting this error. But I'm getting it on the schedule Replication for only 1 of the vm's. Here is what comes on the log for that VM.

"id": "1679486747187",
              "message": "transfer",
              "start": 1679486747187,
              "status": "failure",
              "end": 1679487904889,
              "result": {
                "code": "VDI_IO_ERROR",
                "params": [
                  "Device I/O errors"
                ],
                "url": "https://192.168.2.11/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Aa5f60c35-64c2-497e-ae15-77aa63d14274&session_id=OpaqueRef%3A1593dbd0-4347-4dee-ad02-58ed84f6dbf6&task_id=OpaqueRef%3Aced050e9-1f49-4bbe-8b74-ef787d536d62",
                "task": {
                  "uuid": "a7e3ba73-9d02-c211-48be-6a246828211c",
                  "name_label": "[XO] Importing content into VDI HEALmycroft 0",
                  "name_description": "",
                  "allowed_operations": [],
                  "current_operations": {},
                  "created": "20230322T12:05:52Z",
                  "finished": "20230322T12:25:03Z",
                  "status": "failure",
                  "resident_on": "OpaqueRef:259273e3-6fe1-4f9a-b688-ab0847cb81f8",
                  "progress": 1,
                  "type": "<none/>",
                  "result": "",
                  "error_info": [
                    "VDI_IO_ERROR",
                    "Device I/O errors"
                  ],
                  "other_config": {},
                  "subtask_of": "OpaqueRef:NULL",
                  "subtasks": [],
                  "backtrace": "(((process xapi)(filename ocaml/xapi/vhd_tool_wrapper.ml)(line 77))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/import_raw_vdi.ml)(line 170)))"

olivierlambert

Device I/O error isn't a good sign. Check dmesg and the disk health.

rauly94

@olivierlambert sorry to ask, but where should I do this at?

AtaxyaNetwork

@rauly94 Hi !

You can type "dmesg" directly in your host.
You can check your disk's health with the command "smartctl", directly on the host too

rauly94

@olivierlambert said in VDI_IO_ERROR(Device I/O errors) when you run scheduled backup:

dmesg

that's what i got. I'm I doing it correctly.

The issue is happening on 1 VM. i tried doing a copy of the same vm and same error. just FYI

rauly94

@rauly94 @AtaxyaNetwork

rauly94

@rauly94 Hello everyone.
Anyone can help me on this issue. Now it started happening on 2 vm's instead of 1 vm. It is happening on the backup replication.

Error: VDI_IO_ERROR(Device I/O errors)This is a XenServer/XCP-ng error
Start: Jul 5, 2023, 09:03:18 AM
End: Jul 5, 2023, 09:44:46 AM
Duration: 41 minutes
Error: VDI_IO_ERROR(Device I/O errors)This is a XenServer/XCP-ng error

Start: Jul 5, 2023, 09:03:09 AM
End: Jul 5, 2023, 09:45:12 AM
Duration: 42 minutes
Error: VDI_IO_ERROR(Device I/O errors)This is a XenServer/XCP-ng error
Type: delta