Continuous Replication fails with VDI_IO_ERROR(Device I/O errors) on fresh XOA install after XenServer 7.2 -> XCP-ng 8.0 upgrade

1am3r

Hi,
we upgraded our XenServer 7.2 pool to XCP-ng 8.0.

We previously hat Continuous Replication from a iSCSI storage to a NFS Storage running.

After the update, all replication jobs fail with VDI_IO_ERROR(Device I/O errors) and no apparent erros in SMlog or xensource.log (or I don't know for what to look for).

Is this a known problem? How can we further debug this problem?

Thanks in advance,
Michael

olivierlambert

Hi,

I'm not aware of that issue. You could try to add the following entry in your Xen Orchestra configuration and try again:

guessVhdSizeOnImport: true

If you have XOA, we can do that remotely for you.

1am3r

Changing this setting fixed the replication in the first tests. Are there any negative side effects with this?

olivierlambert

Keep us posted after few runs (to be sure it's not due to a first full that it worked). You shouldn't have any downside using it, it's a setting we didn't put by default until we are 100% sure of the outcome (especially on retrocompatibility for older hosts)

1am3r

It looks good for now. I'll monitor it for a while and report back. Thanks!

olivierlambert

You are welcome

nyash

Hi,

I had noticed a similar problem. I have upgraded from XCP-NG 7.4 to 8.0 via the ISO. Upgrade was painless. Only issue was with a single VM not starting that was resolved with "xe vm-cd-eject --multiple".

I have also upgraded to the latest Xen Orchestra from sources:
:xo-server 5.46.0
:xo-web 5.46.0

I have 2 xenservers with local storage defined like this:
https://i.imgur.com/XCCyzS2.png

I have defined a simple Continous Replication task like this to backup a very small VM from xenserver1 to xenserver2:
https://i.imgur.com/d8nuDAN.png

And started the task manually, however it eventually fails like this:
https://i.imgur.com/B5IS5lB.png

Attached log:
https://pastebin.com/raw/gS9SYXgg

Changing the setting in config.toml of xo-server to: 'guessVhdSizeOnImport: true' seems to have fixed the problem (the backup task completes succesfully).

Thanks.

rizaemet 0

Same problem here after upgrade xcp-ng 7.6 to 8.0

olivierlambert

And same solution worked?

rizaemet 0

@olivierlambert I've just tried and yes, 'guessVhdSizeOnImport: true' solved the problem.

olivierlambert

Okay I'm pinging @julien-f so we can include this as default (because it seems to not break for older XS)

1am3r

We run this config now for a week and didn't have any problems. So it looks good from our site.

txsastre

same problem, same solution. works fine

rizaemet 0

'guessVhdSizeOnImport: true' looks like solved the problem 9 day ago. But I'm getting same error sometimes and randomly since that day.
Daily CR job have 31 VMs. Some day successful 31/31. But some days one or two VM replication fail with tis message: VDI_IO_ERROR(Device I/O errors).
I think my problem still related with this topic.

nyash

@rizaemet-0
Ever since I have changed to guessVhdSizeOnImport: true I have not had any problems. However I only backup 5 VMs which is not much.

In the past I noticed that if I didn't change the default "Concurrency" setting to 1 in the Backup NGs advanced settings (I assume that it tries to parallelize backups by default) then my tasks would fail randomly once in a while with VDI_IO_ERROR(Device I/O errors). I figured that perhaps my servers are not fast enough; they have about 40MB/s write speed to the disks only. Setting to 1 solved it, but I can imagine this may not be desirable if there is lots of VMs to backup since it would slow down the process considerably.

julien-f

@olivierlambert Yep, I think it's safe enough to turn on by default in the next release

olivierlambert

Great, so let's do that!

onur

@olivierlambert said in Continuous Replication fails with VDI_IO_ERROR(Device I/O errors) on fresh XOA install after XenServer 7.2 -> XCP-ng 8.0 upgrade:

Xen Orchestra configuration

should I add "guessVhdSizeOnImport: true" to /opt/xen-orchestra/packages/xo-server/config.toml

olivierlambert

To your config file, yes. Where this config file is depends on where do you installed Xen Orchestra (I suppose from the sources, so I can't guess where do you installed, could be anyware )

edit: I think it's fixed now and integrated by default in the config

onur

@olivierlambert said in Continuous Replication fails with VDI_IO_ERROR(Device I/O errors) on fresh XOA install after XenServer 7.2 -> XCP-ng 8.0 upgrade:

To your config file, yes. Where this config file is depends on where do you installed Xen Orchestra (I suppose from the sources, so I can't guess where do you installed, could be anyware )

edit: I think it's fixed now and integrated by default in the config

Great, this fix already is in the new version.
I was having this problem (Error: VDI_IO_ERROR(Device I/O errors)) and was not be able to do CR of my VMs but now CR works.
Thank you

PS: BTW that was my config file