Hi,

We use the 'Remote syslog' option with XCP-ng 8.2.1. This has been working well - but recently we patched the pool, rebooted the pool (everything came back ok) - but Remote syslog - doesn't log any more.

If I go into XO and look at the option for the Hosts - on both it's set to the correct IP address.

If I try changing that IP to a different host on the network XO quite correctly said something like "can't use that expect port to be open".

If I change it back to the original host, I've just noticed - it starts working again.

This is a bit annoying (as we've lost days of logs) - anyone else seen similar with that option?

[Before figuring that changing it 'fixes' it - I'd already tested the syslog server was up, and I could send test log entries to it using 'logger' from the XCP-ng's dom0].

Thanks!

@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:

Yes, that sounds the next step to do Something is wrong during the mount phase. Also check you have enough RAM in your XOA.

Ok, filesystem checks out - as far as I can see having:

• Done full restore of VM from the same Delta image.
• Booted VM, logged in and done 'tar cvf /dev/null *' of the filesystem I'm trying to restore from - which ran to completion without error.

I upped the RAM for XOA from default 2GiB to 16GiB and re-tried - and same result.

I can try yet more RAM if needs be but would hope 16GiB is more than enough?

The console error on XOA is more detailed than my install- so don't know if this helps:

blk_update_request: I/O error, dev loop0, sector 8912912 op 0x00:(READ) flags 0x80700 phys_seg 1 prio class 0

Looks to be the same issue, just reported differently.

You also (on both systems) get kernel chatter about "tasked blocked" (understandable, as it is).

@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:

That's why you should try on XOA, not the sources. This way, we can see if it's a XO bug or an issue on your local source install

Ok, latest XOA (updated) has the same issue - process dies with loop0 I/O errors logged to console same as my 'from sources' version.

I'm going to do a full restore of the VM spin it up, and do a full filesystem check on it (well as good as ext3/4 can) - just to make 100% sure there's no corruption - as that seems sensible start / relatively easy to do.

@olivierlambert

Sorry - missed the 'XOA' only saw 'latest'... More coffee needed... Off to try.

@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:

First, I would try with an XOA in latest to compare the outcome

Hi,

Ok - moved from 2af84 to apparently 'master' at ce2b918a2 (let me know if that's wrong, I spend my life in SVN not Git).

This does exactly the same thing - same issue, same syslog error.

The backup being used for the file restore, is a 100G 'MBR' (i.e. legacy) disk split into 4 partitions, the last partition '/var' being ~78Gbyte in size (which is the one I select to restore from).

The source is a Delta backup (with a retention of 4).

ps. Sorry my log output above doesn't look formatted well, I don't know if this is a Safari thing or not - I did put 'code' markup before/after but here it looks like it's collapsed down to one line

@olivierlambert said in Can't restore VM's from 'read-only' remote?:

A remote (now Backup repository) is meant to write backups into it. If you aren't able to write, then, sure, you can restore but you won't be able to backup anything.

Yes, understand that - I'll have to re-test here, but I'm sure when it was 'read-only' - I couldn't restore anything from it, as nothing showed up in XO's "Backup / Restore' list of backups...

The remote was mounted on two systems - hence why made R/O for one of them (this was just for testing restores).

Hi,

I'm running XO from sources (2af84) under a Debian stretch VM.

I've got all the dependancies installed to do 'file restore' from backups - and this works, on small backups (e.g. ~8Gbyte).

However - a much larger VM I have backed up (~100Gbyte) fails.

Looking on the system - I can see it's created the loop back (loop0) for it, and mounted it on the XO VM, but then you start getting:

[882294.370559] print_req_error: I/O error, dev loop0, sector 8912912

Type errors. This coincides with using XO to 'browse' through the directory structure of the backup (i.e. traverse the mounted backup filesystem).

There's not a lot else logged in syslog, e.g. from the point you select a backup to file restore from, I see:

[878193.892486] fuse init (API version 7.27)
[878195.968810] loop: module loaded
[878196.269943] EXT4-fs (loop0): write access unavailable, skipping orphan cleanup
[878196.269996] EXT4-fs (loop0): mounted filesystem without journal. Opts: norecovery,norecovery
[882233.697709] EXT4-fs (loop0): mounting ext3 file system using the ext4 subsystem
[882235.713249] EXT4-fs (loop0): write access unavailable, skipping orphan cleanup
[882235.713284] EXT4-fs (loop0): mounted filesystem without journal. Opts: norecovery,norecovery
[882294.370559] print_req_error: I/O error, dev loop0, sector 8912912

The I/O errors just then re-occur. The XO VM is left locked up at this point (kind of understandably).

I'm trying to work out what else I can do to look at this - e.g. is there a way of getting XO to log what it did to create the local loop0 mount? - So I can try and replicate from the CLI?

I've only tried this on small a small VM (~8Gbyte which works) and large (~100Gbyte which fails) VM so far - so I don't know if there's a size at which it "stops working"

If anyone can point me in the right direction of trying to get some more info of what XO's done to get to this point - that'd be great,

Thanks!

Hi,

I've got a volume on an NFS server that's mounted as 'Read Only' - which is exported via NFS to XO as a 'remote'.

XO can 'see' this - but complains it's read-only when you attach it.

Although it does attach - I can't then 'see' any backups on that remote. The only thing that seems to work is to re-mount it on the NFS server as 'read/write' - and re-attach it to XO as read-write.

At this point, the list of VM's within 'xo-vm-backup' appear - and are available to restore.

This kind of leads to the questions: Do 'remotes' always have to be read/write mounted / available to XO to allow even restores to be done?

Thanks!

@florent

Yes, thanks!

We have very limited control over rsync at our end (as it's running within a Synoloy NAS) - I think we probably have to just live with large transfers for now - whilst looking at spit files / VHD's as the way to go in the future.

Thanks again!

@florent

Hi - thanks for the info. That obviously covers the multi-file / block "new" system.

Do you know how the merge is handled on "single file" Delta backups?

e.g. Every day now with the existing delta backups we can see a new delta diff being created (20GByte) - and we can see the base image is "touched" (as the oldest diff gets rolled into it) - but what we can't see is if the old merge process does a "copy base + merge diffs" then a rename (which would suggest copy-on-write filesystems will see it all be touched)

Or if it merges the diff into the base "in place" (i.e. by seeking - and over-writing areas within the existing base 100G VHD).

I'm suspecting for safety it does a copy + merge to a new file, then delete old & rename or something...

@olivierlambert

Hi,

Final question on this - rsync 'block' mode failed miserably to make a difference.

When XO "merges" diffs into the base image for delta backups - does it do this by just changing updated blocks -within the existing file- or does it do something like "copy whole base image to a new file, merging the diffs as it goes - then rename at the end"?

We looked at multiple-blocks - but "not ext4" kind of delays us actually trying it for now...

Thanks!

@olivierlambert

Thanks for the details / options. The boxes we're using support rsync 'block difference' mode - so we're giving that a try (on paper it should help if the boxes have enough 'grunt' to do this).

Failing that we'll check out the 'store backup as multiple data blocks' option.

Looking forward to the backup tiering - backups has been (and continues to be) one of many things XO does really, really well.

Cheers!

Hi,

We have a system where we've setup delta backups to an NFS remote that's on the local LAN.

These work really well, e.g. for the first backup we can see 100Gbyte being backed up, and subsequent backups for our VM run to about 10-15Gbyte.

We've been trying to rsync that local 'Remote' to another box over a WAN - but run into issues (that it takes for ever, a lot of data seems to have changed etc.)

Looking at the raw files on the XO "Remote" - I can see the UUID directory for the VM - and, buried away under the VDI's subdirectory etc. - I can see the VHD files.

This seems to comprise of the 100GByte 'base' VHD - and then a number of smaller differentials (as you'd expect).

But after a backup run - the remote NFS seems to show that the base .VHD has been changed (thus rsync will dutifully copy over the 100GByte file again).

Do the delta backups modify this base VHD? - Maybe to apply the oldest diff to bring it forward in time before deleting that older diff?

Just trying to get my head around how these work - and what the impact will be for rsync.

Is there any better mode we could use when the remote is to be rsync'd off site? - Unfortunately we can't write directly to the off-site NFS - we only really have rsync access.

Thanks!

Hi,

Is there any way to have Disaster Recovery backups remove (i.e. toggle to "off") the auto-start flag on VM's as they're replicated?

A source VM here has 'auto-start: yes' set - if it's replicated twice - that's a lot of VM's sitting around with "auto-start" enabled waiting for something to go wrong (i.e. both start at the same time etc.)

@danp I'd convinced myself that the 'Export Settings' option didn't work when XO was compiled from sources

I've just tried it now, and it does work - what was happening (literally all this time!) is Safari pops up the tiniest "Popup window blocked" icon in the address bar you've ever seen. Click that - and the download starts

Ok, full rebuild from nothing seems to have fixed it... Bonus points if someone can suggest how I can copy the settings from the previous instance to this one?

/var/lib/xo-server/data/leveldb doesn't seem to like being copied (even with XO shutdown).

It'd be great to be able to pull the backup jobs over.

@danp I think there's something out of sync somewhere (though I was careful this was a new/fresh clone of git master) - I can't create VM's either (e.g. ""message": "should not contains property ["bypassMacAddressesCheck"]"".

I'll do a completely clean build not upgrade and see if it's still happening...

@danp Hi - unfortunately not. I don't have access to XOA. I just tried to download the trial / starter edition, which can't restore - and I can't upgrade to the free trial - because I've already had a free trial

@danp
Hi, don't know if you were suggesting I do the rebuild or @jmccoy555 - but I've tried rebuilding re-checking out - and yarn clean / yarn built et'al - and still have the same error.

@danp

Ok - I created a new backup job, for a small VM (again, delta backup) - going to a new test remote.

The backup ran - as it's the first time, it did a full copy as you'd expect, and it completed ok.

I cannot restore it either - shows the same 'invalid parameters' error - and the log details shows the same '"should not contains property ["settings"]"'

Backend server is XCP-ng 8.2