Yes, thanks!
We have very limited control over rsync at our end (as it's running within a Synoloy NAS) - I think we probably have to just live with large transfers for now - whilst looking at spit files / VHD's as the way to go in the future.
Thanks again!
Hi,
We use the 'Remote syslog' option with XCP-ng 8.2.1. This has been working well - but recently we patched the pool, rebooted the pool (everything came back ok) - but Remote syslog - doesn't log any more.
If I go into XO and look at the option for the Hosts - on both it's set to the correct IP address.
If I try changing that IP to a different host on the network XO quite correctly said something like "can't use that expect port to be open".
If I change it back to the original host, I've just noticed - it starts working again.
This is a bit annoying (as we've lost days of logs) - anyone else seen similar with that option?
[Before figuring that changing it 'fixes' it - I'd already tested the syslog server was up, and I could send test log entries to it using 'logger' from the XCP-ng's dom0].
Thanks!
@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:
Yes, that sounds the next step to do
Something is wrong during the mount phase. Also check you have enough RAM in your XOA.
Ok, filesystem checks out - as far as I can see having:
I upped the RAM for XOA from default 2GiB to 16GiB and re-tried - and same result.
I can try yet more RAM if needs be but would hope 16GiB is more than enough?
The console error on XOA is more detailed than my install- so don't know if this helps:
blk_update_request: I/O error, dev loop0, sector 8912912 op 0x00:(READ) flags 0x80700 phys_seg 1 prio class 0
Looks to be the same issue, just reported differently.
You also (on both systems) get kernel chatter about "tasked blocked" (understandable, as it is).
@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:
That's why you should try on XOA, not the sources. This way, we can see if it's a XO bug or an issue on your local source install
Ok, latest XOA (updated) has the same issue - process dies with loop0 I/O errors logged to console same as my 'from sources' version.
I'm going to do a full restore of the VM spin it up, and do a full filesystem check on it (well as good as ext3/4 can) - just to make 100% sure there's no corruption - as that seems sensible start / relatively easy to do.
Sorry - missed the 'XOA' only saw 'latest'... More coffee needed... Off to try.
@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:
First, I would try with an XOA in
latestto compare the outcome
Hi,
Ok - moved from 2af84 to apparently 'master' at ce2b918a2 (let me know if that's wrong, I spend my life in SVN not Git).
This does exactly the same thing - same issue, same syslog error.
The backup being used for the file restore, is a 100G 'MBR' (i.e. legacy) disk split into 4 partitions, the last partition '/var' being ~78Gbyte in size (which is the one I select to restore from).
The source is a Delta backup (with a retention of 4).
ps. Sorry my log output above doesn't look formatted well, I don't know if this is a Safari thing or not - I did put 'code' markup before/after but here it looks like it's collapsed down to one line 
@olivierlambert said in Can't restore VM's from 'read-only' remote?:
A remote (now Backup repository) is meant to write backups into it. If you aren't able to write, then, sure, you can restore but you won't be able to backup anything.
Yes, understand that - I'll have to re-test here, but I'm sure when it was 'read-only' - I couldn't restore anything from it, as nothing showed up in XO's "Backup / Restore' list of backups...
The remote was mounted on two systems - hence why made R/O for one of them (this was just for testing restores).
Hi,
I'm running XO from sources (2af84) under a Debian stretch VM.
I've got all the dependancies installed to do 'file restore' from backups - and this works, on small backups (e.g. ~8Gbyte).
However - a much larger VM I have backed up (~100Gbyte) fails.
Looking on the system - I can see it's created the loop back (loop0) for it, and mounted it on the XO VM, but then you start getting:
[882294.370559] print_req_error: I/O error, dev loop0, sector 8912912
Type errors. This coincides with using XO to 'browse' through the directory structure of the backup (i.e. traverse the mounted backup filesystem).
There's not a lot else logged in syslog, e.g. from the point you select a backup to file restore from, I see:
[878193.892486] fuse init (API version 7.27)
[878195.968810] loop: module loaded
[878196.269943] EXT4-fs (loop0): write access unavailable, skipping orphan cleanup
[878196.269996] EXT4-fs (loop0): mounted filesystem without journal. Opts: norecovery,norecovery
[882233.697709] EXT4-fs (loop0): mounting ext3 file system using the ext4 subsystem
[882235.713249] EXT4-fs (loop0): write access unavailable, skipping orphan cleanup
[882235.713284] EXT4-fs (loop0): mounted filesystem without journal. Opts: norecovery,norecovery
[882294.370559] print_req_error: I/O error, dev loop0, sector 8912912
The I/O errors just then re-occur. The XO VM is left locked up at this point (kind of understandably).
I'm trying to work out what else I can do to look at this - e.g. is there a way of getting XO to log what it did to create the local loop0 mount? - So I can try and replicate from the CLI?
I've only tried this on small a small VM (~8Gbyte which works) and large (~100Gbyte which fails) VM so far - so I don't know if there's a size at which it "stops working"
If anyone can point me in the right direction of trying to get some more info of what XO's done to get to this point - that'd be great,
Thanks!
Hi,
I've got a volume on an NFS server that's mounted as 'Read Only' - which is exported via NFS to XO as a 'remote'.
XO can 'see' this - but complains it's read-only when you attach it.
Although it does attach - I can't then 'see' any backups on that remote. The only thing that seems to work is to re-mount it on the NFS server as 'read/write' - and re-attach it to XO as read-write.
At this point, the list of VM's within 'xo-vm-backup' appear - and are available to restore.
This kind of leads to the questions: Do 'remotes' always have to be read/write mounted / available to XO to allow even restores to be done?
Thanks!
Yes, thanks!
We have very limited control over rsync at our end (as it's running within a Synoloy NAS) - I think we probably have to just live with large transfers for now - whilst looking at spit files / VHD's as the way to go in the future.
Thanks again!
Hi - thanks for the info. That obviously covers the multi-file / block "new" system.
Do you know how the merge is handled on "single file" Delta backups?
e.g. Every day now with the existing delta backups we can see a new delta diff being created (20GByte) - and we can see the base image is "touched" (as the oldest diff gets rolled into it) - but what we can't see is if the old merge process does a "copy base + merge diffs" then a rename (which would suggest copy-on-write filesystems will see it all be touched)
Or if it merges the diff into the base "in place" (i.e. by seeking - and over-writing areas within the existing base 100G VHD).
I'm suspecting for safety it does a copy + merge to a new file, then delete old & rename or something...