@MajorP93 said:
What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer.
Yes, that's it - sorry - I should have been more explicit, even if 'lumbers along' wasn't a hint 
//EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool.
Pedantic, but true 
I technically meant 'not hosts in the source pool' I guess.
But we do that all the time? - i.e. Two non-pool hosts, live migrate between them (these happen to have the same CPU / family etc.)
On XCP-ng?
Yes. All the time, from memory it's always been possible (well, maybe not 'always' depending on how far you go back) - but certainly as long as I can remember (considering we've been Xen since I think Version 6? - now xcp-ng 8.3).
If I understand it right, Tackyone wants to dumb down all VM's to have the least common denominator CPU instructions so he can migrate any guest to any host. I personally think that's a bad direction, but I understand his desire...Pool a bunch of random systems laying around into a common farm. I think that's a bit too low-end of a goal for Vates to spend any time on myself. That seems exclusively homelab to me and counter to moving forward in the industry.
Yes, that's it. It's 'counter to moving forward in the industry' until you can't actually get the machines that make up existing pools / hosts you have to extend them.
Keeping everything 'dumber' would just make life a lot simpler (whether it be home lab, or systems built up over years as they move forward hardware generations).
Hi,
AFAIK this is not possible. Live migration does only work within the same pool. So you can not live migrate VMs to another pool / host that is not part of the pool.
But we do that all the time? - i.e. Two non-pool hosts, live migrate between them (these happen to have the same CPU / family etc.)
@Tackyone Even with the current AI-induced hardware crunch, yester-year hardware isn't all that hard to come by. Why not just get a pile of systems that simply match? You'll have predictable and identical performance across the whole farm without any CPU features or enhancements being lost or ignored. Besides, in order for all the XCP-ng hosts to work together they (ideally) need shared storage and they have to have identical network configurations. Otherwise you've got slow migrations and network re-mapping every time you move something.
Just my thought.
I do see your point, but I already have a pile of systems
We already do live migration on same-CPU set stuff, without shared storage - sure it's lumbers a bit, but running a few in parallel isn't so bad. The network side of things takes care of itself - providing the networks are named the same on the hosts you're moving from / to.
It'd just be handy to settle on a single CPU spec for all this - and then not have to worry about either live migrates, or picking up two of the hosts and pooling them.
Hi,
I've got a number of xcp-ng 8.3 machines running - which are mixed CPU's.
I know if I pool them, the system will look at the CPU's - decide on a 'lowest common denominator' feature set - and use that, so that newly started VM's can be live migrated between members (and old ones can be restarted etc. etc.)
I know there are at least some options for manually setting CPU masks to mask features - but is there any way to tell xcp-ng "I want all my CPU's to be potatoes?"
This would be more akin apparently to how ProxMox works - in that all the VM's think they're running on pretty old CPU's (e.g. SSE2, but nothing new & fancy) - this is apparently done to allow live migrates between dissimilar hosts.
I can see disadvantages to this - but from the perspective from having a bunch of dis-jointed xcp-ng hosts on varying CPU's, doing something similar on xcp-ng (even if it has to be done manually) does seem quite appealing - if it gets live migrates between non-pooled, dissimilar CPU hosts working.
Is there any way to do this - i.e. set some kind of 'master mask' on the xcp-ng host, so any VM's that are brought up just see 'potato CPU' - on all the hosts?
Hi,
We use the 'Remote syslog' option with XCP-ng 8.2.1. This has been working well - but recently we patched the pool, rebooted the pool (everything came back ok) - but Remote syslog - doesn't log any more.
If I go into XO and look at the option for the Hosts - on both it's set to the correct IP address.
If I try changing that IP to a different host on the network XO quite correctly said something like "can't use that expect port to be open".
If I change it back to the original host, I've just noticed - it starts working again.
This is a bit annoying (as we've lost days of logs) - anyone else seen similar with that option?
[Before figuring that changing it 'fixes' it - I'd already tested the syslog server was up, and I could send test log entries to it using 'logger' from the XCP-ng's dom0].
Thanks!
@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:
Yes, that sounds the next step to do
Ok, filesystem checks out - as far as I can see having:
I upped the RAM for XOA from default 2GiB to 16GiB and re-tried - and same result.
I can try yet more RAM if needs be but would hope 16GiB is more than enough?
The console error on XOA is more detailed than my install- so don't know if this helps:
blk_update_request: I/O error, dev loop0, sector 8912912 op 0x00:(READ) flags 0x80700 phys_seg 1 prio class 0
Looks to be the same issue, just reported differently.
You also (on both systems) get kernel chatter about "tasked blocked" (understandable, as it is).
@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:
That's why you should try on XOA, not the sources. This way, we can see if it's a XO bug or an issue on your local source install
Ok, latest XOA (updated) has the same issue - process dies with loop0 I/O errors logged to console same as my 'from sources' version.
I'm going to do a full restore of the VM spin it up, and do a full filesystem check on it (well as good as ext3/4 can) - just to make 100% sure there's no corruption - as that seems sensible start / relatively easy to do.
Sorry - missed the 'XOA' only saw 'latest'... More coffee needed... Off to try.
@olivierlambert said in file restore on large backups ends in print_req_error: I/O error's:
First, I would try with an XOA in
latestto compare the outcome
Hi,
Ok - moved from 2af84 to apparently 'master' at ce2b918a2 (let me know if that's wrong, I spend my life in SVN not Git).
This does exactly the same thing - same issue, same syslog error.
The backup being used for the file restore, is a 100G 'MBR' (i.e. legacy) disk split into 4 partitions, the last partition '/var' being ~78Gbyte in size (which is the one I select to restore from).
The source is a Delta backup (with a retention of 4).
ps. Sorry my log output above doesn't look formatted well, I don't know if this is a Safari thing or not - I did put 'code' markup before/after but here it looks like it's collapsed down to one line 
@olivierlambert said in Can't restore VM's from 'read-only' remote?:
A remote (now Backup repository) is meant to write backups into it. If you aren't able to write, then, sure, you can restore but you won't be able to backup anything.
Yes, understand that - I'll have to re-test here, but I'm sure when it was 'read-only' - I couldn't restore anything from it, as nothing showed up in XO's "Backup / Restore' list of backups...
The remote was mounted on two systems - hence why made R/O for one of them (this was just for testing restores).
Hi,
I'm running XO from sources (2af84) under a Debian stretch VM.
I've got all the dependancies installed to do 'file restore' from backups - and this works, on small backups (e.g. ~8Gbyte).
However - a much larger VM I have backed up (~100Gbyte) fails.
Looking on the system - I can see it's created the loop back (loop0) for it, and mounted it on the XO VM, but then you start getting:
[882294.370559] print_req_error: I/O error, dev loop0, sector 8912912
Type errors. This coincides with using XO to 'browse' through the directory structure of the backup (i.e. traverse the mounted backup filesystem).
There's not a lot else logged in syslog, e.g. from the point you select a backup to file restore from, I see:
[878193.892486] fuse init (API version 7.27)
[878195.968810] loop: module loaded
[878196.269943] EXT4-fs (loop0): write access unavailable, skipping orphan cleanup
[878196.269996] EXT4-fs (loop0): mounted filesystem without journal. Opts: norecovery,norecovery
[882233.697709] EXT4-fs (loop0): mounting ext3 file system using the ext4 subsystem
[882235.713249] EXT4-fs (loop0): write access unavailable, skipping orphan cleanup
[882235.713284] EXT4-fs (loop0): mounted filesystem without journal. Opts: norecovery,norecovery
[882294.370559] print_req_error: I/O error, dev loop0, sector 8912912
The I/O errors just then re-occur. The XO VM is left locked up at this point (kind of understandably).
I'm trying to work out what else I can do to look at this - e.g. is there a way of getting XO to log what it did to create the local loop0 mount? - So I can try and replicate from the CLI?
I've only tried this on small a small VM (~8Gbyte which works) and large (~100Gbyte which fails) VM so far - so I don't know if there's a size at which it "stops working"
If anyone can point me in the right direction of trying to get some more info of what XO's done to get to this point - that'd be great,
Thanks!
Hi,
I've got a volume on an NFS server that's mounted as 'Read Only' - which is exported via NFS to XO as a 'remote'.
XO can 'see' this - but complains it's read-only when you attach it.
Although it does attach - I can't then 'see' any backups on that remote. The only thing that seems to work is to re-mount it on the NFS server as 'read/write' - and re-attach it to XO as read-write.
At this point, the list of VM's within 'xo-vm-backup' appear - and are available to restore.
This kind of leads to the questions: Do 'remotes' always have to be read/write mounted / available to XO to allow even restores to be done?
Thanks!
Yes, thanks!
We have very limited control over rsync at our end (as it's running within a Synoloy NAS) - I think we probably have to just live with large transfers for now - whilst looking at spit files / VHD's as the way to go in the future.
Thanks again!
Hi - thanks for the info. That obviously covers the multi-file / block "new" system.
Do you know how the merge is handled on "single file" Delta backups?
e.g. Every day now with the existing delta backups we can see a new delta diff being created (20GByte) - and we can see the base image is "touched" (as the oldest diff gets rolled into it) - but what we can't see is if the old merge process does a "copy base + merge diffs" then a rename (which would suggest copy-on-write filesystems will see it all be touched)
Or if it merges the diff into the base "in place" (i.e. by seeking - and over-writing areas within the existing base 100G VHD).
I'm suspecting for safety it does a copy + merge to a new file, then delete old & rename or something...
Hi,
Final question on this - rsync 'block' mode failed miserably to make a difference.
When XO "merges" diffs into the base image for delta backups - does it do this by just changing updated blocks -within the existing file- or does it do something like "copy whole base image to a new file, merging the diffs as it goes - then rename at the end"?
We looked at multiple-blocks - but "not ext4" kind of delays us actually trying it for now...
Thanks!
Thanks for the details / options. The boxes we're using support rsync 'block difference' mode - so we're giving that a try (on paper it should help if the boxes have enough 'grunt' to do this).
Failing that we'll check out the 'store backup as multiple data blocks' option.
Looking forward to the backup tiering - backups has been (and continues to be) one of many things XO does really, really well.
Cheers!
Hi,
We have a system where we've setup delta backups to an NFS remote that's on the local LAN.
These work really well, e.g. for the first backup we can see 100Gbyte being backed up, and subsequent backups for our VM run to about 10-15Gbyte.
We've been trying to rsync that local 'Remote' to another box over a WAN - but run into issues (that it takes for ever, a lot of data seems to have changed etc.)
Looking at the raw files on the XO "Remote" - I can see the UUID directory for the VM - and, buried away under the VDI's subdirectory etc. - I can see the VHD files.
This seems to comprise of the 100GByte 'base' VHD - and then a number of smaller differentials (as you'd expect).
But after a backup run - the remote NFS seems to show that the base .VHD has been changed (thus rsync will dutifully copy over the 100GByte file again).
Do the delta backups modify this base VHD? - Maybe to apply the oldest diff to bring it forward in time before deleting that older diff?
Just trying to get my head around how these work - and what the impact will be for rsync.
Is there any better mode we could use when the remote is to be rsync'd off site? - Unfortunately we can't write directly to the off-site NFS - we only really have rsync access.
Thanks!
Hi,
Is there any way to have Disaster Recovery backups remove (i.e. toggle to "off") the auto-start flag on VM's as they're replicated?
A source VM here has 'auto-start: yes' set - if it's replicated twice - that's a lot of VM's sitting around with "auto-start" enabled waiting for something to go wrong (i.e. both start at the same time etc.)