[Solved] Yet another question about Backup - why so small backup created with XOA?

AlexTheBest

Have a nice day

I have got:
XCP-ng 8.2.1;
XOA from sources (started at another host);
1-st ssd with OS (XCP-ng 8.2.1);
2-nd ssd with VM's (img-files);
3-rd remote volume for backup through NFS;
VM (ubuntu server 22.04) (20 GB disk, used 10 Gb, 4 Gb RAM, 1 LAN etc);
Backup configured and works (Backup for "Backup" option, NFS etc) with XOA. It works only when VM is stopped.

So there 3 questions:

When I create Backup everything works fine and in "Logs" I see proud string "Successfull" and "Transfer: 27 KiB". Seriously, 27 KILOBYTES.

# ls -l
total 48
-rw-rw-r-- 1 x x  8878 Jul 28 05:06 20230728T050624Z.json
-rw-rw-r-- 1 x x 27648 Jul 28 05:06 20230728T050624Z.xva
-rw-rw-r-- 1 x x    36 Jul 28 05:06 20230728T050624Z.xva.checksum
-rw-rw-r-- 1 x x  2156 Jul 28 05:06 cache.json.gz

#

(don't know why Code tag shows 6 strings as one)

# ls -l
total 48
-rw-rw-r-- 1 x x  8878 Jul 28 05:06 20230728T050624Z.json

-rw-rw-r-- 1 x x 27648 Jul 28 05:06 20230728T050624Z.xva

-rw-rw-r-- 1 x x    36 Jul 28 05:06 20230728T050624Z.xva.checksum

-rw-rw-r-- 1 x x  2156 Jul 28 05:06 cache.json.gz

I realize that these "27 KiB" could be everything but not backup of VM (20 GB disk, used 10 Gb, 4 Gb RAM, 1 LAN etc).
So, how can I create real backup?

According point 1. every morning I copy IMG-file of my VM.
So, is it real backup which I could use in case of everything bad will happen with my VM to restore one to morning state? Just copy morning IMG instead of real?
Everybody I know (at least 2 persons, work at different places) why I asked to help me tells that "Backup with XOA" is useless and they use some magic scripts for backup and restore. As far as I understand, these scripts work directly with internal commands of XCP-ng.
So, could you please share with me link(s) to this scripts because I realize that it is unseriously to copy IMG-file(s) by hand every day

olivierlambert

You are doing "basic" backup, which is (ironically) not generated by XO but by XAPI/XCP-ng. So if you have a 1kiB backup (probably incorrect), XO has no mean to know it if XAPI says "backup successful". For that, you have backup healthcheck, but I recommend to go for a real backup code, the one made by XO.

Everybody I know (at least 2 persons, work at different places) why I asked to help me tells that "Backup with XOA" is useless and they use some magic scripts for backup and restore

We have thousands of users just getting XO for backup. I can guarantee that if XO backup didn't work, we (Vates) wouldn't exist anymore. See also my first point about where can be the issue.

edit: I'm editing your post to use Markdown correctly

florent

we also use some magic scripts for backup and restore, mostly here :

The only difference is that we share them, and support them in the long run.

edit : still no success to use this magic to summon an entity from a superior realm, but we 're working on it

AlexTheBest

olivierlambert, florent,

thank for your responses and links!

@olivierlambert said in Yet another question about Backup - why so small backup created with XOA?:

but I recommend to go for a real backup code, the one made by XO.

You are talking about console commands in XCP-ng console?
Something like

xe <command> <argument>=value <argument>=value

?

What about my 2-nd question?

(every morning I copy IMG-file of my VM.
So, is it real backup which I could use in case of everything bad will happen with my VM to restore one to morning state? Just copy morning IMG instead of real?)

florent

@AlexTheBest I don't really understand what is a IMG file in this context , to be fair

could you share me this 27Kb xva file ? it's clearly broken (like it probably don't have any disk data in it) and I may be able to help

AlexTheBest

@florent said in Yet another question about Backup - why so small backup created with XOA?:

@AlexTheBest I don't really understand what is a IMG file in this context , to be fair

Ok, lets go from begin.
How, where and in what kind XCP-ng store VM user created if selected "normal" volume (ext3) for VMs storage?
In my case it is folder with several large files (/ls). When I create a new VM, new file is appears there. If I remove some VM corresponding file disappears too at once. Thus, I have every reason to believe that these are images of my VMs. They sizes look like sizes of disk spaces of my VMs. And they called *img.
Here is part of list of them:

# ls -l /ls
...
-rw-r--r-- 1 root root 16106127360 Jul 24 14:42 5373f89b-ed31-4848-91c3-9aaca017543c.img
-rw-r--r-- 1 root root 21474836480 Jul 28 10:59 53ab0887-3dba-47e9-a3c0-7d04f9fe71b0.img
-rw-r--r-- 1 root root   524288000 Jul 28 04:02 dfded1dc-6847-462a-baea-472a0d86ee39.img
...

that is why I call them IMG-files.
(the third file seems to be some system file, it is not my VM)

@florent said in Yet another question about Backup - why so small backup created with XOA?:

could you share me this 27Kb xva file ? it's clearly broken (like it probably don't have any disk data in it) and I may be able to help

Please check your private chat. I've just sent you file.

florent

@AlexTheBest strange, the ova file doesn't have any disk inside , only the metadata (for example the network)

I think I understand : xen (and xen orchestra) can only manage disks that are handled through the xen api These are created as Vhd files.

If you use passthrough/remote disks/raw disk , you're on your own.

Do you use this form to create VM ?

If yes, what is a "normal" volume ?

AlexTheBest

florent,

yes I use this form when create a new VM

My "default SR" is folder "/ls"

# mount | grep ls
/dev/sdc1 on /ls type ext3 (rw,relatime)

not LVM

olivierlambert

This is not a correctly mounted SR

How did you created it in the first place?

AlexTheBest

@olivierlambert said in Yet another question about Backup - why so small backup created with XOA?:

How did you created it in the first place?

I suppose I just created with XOA. I have no other tools to deal with XCP-ng.

What should I do to correct this situation?
VM's are working...
How to move them?

olivierlambert

Can you do a mount command? /ls is NOT a mountpoint created by XAPI/XCP-ng, but something mounted manually, that's why I'm a bit surprised

AlexTheBest

@olivierlambert,

now I could not unmount /ls because now one of VM is working.
I suppose umount just will not unmount /ls because of opened files
Moreover /ls is only SR...
It could be little bit catastrofical for system. Isn't it?

olivierlambert

I said mount, to display all the mount, please read my posts carefully

AlexTheBest

@olivierlambert

sorry...

# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=3727468k,nr_inodes=931867,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/md126p1 on / type ext3 (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
xenfs on /proc/xen type xenfs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
xenstore on /var/lib/xenstored type tmpfs (rw,relatime,mode=755)
/dev/md126p3 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/md126p5 on /var/log type ext3 (rw,relatime)
/dev/sdd1 on /backups type ext3 (rw,relatime)
/dev/sdc1 on /ls type ext3 (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=747980k,mode=700)

#

olivierlambert

So you did mount manually /ls. Please explain each step you did, because I think you are not doing it the right way.

When you create a SR in XCP-ng/XO, the whole system will format the drive (or partition) for you, and mount it too.

It's mounted usually in /run/sr-mount/<SR UUID>.

So I'm sure you did that somehow different, and that might cause many issues. Please try to remember what you did exactly But in any case, you shouldn't NOT have img file in that folder. I'm under the impression you are using a RAW SR mounted manually as an ISO SR or something really really incorrect.

So, the issue is very likely a configuration issue form the start, that is making XVA export (and any kind of backup) failed. Just to be clear: XCP-ng is NOT your usual distro. You shouldn't NOT have to manually mount drives in it and go 100% via the API/XO/whatever client.

AlexTheBest

@olivierlambert,
What I did as I think:

Made standard installation with standard SR at VLM volume;
Created 2-nd SR (/ls);
Removed 1-st SR and set /ls as main SR;
Created VM;

I did not know that it is some special linux.

Do you think it is possible to convert current VM(s) to normal state?

I have one more ssd disk for temporary using. It could make process faster.

olivierlambert

How did you create the SR?

If you have an available extra partition or disk, you can use it do make a normal/viable SR. In XO, new/storage, local LVM/ext (both are OK) and enter the partition. Please use a NON formatted partition. It will do everything for you (install the FS, mount it). Then migrate your VMs to it.

See https://docs.xcp-ng.org/storage/#local for a screenshot on SR creation.

When you'll have a correct setup, backup will magically work

AlexTheBest

@olivierlambert,

Do you think that my current VMs normally migrate to new created SR?
Ok, I will try and see by myself and write here.

AlexTheBest

I have got reason to be proud of me.

I did!

Firstly, I installed spare ssd disk and add one as new SR (ext3).

[for those looking for an adventure up their own arse]
this morning was started disgustingly
all VMs have forgot their disks (CPU presents, RAM and LAN too, DISK absent (0 kb)) and did not started.
I honestly tried to use my "copy(es) of images" - IMG-files I wrote about above (do not ask me how these IMG-files was created - I don't know).
It did not works directly. System did not recognize IMG-files as something useful.
I converted my IMG-files to vhd

qemu-img convert  53ab0887-3dba-47e9-a3c0-7d04f9fe71b0.img -O vpc -o subformat=dynamic  53ab0887-3dba-47e9-a3c0-7d04f9fe71b0.vhd

Ok, last week one of my VM had 20 Gb of disk space (now no disk at all). I added new empty 20 Gb disk to VM. And then "Import VDI" just converted vhd-file to newborn 20 Gb disk.
Voilà! VM started!!!
[/for those looking for an adventure up their own arse]

After all my VM started NORMALLY (!) (backup works and creates gigabytes' sized backup, snapshots work too) I migrated all VM to new SR.

There are some questions I would like to clarify:
New SR - SR I created today, right SR;
Old SR - bad SR (/ls), where IMG-files are.
~~1. Although I have migrated all my VM to new SR, IMG-files are still at old SR. I would like to remove old SR and create a new (right, ext3) one instead.~~
~~So, when I remove these IMG-files if I don't get into another round of trouble (something like "i won't start. where is my starting image?")?~~
2. One of my acquaintances insists on that SR should be under RAID. At least "mirror". I have OS's volume with "mirror" and about 2 Tb of unused space there.
So, is RAID necessary for SR?
And, is it a good idea to place SR at system disk (RAID1)?
~~And how can I set a selected SR as default?~~

Upd1
@olivierlambert said in [Solved] Yet another question about Backup - why so small backup created with XOA?:

How did you create the SR?

In the status of this SR was one interesting string "legacy=yes". I suppose this explains how I reached this unusual result (IMG-files which nobody beside me did not see before, very small backup etc)

Upd2
I have answered to my questions by myself
(firstly, migrate everything to a new SR, set it as default if necessary

xe pool-param-set uuid=xxxx default-SR=xxxx

)

florent

@AlexTheBest great work and effort alex , and welcome to modern storage .