Problem with file level restore from delta backup from LVM parition

Danp

othmar Install XOA in trial mode and then retest to see if the issue is present with it.

othmar

Danp
yes it's the same on XOA in trial mode latest patched.

lukas

It's same here. Problem is in community version and in xoa trial.

olivierlambert

There's obviously something different between the VMs you use and ours for tests. Would you be able to provide a VM export so we can deploy it on our side, backup it and see why file restore doesn't work?

shadowdao

I am seeing the same issue with LVM restores. Exists between Trial and Community editions.

backupNg.listFiles
{
  "remote": "c6f2b11a-4065-4a8b-b75f-e16bf2aeb5f5",
  "disk": "xo-vm-backups/8a0d70df-9659-b307-43f4-fc37133d9d66/vdis/2dcbfc0b-5d0a-4f84-b7ec-e89b1747e0b4/47f611b1-7c83-4319-9fcf-aad09a025edc/20200319T145807Z.vhd",
  "path": "/",
  "partition": "ea853d6d-01"
}
{
  "command": "mount --options=loop,ro,offset=1048576 --source=/tmp/tmp-458p174rEvrE3fU/vhdi1 --target=/tmp/tmp-458llR4TibV7F12",
  "exitCode": 32,
  "stdout": "",
  "stderr": "mount: /tmp/tmp-458llR4TibV7F12: failed to setup loop device for /tmp/tmp-458p174rEvrE3fU/vhdi1.",
  "failed": true,
  "timedOut": false,
  "isCanceled": false,
  "killed": false,
  "message": "Command failed with exit code 32: mount --options=loop,ro,offset=1048576 --source=/tmp/tmp-458p174rEvrE3fU/vhdi1 --target=/tmp/tmp-458llR4TibV7F12",
  "name": "Error",
  "stack": "Error: Command failed with exit code 32: mount --options=loop,ro,offset=1048576 --source=/tmp/tmp-458p174rEvrE3fU/vhdi1 --target=/tmp/tmp-458llR4TibV7F12
    at makeError (/opt/xen-orchestra/node_modules/execa/lib/error.js:56:11)
    at handlePromise (/opt/xen-orchestra/node_modules/execa/index.js:114:26)
    at <anonymous>"
}

olivierlambert

Do you have a VG using multiple PVs?

shadowdao

That one does not. I assumed that was the case for another one that had the issue, and working to move the VMs to VHDs that are not on LVM.

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/xvda2
  VG Name               centos
  PV Size               79.51 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              20354
  Free PE               16
  Allocated PE          20338
  PV UUID               zkCwDd-03ZP-LhLO-drIw-5fpY-Bf7f-J99g0U

olivierlambert

Same problem on XOA latest?

edit: ping julien-f and/or badrAZ

shadowdao

Both the built community edition and the XOA appliance seem to have this issue.

sccf

badrAZ julien-f

A little more info... I'm also having this problem, but it seems only non-Linux file systems have a problem on my side. Let me know if I can run/supply anything on this side that might be useful?

Debian, CentOS and Ubuntu files recoveries are all fine, but Windows Server, Windows 10 and pfSense drives are throwing the same error others have posted previously. Error seems to start a step earlier than what has been previously suggested though?

Server is XO Community on Debian, NFS remote (Synology NAS) all LVMs I think.

Selecting backup date is fine. Selecting drive from the backup gives this error:

Mar 25 06:05:32 BKP01 xo-server[431]: { Error: spawn fusermount ENOENT
Mar 25 06:05:32 BKP01 xo-server[431]:     at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
Mar 25 06:05:32 BKP01 xo-server[431]:     at onErrorNT (internal/child_process.js:362:16)
Mar 25 06:05:32 BKP01 xo-server[431]:     at _combinedTickCallback (internal/process/next_tick.js:139:11)
Mar 25 06:05:32 BKP01 xo-server[431]:     at process._tickCallback (internal/process/next_tick.js:181:9)
Mar 25 06:05:32 BKP01 xo-server[431]:   errno: 'ENOENT',
Mar 25 06:05:32 BKP01 xo-server[431]:   code: 'ENOENT',
Mar 25 06:05:32 BKP01 xo-server[431]:   syscall: 'spawn fusermount',
Mar 25 06:05:32 BKP01 xo-server[431]:   path: 'fusermount',
Mar 25 06:05:32 BKP01 xo-server[431]:   spawnargs: [ '-uz', '/tmp/tmp-431GWsDT1lOqXyY' ],
Mar 25 06:05:32 BKP01 xo-server[431]:   originalMessage: 'spawn fusermount ENOENT',
Mar 25 06:05:32 BKP01 xo-server[431]:   command: 'fusermount -uz /tmp/tmp-431GWsDT1lOqXyY',
Mar 25 06:05:32 BKP01 xo-server[431]:   exitCode: undefined,
Mar 25 06:05:32 BKP01 xo-server[431]:   signal: undefined,
Mar 25 06:05:32 BKP01 xo-server[431]:   signalDescription: undefined,
Mar 25 06:05:32 BKP01 xo-server[431]:   stdout: '',
Mar 25 06:05:32 BKP01 xo-server[431]:   stderr: '',
Mar 25 06:05:32 BKP01 xo-server[431]:   failed: true,
Mar 25 06:05:32 BKP01 xo-server[431]:   timedOut: false,
Mar 25 06:05:32 BKP01 xo-server[431]:   isCanceled: false,
Mar 25 06:05:32 BKP01 xo-server[431]:   killed: false }

Partitions still load, but then this error occurs when selecting a partition:

Mar 25 06:07:51 BKP01 xo-server[431]: { Error: spawn fusermount ENOENT
Mar 25 06:07:51 BKP01 xo-server[431]:     at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
Mar 25 06:07:51 BKP01 xo-server[431]:     at onErrorNT (internal/child_process.js:362:16)
Mar 25 06:07:51 BKP01 xo-server[431]:     at _combinedTickCallback (internal/process/next_tick.js:139:11)
Mar 25 06:07:51 BKP01 xo-server[431]:     at process._tickCallback (internal/process/next_tick.js:181:9)
Mar 25 06:07:51 BKP01 xo-server[431]:   errno: 'ENOENT',
Mar 25 06:07:51 BKP01 xo-server[431]:   code: 'ENOENT',
Mar 25 06:07:51 BKP01 xo-server[431]:   syscall: 'spawn fusermount',
Mar 25 06:07:51 BKP01 xo-server[431]:   path: 'fusermount',
Mar 25 06:07:51 BKP01 xo-server[431]:   spawnargs: [ '-uz', '/tmp/tmp-431yl3BiFVbSmWB' ],
Mar 25 06:07:51 BKP01 xo-server[431]:   originalMessage: 'spawn fusermount ENOENT',
Mar 25 06:07:51 BKP01 xo-server[431]:   command: 'fusermount -uz /tmp/tmp-431yl3BiFVbSmWB',
Mar 25 06:07:51 BKP01 xo-server[431]:   exitCode: undefined,
Mar 25 06:07:51 BKP01 xo-server[431]:   signal: undefined,
Mar 25 06:07:51 BKP01 xo-server[431]:   signalDescription: undefined,
Mar 25 06:07:51 BKP01 xo-server[431]:   stdout: '',
Mar 25 06:07:51 BKP01 xo-server[431]:   stderr: '',
Mar 25 06:07:51 BKP01 xo-server[431]:   failed: true,
Mar 25 06:07:51 BKP01 xo-server[431]:   timedOut: false,
Mar 25 06:07:51 BKP01 xo-server[431]:   isCanceled: false,
Mar 25 06:07:51 BKP01 xo-server[431]:   killed: false }
Mar 25 06:07:51 BKP01 xo-server[431]: 2020-03-24T20:07:51.178Z xo:api WARN admin@admin.net | backupNg.listFiles(...) [135ms] =!> Error: Command failed with exit code 32: mount --options=loop,ro,offset=608174080 --source=/tmp/tmp-431yl3BiFVbSmWB/vhdi15 --target=/tmp/tmp-431ieOQYw52Cr6a

olivierlambert

sccf can you confirm same issue on XOA?

sccf

OK, so XOA is working fine on my side except for the pfSense/FreeBSD VM, but to be honest I'm not sure I have ever tried a file level restore on it before and can't see any reason why I would ever need to in future.

I guess I'm looking for an environment issue then, perhaps a dependency? I've got 2 independent sites (one at home, one at our local church) that are configured the same and having the same problems. Will try setting up on Ubuntu and see if that resolves.

badrAZ

shadowdao sccf

Hi,

This issue isn't easy to reproduce in our lab. Can you please provide a VM export with steps to reproduce this issue please?

It will help us to diagnostic the issue.

shadowdao

The VHD is about 80GB and this is one of the smaller ones. Do you have a secure location that I can upload it. The VHD has client data on it, so even if I scrub it, I don't want accessible publicly.

olivierlambert

Please open a support ticket on xen-orchestra.com

sccf

Hi again, apologies for my tardiness. I've switched to Ubuntu server rather than Debian and all appears to be well again. I can only assume that it is an environment/dependency issue in Debian.

wuchererbe

Hi,

i have the same Problem - im using Ubuntu 20.04 with logic volumes (LVM). If I restore a file it only shows the red triangle. Is there any solution?

backupNg.listFiles
{
  "remote": "9f5e3132-e828-4aef-b083-1053891ce2e7",
  "disk": "/xo-vm-backups/e28ecb04-027c-00cf-8c30-6977d1f15490/vdis/7d5759ec-475d-428a-a945-57fde5c89562/c83dbbd2-9333-4a6d-aee2-e807c97f4b4b/20220114T210004Z.vhd",
  "path": "/",
  "partition": "f80fbe2e-0ce9-448a-a56a-42f73bbb6373"
}
{
  "killed": false,
  "code": 32,
  "signal": null,
  "cmd": "mount --options=loop,ro,sizelimit=106297294848,offset=1075838976 --source=/tmp/y7s1u7vrnrf/vhdi10 --target=/tmp/xysii549sk9",
  "message": "Command failed: mount --options=loop,ro,sizelimit=106297294848,offset=1075838976 --source=/tmp/y7s1u7vrnrf/vhdi10 --target=/tmp/xysii549sk9
mount: /tmp/xysii549sk9: unknown filesystem type 'LVM2_member'.
",
  "name": "Error",
  "stack": "Error: Command failed: mount --options=loop,ro,sizelimit=106297294848,offset=1075838976 --source=/tmp/y7s1u7vrnrf/vhdi10 --target=/tmp/xysii549sk9
mount: /tmp/xysii549sk9: unknown filesystem type 'LVM2_member'.

    at ChildProcess.exithandler (child_process.js:383:12)
    at ChildProcess.emit (events.js:400:28)
    at ChildProcess.emit (domain.js:475:12)
    at ChildProcess.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202201061020/@xen-orchestra/log/configure.js:118:17)
    at maybeClose (internal/child_process.js:1058:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:293:5)
    at Process.callbackTrampoline (internal/async_hooks.js:130:17)"
}

olivierlambert

Are you testing on latest XO virtual Appliance with XO also up to date?

wuchererbe

olivierlambert

xo-server 5.86.3 and xo-web 5.91.2
I also have tested it with the latest XOA.

olivierlambert

Are you using multiple disks under the same VG or LV?