<![CDATA[XO - Files Restore]]>Hi o/

I'm facing an issue with file restoration.

On some VMs, I can see the file and restore them :

11242d2c-4544-4498-bc73-038193ddba6b-image.png

On other VMs, I can't :

2cb81d39-d309-4714-aa7b-1c6567ebef44-image.png

Both VMs are backed up in the same way, using the same job.
Backups are done on an NFS share, and remote backups are encrypted on XO.

Any idea why this is happening?

]]>https://xcp-ng.org/forum/topic/10789/xo-files-restoreRSS for NodeMon, 08 Jun 2026 05:24:23 GMTMon, 28 Apr 2025 15:45:50 GMT60<![CDATA[Reply to XO - Files Restore on Tue, 03 Jun 2025 13:11:31 GMT]]>@lsouai-vates Is this a description of the cause of the problem ? From what you describe, if one selects a backup containing a LVM partition scheme, it should be able to be mounted at least the first time ?
As in here: I verified that this machine is using LVM first, and when selecting it, I immediately selected the large LVM partition to try to restore a file from that (which failed).
e7148e43-aa01-433d-8af4-a936e30550e6-image.png
The machine (Debian 12) running XO do not itself use LVM, so "ubuntu-vg" should be free for mounting this first time.

]]>https://xcp-ng.org/forum/post/93631https://xcp-ng.org/forum/post/93631Tue, 03 Jun 2025 13:11:31 GMT<![CDATA[Reply to XO - Files Restore on Tue, 03 Jun 2025 09:04:26 GMT]]>@xcplak @peo @Andrew some reply from XO dev:

"from ubuntu 20 ( I think) the default partition scheme includes a lvm group name ubuntu-vg . Any duplicate on this will be unmountable in XO
the first one will work, but not the next one, at least until it is dismounted ( after 10 minutes unused)
DEcreasing this delay won't change the root cause, and will cause other issues when user deselect/Reselect a disk"

]]>https://xcp-ng.org/forum/post/93621https://xcp-ng.org/forum/post/93621Tue, 03 Jun 2025 09:04:26 GMT<![CDATA[Reply to XO - Files Restore on Wed, 28 May 2025 23:29:23 GMT]]>I can confirm this as well

several new ubuntu 24 vm's with LVM having this problem. I thought i was going crazy

existing debian 10,11,12 VM's using LVM are not effected
vm's not using LVM are not effected.

]]>https://xcp-ng.org/forum/post/93413https://xcp-ng.org/forum/post/93413Wed, 28 May 2025 23:29:23 GMT<![CDATA[Reply to XO - Files Restore on Wed, 28 May 2025 15:53:38 GMT]]>Great that this bug/problem is being confirmed by others. Reproducing is as simple as create a new Linux VM (only using defaults when installing Linux), back it up, then try to restore files.
Restoring single files is a feature that is at least needed in production environments (anything outside the "home lab"). Personally, I have no problem with waiting an hour or two for a full restore to a temporary VM to be able to access a file deleted or modified by mistake.
There are people willing to help find and pinpoint problems like this, but having to use XOA to get attention to problems, given that it requires a license beyond the free trial month makes it less appealing for us spend our free time to help with this.

]]>https://xcp-ng.org/forum/post/93400https://xcp-ng.org/forum/post/93400Wed, 28 May 2025 15:53:38 GMT<![CDATA[Reply to XO - Files Restore on Tue, 27 May 2025 23:59:20 GMT]]>@Andrew
I'm not sure I understand what you meant.
Full VM restore is working, the issue is with file-level restore. So restoring a full 200GB VM just to retrieve 2 or 3 files isn't really a viable solution.

]]>https://xcp-ng.org/forum/post/93376https://xcp-ng.org/forum/post/93376Tue, 27 May 2025 23:59:20 GMT<![CDATA[Reply to XO - Files Restore on Tue, 27 May 2025 23:39:17 GMT]]>@xcplak With a LVM guest, you can restore the whole VM now. The backup data is actually saved correctly and it is possible to manually restore a single file. So they can fix XO and the old backup data is still good.

]]>https://xcp-ng.org/forum/post/93375https://xcp-ng.org/forum/post/93375Tue, 27 May 2025 23:39:17 GMT<![CDATA[Reply to XO - Files Restore on Tue, 27 May 2025 23:24:54 GMT]]>@peo Thanks for the info!
I also tested it with XOA and got the same result.
However, when deploying the same VM without LVM, file restore worked perfectly with both XO and XOA.
It definitely seems related to the GitHub issue.

In the meantime, I've switched to a commercial backup solution until it's resolved.

]]>https://xcp-ng.org/forum/post/93374https://xcp-ng.org/forum/post/93374Tue, 27 May 2025 23:24:54 GMT<![CDATA[Reply to XO - Files Restore on Mon, 26 May 2025 15:08:21 GMT]]>@olivierlambert better ask @florent

]]>https://xcp-ng.org/forum/post/93302https://xcp-ng.org/forum/post/93302Mon, 26 May 2025 15:08:21 GMT<![CDATA[Reply to XO - Files Restore on Mon, 26 May 2025 05:04:37 GMT]]>Pinging @lsouai-vates

]]>https://xcp-ng.org/forum/post/93286https://xcp-ng.org/forum/post/93286Mon, 26 May 2025 05:04:37 GMT<![CDATA[Reply to XO - Files Restore on Mon, 26 May 2025 01:35:06 GMT]]>@peo Still an existing problem:
https://github.com/vatesfr/xen-orchestra/issues/7029

andrew64k created this issue in vatesfr/xen-orchestra

open File level restoration not working on LVM partition #7029

]]>https://xcp-ng.org/forum/post/93283https://xcp-ng.org/forum/post/93283Mon, 26 May 2025 01:35:06 GMT<![CDATA[Reply to XO - Files Restore on Sun, 25 May 2025 23:12:57 GMT]]>From the log on the XOA-VM (/var/lib/xo-server/data/leveldb/001205.log)

{"method":"backupNg.listFiles","params":{"remote":"3f679c65-1290-4264-8ede-d91d08b2abf0","disk":"/xo-vm-backups/f94129f5-5c2c-6d6b-51e7-eeed24bf965b/vdis/6f239189-6971-4d3a-8e6a-5d43bd8bdce3/e163632c-9dca-4947-9c44-b2b794008294/20250522T184952Z.alias.vhd","path":"/","partition":"23a71973-6022-44f5-b51a-6d48b93c6cfd"},"name":"API call: backupNg.listFiles","userId":"3ffcc4df-8f09-41ce-99dc-8a1f1c568c56","type":"api.call"},"start":1748214110665,"status":"failure","updatedAt":1748214110991,"end":1748214110991,"result":{"code":32,"killed":false,"signal":null,"cmd":"mount --options=loop,ro,norecovery,sizelimit=18594398208,offset=2879389696 --source=/tmp/axvvrbout2/vhd0 --target=/tmp/4rksxkr68si","message":"Command failed: mount --options=loop,ro,norecovery,sizelimit=18594398208,offset=2879389696 --source=/tmp/axvvrbout2/vhd0 --target=/tmp/4rksxkr68si\nmount: /tmp/4rksxkr68si: unknown filesystem type 'LVM2_member'.\n       dmesg(1) may have more information after failed mount system call.\n","name":"Error","stack":"Error: Command failed: mount --options=loop,ro,norecovery,sizelimit=18594398208,offset=2879389696 --source=/tmp/axvvrbout2/vhd0 --target=/tmp/4rksxkr68si\nmount: /tmp/4rksxkr68si: unknown filesystem type 'LVM2_member'.\n       dmesg(1) may have more information after failed mount system call.\n\n    at genericNodeError (node:internal/errors:984:15)\n    at wrappedFn (node:internal/errors:538:14)\n    at ChildProcess.exithandler (node:child_process:422:12)\n    at ChildProcess.emit (node:events:518:28)\n    at ChildProcess.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)\n    at maybeClose (node:internal/child_process:1104:16)\n    at Socket.<anonymous> (node:internal/child_process:456:11)\n    at Socket.emit (node:events:518:28)\n    at Socket.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)\n    at Pipe.<anonymous> (node:net:343:12)\n    at Pipe.callbackTrampoline (node:internal/async_hooks:130:17)"}}j▒▒5▒▒▒!tasks!0mb49jl91%▒▒▒▒▒!tasks!0mb49jl97▒{
]]>https://xcp-ng.org/forum/post/93282https://xcp-ng.org/forum/post/93282Sun, 25 May 2025 23:12:57 GMT<![CDATA[Reply to XO - Files Restore on Sun, 25 May 2025 22:53:45 GMT]]>@xcplak for what it's worth, I confirm that file restore from LVM partitions do not work using XOA (I activated my trial just to test this):
821f8ea3-54a5-4d61-a860-ea68b9229742-image.png

]]>https://xcp-ng.org/forum/post/93280https://xcp-ng.org/forum/post/93280Sun, 25 May 2025 22:53:45 GMT<![CDATA[Reply to XO - Files Restore on Tue, 29 Apr 2025 22:00:28 GMT]]>@olivierlambert,

Why would this issue be related to my NFS share? I can access the share without any problems, read and write operations work fine. Other NFS shares with multiple applications running are also functioning normally.

I checked the XO logs and found the following:

xo-server[679]: 2025-04-29T21:45:58.276Z xo:api WARN  | backupNg.listFiles(...) [114ms] =!> Error: Command failed: vgchange -ay vg_iredmail
xo-server[679]: File descriptor 22 (/var/lib/xo-server/data/leveldb/LOG) leaked on vgchange invocation. Parent PID 679: node
xo-server[679]: File descriptor 24 (/var/lib/xo-server/data/leveldb/LOCK) leaked on vgchange invocation. Parent PID 679: node
xo-server[679]: File descriptor 25 (/dev/fuse) leaked on vgchange invocation. Parent PID 679: node
xo-server[679]: File descriptor 26 (/var/lib/xo-server/data/leveldb/MANIFEST-000103) leaked on vgchange invocation. Parent PID 679: node
xo-server[679]: File descriptor 31 (/dev/fuse) leaked on vgchange invocation. Parent PID 679: node
xo-server[679]: File descriptor 34 (/var/lib/xo-server/data/leveldb/000356.log) leaked on vgchange invocation. Parent PID 679: node
xo-server[679]:   WARNING: Couldn't find device with uuid oWTE8A-LNny-Ifsc-Jmtp-lTkB-dSjt-2AqhwY.
xo-server[679]:   WARNING: VG vg_iredmail is missing PV oWTE8A-LNny-Ifsc-Jmtp-lTkB-dSjt-2AqhwY (last written to /dev/xvda4).
xo-server[679]:   Refusing activation of partial LV vg_iredmail/lv_root.  Use '--activationmode partial' to override.

To me, this looks more like an issue with file restore than the NFS itself.

It seems similar to what's reported here:
https://github.com/vatesfr/xen-orchestra/issues/7029

andrew64k created this issue in vatesfr/xen-orchestra

open File level restoration not working on LVM partition #7029

]]>https://xcp-ng.org/forum/post/92450https://xcp-ng.org/forum/post/92450Tue, 29 Apr 2025 22:00:28 GMT<![CDATA[Reply to XO - Files Restore on Tue, 29 Apr 2025 19:43:30 GMT]]>You have a remote (backup repo) configuration issue. That's unrelated, check your permissions.

]]>https://xcp-ng.org/forum/post/92444https://xcp-ng.org/forum/post/92444Tue, 29 Apr 2025 19:43:30 GMT<![CDATA[Reply to XO - Files Restore on Tue, 29 Apr 2025 08:20:44 GMT]]>I just deployed a new NFS share for testing.
The share is mounted on both XO and the newly deployed XOA.
I backed up a VM that I can see the files in using XO.
I faced the same issue with both XOA and XO, I can’t see the files.
I backed up the same VM using XOA and tried to restore the files, same issue.

]]>https://xcp-ng.org/forum/post/92433https://xcp-ng.org/forum/post/92433Tue, 29 Apr 2025 08:20:44 GMT<![CDATA[Reply to XO - Files Restore on Tue, 29 Apr 2025 06:08:20 GMT]]>@olivierlambert
Just tried on XOA, I can't mount the NFS share on it. I get :

Test remote
Unsupported state or unable to authenticate data
]]>https://xcp-ng.org/forum/post/92430https://xcp-ng.org/forum/post/92430Tue, 29 Apr 2025 06:08:20 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 20:41:11 GMT]]>@olivierlambert

Ok, now it's clear!

@AtaxyaNetwork
Already installed :

lvm2 is already the newest version (2.03.16-2).
]]>https://xcp-ng.org/forum/post/92425https://xcp-ng.org/forum/post/92425Mon, 28 Apr 2025 20:41:11 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 20:34:14 GMT]]>@xcplak you can try to install the package lvm2 on your xo source

]]>https://xcp-ng.org/forum/post/92423https://xcp-ng.org/forum/post/92423Mon, 28 Apr 2025 20:34:14 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 20:31:52 GMT]]>That's 100% unrelated. If you do file level restore, you need to have an XO environment that's able to mount volumes from LVM. That's why I said you should test with XOA to validate your environment (just import a fresh one, connect to your backup repo and try to file restore).

]]>https://xcp-ng.org/forum/post/92421https://xcp-ng.org/forum/post/92421Mon, 28 Apr 2025 20:31:52 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 20:23:09 GMT]]>@olivierlambert
Don’t have XOA on hand.
Besides that, all backup jobs are already set up on XO…
Is this a common issue? Is there something I can do to fix it?
I was thinking of doing the backup on an unencrypted NFS share. Would that fix it?

]]>https://xcp-ng.org/forum/post/92420https://xcp-ng.org/forum/post/92420Mon, 28 Apr 2025 20:23:09 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 19:55:16 GMT]]>Does it work on XOA?

]]>https://xcp-ng.org/forum/post/92416https://xcp-ng.org/forum/post/92416Mon, 28 Apr 2025 19:55:16 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 17:16:25 GMT]]>@olivierlambert
My bad, it's on XO from the sources.
I didn’t take the filesystem into account.
After checking my backups, I can see files on EXT filesystems but not on LVM filesystems.

]]>https://xcp-ng.org/forum/post/92412https://xcp-ng.org/forum/post/92412Mon, 28 Apr 2025 17:16:25 GMT<![CDATA[Reply to XO - Files Restore on Mon, 28 Apr 2025 15:47:31 GMT]]>Hi,

This is highly coupled with the filesystem you use inside the VM (LVM, ext, xfs etc.)

Since you are not providing any hint on what you use (XOA or XO from the sources?), it's really hard to answer 🙂

]]>https://xcp-ng.org/forum/post/92403https://xcp-ng.org/forum/post/92403Mon, 28 Apr 2025 15:47:31 GMT