XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO - Files Restore

    Scheduled Pinned Locked Moved Backup
    24 Posts 7 Posters 1.0k Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • X Offline
      xcplak @olivierlambert
      last edited by

      @olivierlambert
      My bad, it's on XO from the sources.
      I didn’t take the filesystem into account.
      After checking my backups, I can see files on EXT filesystems but not on LVM filesystems.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Does it work on XOA?

        X 1 Reply Last reply Reply Quote 0
        • X Offline
          xcplak @olivierlambert
          last edited by

          @olivierlambert
          Don’t have XOA on hand.
          Besides that, all backup jobs are already set up on XO…
          Is this a common issue? Is there something I can do to fix it?
          I was thinking of doing the backup on an unencrypted NFS share. Would that fix it?

          AtaxyaNetworkA 1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            That's 100% unrelated. If you do file level restore, you need to have an XO environment that's able to mount volumes from LVM. That's why I said you should test with XOA to validate your environment (just import a fresh one, connect to your backup repo and try to file restore).

            X 2 Replies Last reply Reply Quote 0
            • AtaxyaNetworkA Offline
              AtaxyaNetwork Ambassador @xcplak
              last edited by

              @xcplak you can try to install the package lvm2 on your xo source

              1 Reply Last reply Reply Quote 0
              • X Offline
                xcplak @olivierlambert
                last edited by

                @olivierlambert

                Ok, now it's clear!

                @AtaxyaNetwork
                Already installed :

                lvm2 is already the newest version (2.03.16-2).
                
                1 Reply Last reply Reply Quote 0
                • X Offline
                  xcplak @olivierlambert
                  last edited by

                  @olivierlambert
                  Just tried on XOA, I can't mount the NFS share on it. I get :

                  Test remote
                  Unsupported state or unable to authenticate data
                  
                  X 1 Reply Last reply Reply Quote 0
                  • X Offline
                    xcplak @xcplak
                    last edited by

                    I just deployed a new NFS share for testing.
                    The share is mounted on both XO and the newly deployed XOA.
                    I backed up a VM that I can see the files in using XO.
                    I faced the same issue with both XOA and XO, I can’t see the files.
                    I backed up the same VM using XOA and tried to restore the files, same issue.

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      You have a remote (backup repo) configuration issue. That's unrelated, check your permissions.

                      X 1 Reply Last reply Reply Quote 0
                      • X Offline
                        xcplak @olivierlambert
                        last edited by

                        @olivierlambert,

                        Why would this issue be related to my NFS share? I can access the share without any problems, read and write operations work fine. Other NFS shares with multiple applications running are also functioning normally.

                        I checked the XO logs and found the following:

                        xo-server[679]: 2025-04-29T21:45:58.276Z xo:api WARN  | backupNg.listFiles(...) [114ms] =!> Error: Command failed: vgchange -ay vg_iredmail
                        xo-server[679]: File descriptor 22 (/var/lib/xo-server/data/leveldb/LOG) leaked on vgchange invocation. Parent PID 679: node
                        xo-server[679]: File descriptor 24 (/var/lib/xo-server/data/leveldb/LOCK) leaked on vgchange invocation. Parent PID 679: node
                        xo-server[679]: File descriptor 25 (/dev/fuse) leaked on vgchange invocation. Parent PID 679: node
                        xo-server[679]: File descriptor 26 (/var/lib/xo-server/data/leveldb/MANIFEST-000103) leaked on vgchange invocation. Parent PID 679: node
                        xo-server[679]: File descriptor 31 (/dev/fuse) leaked on vgchange invocation. Parent PID 679: node
                        xo-server[679]: File descriptor 34 (/var/lib/xo-server/data/leveldb/000356.log) leaked on vgchange invocation. Parent PID 679: node
                        xo-server[679]:   WARNING: Couldn't find device with uuid oWTE8A-LNny-Ifsc-Jmtp-lTkB-dSjt-2AqhwY.
                        xo-server[679]:   WARNING: VG vg_iredmail is missing PV oWTE8A-LNny-Ifsc-Jmtp-lTkB-dSjt-2AqhwY (last written to /dev/xvda4).
                        xo-server[679]:   Refusing activation of partial LV vg_iredmail/lv_root.  Use '--activationmode partial' to override.
                        

                        To me, this looks more like an issue with file restore than the NFS itself.

                        It seems similar to what's reported here:
                        https://github.com/vatesfr/xen-orchestra/issues/7029

                        andrew64k created this issue in vatesfr/xen-orchestra

                        open File level restoration not working on LVM partition #7029

                        P 1 Reply Last reply Reply Quote 0
                        • P Offline
                          peo @xcplak
                          last edited by

                          @xcplak for what it's worth, I confirm that file restore from LVM partitions do not work using XOA (I activated my trial just to test this):
                          821f8ea3-54a5-4d61-a860-ea68b9229742-image.png

                          P 1 Reply Last reply Reply Quote 0
                          • P Offline
                            peo @peo
                            last edited by

                            From the log on the XOA-VM (/var/lib/xo-server/data/leveldb/001205.log)

                            {"method":"backupNg.listFiles","params":{"remote":"3f679c65-1290-4264-8ede-d91d08b2abf0","disk":"/xo-vm-backups/f94129f5-5c2c-6d6b-51e7-eeed24bf965b/vdis/6f239189-6971-4d3a-8e6a-5d43bd8bdce3/e163632c-9dca-4947-9c44-b2b794008294/20250522T184952Z.alias.vhd","path":"/","partition":"23a71973-6022-44f5-b51a-6d48b93c6cfd"},"name":"API call: backupNg.listFiles","userId":"3ffcc4df-8f09-41ce-99dc-8a1f1c568c56","type":"api.call"},"start":1748214110665,"status":"failure","updatedAt":1748214110991,"end":1748214110991,"result":{"code":32,"killed":false,"signal":null,"cmd":"mount --options=loop,ro,norecovery,sizelimit=18594398208,offset=2879389696 --source=/tmp/axvvrbout2/vhd0 --target=/tmp/4rksxkr68si","message":"Command failed: mount --options=loop,ro,norecovery,sizelimit=18594398208,offset=2879389696 --source=/tmp/axvvrbout2/vhd0 --target=/tmp/4rksxkr68si\nmount: /tmp/4rksxkr68si: unknown filesystem type 'LVM2_member'.\n       dmesg(1) may have more information after failed mount system call.\n","name":"Error","stack":"Error: Command failed: mount --options=loop,ro,norecovery,sizelimit=18594398208,offset=2879389696 --source=/tmp/axvvrbout2/vhd0 --target=/tmp/4rksxkr68si\nmount: /tmp/4rksxkr68si: unknown filesystem type 'LVM2_member'.\n       dmesg(1) may have more information after failed mount system call.\n\n    at genericNodeError (node:internal/errors:984:15)\n    at wrappedFn (node:internal/errors:538:14)\n    at ChildProcess.exithandler (node:child_process:422:12)\n    at ChildProcess.emit (node:events:518:28)\n    at ChildProcess.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)\n    at maybeClose (node:internal/child_process:1104:16)\n    at Socket.<anonymous> (node:internal/child_process:456:11)\n    at Socket.emit (node:events:518:28)\n    at Socket.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)\n    at Pipe.<anonymous> (node:net:343:12)\n    at Pipe.callbackTrampoline (node:internal/async_hooks:130:17)"}}j▒▒5▒▒▒!tasks!0mb49jl91%▒▒▒▒▒!tasks!0mb49jl97▒{
                            
                            
                            A 1 Reply Last reply Reply Quote 0
                            • A Offline
                              Andrew Top contributor @peo
                              last edited by

                              @peo Still an existing problem:
                              https://github.com/vatesfr/xen-orchestra/issues/7029

                              andrew64k created this issue in vatesfr/xen-orchestra

                              open File level restoration not working on LVM partition #7029

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Pinging @lsouai-vates

                                lsouai-vatesL 1 Reply Last reply Reply Quote 0
                                • lsouai-vatesL Offline
                                  lsouai-vates Vates 🪐 XO Team @olivierlambert
                                  last edited by

                                  @olivierlambert better ask @florent

                                  X 1 Reply Last reply Reply Quote 0
                                  • X Offline
                                    xcplak @lsouai-vates
                                    last edited by

                                    @peo Thanks for the info!
                                    I also tested it with XOA and got the same result.
                                    However, when deploying the same VM without LVM, file restore worked perfectly with both XO and XOA.
                                    It definitely seems related to the GitHub issue.

                                    In the meantime, I've switched to a commercial backup solution until it's resolved.

                                    A 1 Reply Last reply Reply Quote 0
                                    • A Offline
                                      Andrew Top contributor @xcplak
                                      last edited by

                                      @xcplak With a LVM guest, you can restore the whole VM now. The backup data is actually saved correctly and it is possible to manually restore a single file. So they can fix XO and the old backup data is still good.

                                      X 1 Reply Last reply Reply Quote 0
                                      • X Offline
                                        xcplak @Andrew
                                        last edited by

                                        @Andrew
                                        I'm not sure I understand what you meant.
                                        Full VM restore is working, the issue is with file-level restore. So restoring a full 200GB VM just to retrieve 2 or 3 files isn't really a viable solution.

                                        lsouai-vatesL 1 Reply Last reply Reply Quote 0
                                        • P Offline
                                          peo
                                          last edited by

                                          Great that this bug/problem is being confirmed by others. Reproducing is as simple as create a new Linux VM (only using defaults when installing Linux), back it up, then try to restore files.
                                          Restoring single files is a feature that is at least needed in production environments (anything outside the "home lab"). Personally, I have no problem with waiting an hour or two for a full restore to a temporary VM to be able to access a file deleted or modified by mistake.
                                          There are people willing to help find and pinpoint problems like this, but having to use XOA to get attention to problems, given that it requires a license beyond the free trial month makes it less appealing for us spend our free time to help with this.

                                          1 Reply Last reply Reply Quote 0
                                          • J Offline
                                            jshiells
                                            last edited by

                                            I can confirm this as well

                                            several new ubuntu 24 vm's with LVM having this problem. I thought i was going crazy

                                            existing debian 10,11,12 VM's using LVM are not effected
                                            vm's not using LVM are not effected.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post