XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. jvanabra
    J
    Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 1
    • Posts 2
    • Groups 0

    jvanabra

    @jvanabra

    1
    Reputation
    1
    Profile views
    2
    Posts
    0
    Followers
    0
    Following
    Joined
    Last Online

    jvanabra Unfollow Follow

    Latest posts made by jvanabra

    • RE: Backups Fail with ENOENT: no such file or directory

      @pierrebrunet I did try the "merge synchronously" option, but that didn't change anything.

      Can you expand on "your chains have an issue?" I understand the concept of chains, but in this particular case the situation is:

      1. Fresh install of XCP
      2. Brand new VM that has never been backed up before (though was built originally on another pool). There was no previously existing snapshot.
      3. New install of XO targeting a new NFS server with a ne backup job

      There shouldn't be any chain involved as literally everything is brand new.

      I cleaned up:

      1. Removed the snapshots from the VMs
      2. Unmounted the NFS
      3. Cleaned up the filesystem on NFS
      4. Remounted NFS
      5. Created a new backup job
      6. Left NFS with v4.2, enabled synchronous merge
      7. Restarted the job
      8. Same failure

      Some thing I noticed: When kicking off the job, the VHD appears in the NFS filesystem:

      /var/run/xo-server/mounts/d806298f-85cf-4565-bb7e-9fad479b941f/xo-vm-backups/e5f62657-8efb-3c2c-3397-2bfe01df5f0d/vdis/c82e4cd1-a0ee-4d7e-9e57-c77d8102593e# ls -alh 82c9959a-b4fa-46a1-a6ea-8cfbbeca864f/.1777916466905.20260504T174106Z.vhd

      In this example, this vhd file was 951mb.

      It remains there for about one minute, then disappeared. Shortly later the backup job fails, but I notice the snapshot remains, owned by control domain. That never seems to disappear.

      I think it's worth noting that I have imported several test VMs, as well as copied VMs cross-pool (from somewhere else to this pool) so it seems like the normal import/export mechanisms work properly.

      If you let me know what logs you'd like, I'm happy to upload them.

      posted in Backup
      J
      jvanabra
    • Backups Fail with ENOENT: no such file or directory

      Not sure what's going on here. Brand new XO installed from Commit 06c3933, running against brand new fully patched XCP 8.3, targeting a TrueNAS Scale dataset exposed via NFS. Backup test is a Windows server VM with two VHDs attached, totally about 300gb. The error is always the same:

      Error: read ETIMEDOUT
      Error: ENOENT: no such file or directory, open '/mnt/tn-bk02/xo-vm-backups/e5f62657-8efb-3c2c-3397-2bfe01df5f0d/vdis/c82e4cd1-a0ee-4d7e-9e57-c77d8102593e/9110f4f4-e7d0-42bc-94be-b688b35b6a1c/20260501T213823Z.vhd'

      It takes somewhere between 3 minutes and 15 minutes for the backup to fail. While the backup is running, I see the vhd file in the file system appear, then disappear. A while later the backup fails. While the vhd file persists, it ranges in side from about 800mb to 2gb.

      XO mounts the export just fine, the test always runs successfully. I have tried mounting the export directly in the OS (Ubuntu) and then directing XO to "local" but get the same result. I've tried forcing NFS to v3, it defaults to 4.2. No change.

      From the OS, I can generate big files without issue, like:

      dd if=/dev/urandom of=/mnt/tn-bk02/random.file bs=1G count=100 iflag=fullblock

      When XO is creating the remote, the mount looks like:

      172.16.22.9:/mnt/backup-pool-01/xcp-backups /run/xo-server/mounts/0e60da8f-d41d-4d9a-8f8c-6c0e1ca03a89 nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.22.26,local_lock=none,addr=172.16.22.9 0 0

      which seems fine to me.

      I have tried a few adjustments on the TrueNAS side, including adding and removing permissions and forcing NFS versions.

      I think if it I could explain why the vhd file appears in the file system, but then disappears, I'd be closer to understanding what's happening....

      If anyone has any thoughts, I'd love to hear them. FWIW, I have three deployments very similar to this elsewhere that work perfectly. The only difference between this install the other three is that this one is on the latest version of XO, and the other installs are considerably older.

      posted in Backup
      J
      jvanabra