Backups Fail with ENOENT: no such file or directory
-
Not sure what's going on here. Brand new XO installed from Commit 06c3933, running against brand new fully patched XCP 8.3, targeting a TrueNAS Scale dataset exposed via NFS. Backup test is a Windows server VM with two VHDs attached, totally about 300gb. The error is always the same:
Error: read ETIMEDOUT
Error: ENOENT: no such file or directory, open '/mnt/tn-bk02/xo-vm-backups/e5f62657-8efb-3c2c-3397-2bfe01df5f0d/vdis/c82e4cd1-a0ee-4d7e-9e57-c77d8102593e/9110f4f4-e7d0-42bc-94be-b688b35b6a1c/20260501T213823Z.vhd'It takes somewhere between 3 minutes and 15 minutes for the backup to fail. While the backup is running, I see the vhd file in the file system appear, then disappear. A while later the backup fails. While the vhd file persists, it ranges in side from about 800mb to 2gb.
XO mounts the export just fine, the test always runs successfully. I have tried mounting the export directly in the OS (Ubuntu) and then directing XO to "local" but get the same result. I've tried forcing NFS to v3, it defaults to 4.2. No change.
From the OS, I can generate big files without issue, like:
dd if=/dev/urandom of=/mnt/tn-bk02/random.file bs=1G count=100 iflag=fullblock
When XO is creating the remote, the mount looks like:
172.16.22.9:/mnt/backup-pool-01/xcp-backups /run/xo-server/mounts/0e60da8f-d41d-4d9a-8f8c-6c0e1ca03a89 nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.22.26,local_lock=none,addr=172.16.22.9 0 0
which seems fine to me.
I have tried a few adjustments on the TrueNAS side, including adding and removing permissions and forcing NFS versions.
I think if it I could explain why the vhd file appears in the file system, but then disappears, I'd be closer to understanding what's happening....
If anyone has any thoughts, I'd love to hear them. FWIW, I have three deployments very similar to this elsewhere that work perfectly. The only difference between this install the other three is that this one is on the latest version of XO, and the other installs are considerably older.
-
I had a similar issue and mine was resolved by changing the quota size, apparantly my disk was getting bigger.
Also the ETIMEDOUT could show that your backup job runs into a timeout somewhere - maybe the backup job timeout?
-
Had major issues with backups a while ago, it was due to the firmware of the NIC in the physical server. We found out by moving our XO vm to another physical host and then error went away, moving it back made it come back.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login