@momofhlp I suspect it is a UID/GID mapping or permissions problem.
Mount the share manually no-root-squash. run a backup on that share. see what UID/GID xo expects you to use. Then set up an nfsshare for only that matching UID/GID and all should be good.
it is easiest to set it up wrong security wise with NFSv3 but easiest to make it work. Once it works, go back and switch to nfs4.1. read and weep. https://www.truenas.com/community/threads/nfs-sys-security-option.86501/

old nfs (v3) has little or no security. nfs4,1 has SYS for slightly better security. NFS4.2 + kerb5 has best security but a learning curve like:
|
|
|
|

only much much steeper. a better nfs security solution is nfs via stunnel
see:

• https://www.linuxjournal.com/content/encrypting-nfsv4-stunnel-tls
• https://bobcares.com/blog/nfsv4-encryption-with-stunnel-tls/
• historical: https://www.stunnel.org/mailman3/hyperkitty/list/stunnel-users@stunnel.org/thread/7FVHLCPSDHL2S6KQ7DBN24HLPSX4SHWV/
• other implementation details https://github.com/chadgeary/tls_nfs/tree/master/playbook
• another imp. https://forge.puppet.com/modules/simp/nfs/readme

I should turn this info into a package for xcp-ng.

@s-master I was exporting some VMS from esxi via XO a few months ago and exporting machines with one or two drives seems to work okay. Failing that, can your esxi system export to OVA?if not, that, your clonezilla approach will probably work. Sometimes you just have to bite the bullet and start over again from scratch. It sucks, it's horrible but it is better than a half-assed copy that flakes out at you at random times.

@olivierlambert that does look interesting. I really need to find a way to make it work over scp/sftp unless you have an in with rsync.net and can get them to add an S3-compatible service

It's not really practical to make a move to an S3 provider at this time; we are 230 TB into rsync.net for a damn good price with no extra charges for data ingress or egress. If I add VM backup, the additional 10+ terabytes are a drop in the bucket of the overall usage.

@florent, sorry for the long delay in getting back to you. I'm still working on getting Christmas cards out from 2017.

two ideas:

If I can get an assessment of how reliable the rclone mount capability is, I should be able to run a second incremental backup on that mount point. The big problem there is detecting when the rclone mount has failed and halting/restarting the backup once the connection is restored. I have some ideas on how to do this and I want to chew on them a bit for writing them down.

restoration would then become easy because it's just another "network share"

If I am restricted to rsync then it probably would be safer to use full backup and move those upstream. I'm assuming that the full backup file names are the same from creation until expiration. If that's the case, then it'll be much easier to push up because rsync will only push what it does not have at the destination.

Thinking about it, it looks like I will have a lot of storage dedicated to backups, and I might be forced into having just one copy of full disk image backups that I use for the local copy and also replicate it upstream.

thanks for the pointers to the documentation I need.

@momofhlp I suspect it is a UID/GID mapping or permissions problem.
Mount the share manually no-root-squash. run a backup on that share. see what UID/GID xo expects you to use. Then set up an nfsshare for only that matching UID/GID and all should be good.
it is easiest to set it up wrong security wise with NFSv3 but easiest to make it work. Once it works, go back and switch to nfs4.1. read and weep. https://www.truenas.com/community/threads/nfs-sys-security-option.86501/

old nfs (v3) has little or no security. nfs4,1 has SYS for slightly better security. NFS4.2 + kerb5 has best security but a learning curve like:
|
|
|
|

only much much steeper. a better nfs security solution is nfs via stunnel
see:

• https://www.linuxjournal.com/content/encrypting-nfsv4-stunnel-tls
• https://bobcares.com/blog/nfsv4-encryption-with-stunnel-tls/
• historical: https://www.stunnel.org/mailman3/hyperkitty/list/stunnel-users@stunnel.org/thread/7FVHLCPSDHL2S6KQ7DBN24HLPSX4SHWV/
• other implementation details https://github.com/chadgeary/tls_nfs/tree/master/playbook
• another imp. https://forge.puppet.com/modules/simp/nfs/readme

I should turn this info into a package for xcp-ng.

@s-master I was exporting some VMS from esxi via XO a few months ago and exporting machines with one or two drives seems to work okay. Failing that, can your esxi system export to OVA?if not, that, your clonezilla approach will probably work. Sometimes you just have to bite the bullet and start over again from scratch. It sucks, it's horrible but it is better than a half-assed copy that flakes out at you at random times.

I am in a bit of a quandary. I need to store backups on rsync.net I have a couple of clients committed to that as a backup repository because they were using it for their off-site backups and other systems previously.

When I raised this issue before, it was suggested I use SSHFS for a borg backup target, which has several problems. It's not guaranteed reliable; the mount point can vanish, and you won't know it. It slows down backup significantly because of the de-duplication and archive validation process.

The structure of SRs is pretty opaque. Looking at the current backup remote file system, It looks like, under xo-vm-backups, a series of directories look like UUIDs. I assume each UUID refers to a specific VM. Under the UUID directories, there are snapshots of each VM (xva), a hash for validating xva file, and a JSON file with parameters for the VM. I suspect that if you have a multi-day set of full backups, there will be multiple trios of files under the same UUID. Looks like I found a case where the backup images are cleaned up properly

I'm assuming (and this may be a bad bet) that the backup data set starting with the UUID is a logically self-contained definition of a preserved VM or VMs. If I place the backup data set. starting with the UUID in an unused remote target, it should be visible via the XO backup restore interface

Would placing a copy of a Borg archive in an unused backup remote target create a restorable image? Assuming my technique works, the next question is how to know that today's backup is done so I can start the remote copy.

Thanks for your help!

@olivierlambert thanks. wasn't sure if that version will work with 22.04.

Started up an Ubuntu 22.04 server VM and discovered that xe-guest-utilities was not part of the 22.04 repository. What package should I use for guest code

@stormi that's a good thing to look at and I am paying attention to it. Things I've learned since my original post are leading me to suspect a bet update is has caused the VM to crash.

@stormi this is the recovery boot I was speaking about.

Clicking the recovery start and I get:

Am I misunderstanding something about the functionality provided by the recovery start button?

I tried to boot off a rescue image earlier but there was some problem with the DVD virtual device attached to the VM. A couple of reboots later it seems to not be a problem anymore and I was able to successfully boot onto PartitionMagic. Now I can look into the logs in the VM and see if they show any hints as to why it's failing.

I believe I'm having the same problem. A VM failed to reboot and ended up in busy box recovery state. I also tried the recovery start and it failed. First it claimed that the DVD drive was not connected and I needed to start up and shut down the VM. I did so, the DVD drive is connected but with nothing in the drive, I get

With the guest tools DVD in the drive I get:

Am I missing some sort of disk image the rescue mode needs?