We use Minio to store standard backups that use it as an S3 target.

It works fine running on xo-web 5.92.0 and xo-server 5.87.0 from commit https://github.com/vatesfr/xen-orchestra/commit/6e6886a6bac6d8672c775991eebb00c991e54cfc

But if I upgrade to the latest from https://github.com/vatesfr/xen-orchestra/commit/a5e9f051a2f80358f2decf67e8cba64f2dbcb937 causes a backups to terminate shortly after starting with a very unhelpful error:

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1654646632287",
  "jobId": "a3eaece2-b377-4dba-b7f2-bd9b4bc3ca7c",
  "jobName": "Backups for Resources Tagged with \"Backups-Daily-7d-3AM\"",
  "message": "backup",
  "scheduleId": "dc5f892c-8427-43d3-9eeb-9d1c18bc8f47",
  "start": 1654646632287,
  "status": "failure",
  "end": 1654646634770,
  "result": {}
}

I rolled back to our original build and its back to working.

I have run into this on another upgrade attempt in early May (I have a timestamped backup of that XOA build marked with that date indicating I rolled back then).

Let me know if there is anyother log I can try to fetch for this.

0 Rajaa-BARHTAOUI committed to vatesfr/xen-orchestra

feat(xo-web/new/sr): creating local SR requires confirmation (#6121)

0 julien-f committed to vatesfr/xen-orchestra

docs(REST API): `content-type` is no longer necessary with `-T`

Because it is no longer set by default to `application/x-www-form-urlencoded` like it was with `--data-binary`.

@rkelley Thank you for doing all the work for us.

When there is a proper patch, I be happy to test it in our environment as well.

@olivierlambert I have swapped to normal full backups for now.

I have non-emergency need to recover at least one VM and have a couple options on how to do so since the data appears intact.

In all fairness to xcp-ng, the hardware we use is a mix of older machines (it is not a funded enterprise setup). I would be willing to provide your team specific hardware details among other things but I will be slow to gather info.

I will try out a delta backup of a bios VM as soon as I find the time. My only theory, which I have little evidence or ability to test, is that I have triggered a bug in the uefi firmware used by the VM. I can at least narrow down a bit more by comparing uefi versus legacy.

@olivierlambert so I booted into Finnix, and was able to mount the root LVM and verify that the filesystem is intact and the files I expect to see are all there. The chroot environment is functional and I don't see anything worry about.

I was able to mount and rebuild the /boot through efi/EFI paths, I have not yet bothered checking file hashes or anything forensic, but I am seeing all the contents in the backups that I see in the running VM from which this backup originated.

One thing I have found, is that only deltas are failing to restore in bootable form. Full backups are fine. I tried to restore the periodic full snapshots within the delta cycles, but those aren't any better than restoring a later delta.

I will be switching to full backups only until I know why delta is broken in my environment.

@olivierlambert

I got a VM to boot into an Ubuntu live CD and could see the disk and its respective partitions, and LVM setup.

I posted here originally

I have both Delta backups and full backups failing to restore properly from an NFS source. I have them throwing into a UEFI rescue mode. I have Debian Buster images doing this and Centos 8 with xo-server 5.81.2 and XCP-NG 8.2.0.

My original VM's are using Secure Boot with UEFI, I have learned that secure boot isnt supported by XCP-NG, and I have not tested non-UEFI VM's and their backups

I do not know what I am missing, nothing about the restore fails, they just will not boot.

@stormi

Well that's news to me.

I just tested disabling secure boot on a VM and rebooting it. It made no difference, which is what I'd expect if it was unsupported.

I guess we can rule it out as a factor in why our backups don't work.

@ninghe I may be having a similar issue.

I have both Delta backups and full backups failing to restore properly from an NFS source. I have them throwing into a UEFI rescue mode. I have Debian Buster images doing this and Centos 8 with xo-server 5.81.2 and XCP-NG 8.2.0.

My original VM's are using Secure Boot with UEFI, I have not yet tested non-Secure Boot or non-UEFI VM's and their backups.

I do not know what I am missing, nothing about the restore fails, they just will not boot.