Hi All,

I have my VM storage centralized on a dedicated NAS using NFS. However, this is not a datacenter environment and whenever there is a power outage, the VM's dont shutdown properly and end up having filesystem corruption that blocks them from booting normally.

Running fsck generally fixes the issue and allows them to boot. But I would like to find a way to use remote network storage for VM's without this issue.

Any guidance is appreciated.

I have a rack of servers that are all recycled hardware running XCP-NG managed with Xen Orchestra. CPU's often don't match between hosts, so managing them as one pool is a "no-no" based on documentation.

The network is a mess partly because each host has been managed as a separate pool with its own templates and network configurations.

I want to be able to configure templates and networks across them in a common place. Each pool has diverged over time and I have managed them poorly, which has led to annoyances and mistakes.

Any suggestions on how I could generate and store templates in a common place (I like being able to create templates with cloud-init) and be able to trunk VLAN's and networks across all hosts without having to configure them independently.

We use Minio to store standard backups that use it as an S3 target.

It works fine running on xo-web 5.92.0 and xo-server 5.87.0 from commit https://github.com/vatesfr/xen-orchestra/commit/6e6886a6bac6d8672c775991eebb00c991e54cfc

But if I upgrade to the latest from https://github.com/vatesfr/xen-orchestra/commit/a5e9f051a2f80358f2decf67e8cba64f2dbcb937 causes a backups to terminate shortly after starting with a very unhelpful error:

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1654646632287",
  "jobId": "a3eaece2-b377-4dba-b7f2-bd9b4bc3ca7c",
  "jobName": "Backups for Resources Tagged with \"Backups-Daily-7d-3AM\"",
  "message": "backup",
  "scheduleId": "dc5f892c-8427-43d3-9eeb-9d1c18bc8f47",
  "start": 1654646632287,
  "status": "failure",
  "end": 1654646634770,
  "result": {}
}

I rolled back to our original build and its back to working.

I have run into this on another upgrade attempt in early May (I have a timestamped backup of that XOA build marked with that date indicating I rolled back then).

Let me know if there is anyother log I can try to fetch for this.

0 Rajaa-BARHTAOUI committed to vatesfr/xen-orchestra

feat(xo-web/new/sr): creating local SR requires confirmation (#6121)

0 julien-f committed to vatesfr/xen-orchestra

docs(REST API): `content-type` is no longer necessary with `-T`

Because it is no longer set by default to `application/x-www-form-urlencoded` like it was with `--data-binary`.

@rkelley Thank you for doing all the work for us.

When there is a proper patch, I be happy to test it in our environment as well.

@olivierlambert I have swapped to normal full backups for now.

I have non-emergency need to recover at least one VM and have a couple options on how to do so since the data appears intact.

In all fairness to xcp-ng, the hardware we use is a mix of older machines (it is not a funded enterprise setup). I would be willing to provide your team specific hardware details among other things but I will be slow to gather info.

I will try out a delta backup of a bios VM as soon as I find the time. My only theory, which I have little evidence or ability to test, is that I have triggered a bug in the uefi firmware used by the VM. I can at least narrow down a bit more by comparing uefi versus legacy.

@olivierlambert so I booted into Finnix, and was able to mount the root LVM and verify that the filesystem is intact and the files I expect to see are all there. The chroot environment is functional and I don't see anything worry about.

I was able to mount and rebuild the /boot through efi/EFI paths, I have not yet bothered checking file hashes or anything forensic, but I am seeing all the contents in the backups that I see in the running VM from which this backup originated.

One thing I have found, is that only deltas are failing to restore in bootable form. Full backups are fine. I tried to restore the periodic full snapshots within the delta cycles, but those aren't any better than restoring a later delta.

I will be switching to full backups only until I know why delta is broken in my environment.

@olivierlambert

I got a VM to boot into an Ubuntu live CD and could see the disk and its respective partitions, and LVM setup.

I posted here originally

I have both Delta backups and full backups failing to restore properly from an NFS source. I have them throwing into a UEFI rescue mode. I have Debian Buster images doing this and Centos 8 with xo-server 5.81.2 and XCP-NG 8.2.0.

My original VM's are using Secure Boot with UEFI, I have learned that secure boot isnt supported by XCP-NG, and I have not tested non-UEFI VM's and their backups

I do not know what I am missing, nothing about the restore fails, they just will not boot.

@stormi

Well that's news to me.

I just tested disabling secure boot on a VM and rebooting it. It made no difference, which is what I'd expect if it was unsupported.

I guess we can rule it out as a factor in why our backups don't work.

@ninghe I may be having a similar issue.

I have both Delta backups and full backups failing to restore properly from an NFS source. I have them throwing into a UEFI rescue mode. I have Debian Buster images doing this and Centos 8 with xo-server 5.81.2 and XCP-NG 8.2.0.

My original VM's are using Secure Boot with UEFI, I have not yet tested non-Secure Boot or non-UEFI VM's and their backups.

I do not know what I am missing, nothing about the restore fails, they just will not boot.