@zxcp what about the
IOMMU Disabled message that you posted above?

@indyj The problem appears to be in the 5.15 kernel as well but the workaround seems to fix the issue.

@indyj The file restore issue was due to installing XO on FreeBSD (fine for VM restores, not file extraction).

I then installed XO on Ubuntu 20, and this worked for restoring files from Linux VMs but not Windows VMs.
Then I installed on CentOS 8 and had the same problem as Ubuntu.

I then installed on Fedora 34 and everthing works fine.

@indyj So I went to the logs and pasted the command directly into the shell. while the vhd file does exist the mount point does not.

The command assumes a temporary directory within /tmp that is not getting created by the script.

If I create the directory and paste the command into the shell the vhd does mount.

Thanks for the tips.
When I try to restore an individual file from the server using ZFS compression & dedupe XOA gives me an error: "Error while scanning disk"

I'm wondering if the compression or dedupe is the likely cause. My money is on the Deduplication. Any thoughts on this?

Not being a ZFS expert here is an oddity: I have two old servers I use for backups. One is NFS with XFS filesystem (testing XO backups). The second is NFS with ZFS.

I copied my data from the server with XFS to the one using ZFS via rsync: A 17 Terabyte NFS/XFS share used only approximately 7 Terabytes on the ZFS server.

However If I backup data via XO to the NFS share on the ZFS system I don't seem to save space compared to backing up to NFS on the XFS filesystem.

I'm using lz4 compression & Dedupe on the ZFS system..

I'm testing XO. I have an NFS server with ZFS (compression + deduplication enabled).

Would backing up to this nfs share cause problems for backups & restores?

@indyj said in Xen VMs & elrepo kernel:

@danp said in Xen VMs & elrepo kernel:

@indyj May this is your issue? https://elrepo.org/bugs/view.php?id=1137&nbn=1

For those who install the kernel-ml from Elrepo: I tested this on all my CentOS 7/8 VMs: The results were mixed, the procedure worked on about half of them. Proceed with Caution.

Update:
This same issue has found its way into Fedora 34 in the stock kernel. No special repos, just "dnf update" and it happened. Could this affect the entire 5.14 series?

@danp said in Xen VMs & elrepo kernel:

@indyj May this is your issue? https://elrepo.org/bugs/view.php?id=1137&nbn=1

For those who install the kernel-ml from Elrepo: I tested this on all my CentOS 7/8 VMs: The results were mixed, the procedure worked on about half of them. Proceed with Caution.

@danp That is the very problem I'm having. Thanks for looking that up!

I have been unable to get elrepo kernel 5.14 installed and booting on my CentOS 7/8 XCP VMs.
I can however get the Kernel installed on other platforms just not Xen (xcp) VMs.
Is anyone having this issue or heard anything about it?

@indyj FYI when I ran this test again it worked.
this time:

• I ran it as a "Delta" backup.
• And then rescanned the removable media.
• I was then able to restore an individual file from the backup on USB media.

I tested backup directly to USB drives by attaching to the XO VM. Backup and restore works fine as long as I restore the whole VM from USB.

If I try to drill down and restore an individual file from the backup on USB I cannot.

Is there a way to do this?

@indyj I tested mounting the bucket via s3fs and using rsync to copy some backups from my server to Wasabi.

The transfer rates were ok early in the sync process but eventually the speed dropped off into the 20 to 40 KBs speed. I'm guessing this is where XO would give up.

@olivierlambert I have a 100x100 over fiber. I was getting between 4 & 8 Megabits.

Wasabi was fine for small VMs, but not really feasible for my whole environment (40 VMs some as large as 1-terabyte).

I did notice that backing up larger VMs via XO to Wasabi (100 GB or more) would time out and fail between 60% & 80% complete. Small VMs were not a problem.

I was able to get Wasabi working with Xen Orchestra. In my case referring to some documentation got me through the configuration.

https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions-

XO's support for Amazon was sufficient.

The proper AWS endpoint was crucial. As is your access key and secret key.

create a bucket, then create a folder under the bucket. Give XO the bucket name and the folder name.

The caveat is my backup speed is slow.

I am also interested in Wasabi. Let me know if it works.

@jefftee I prefer Alpine Linux.

@olivierlambert Thanks. I notice the naming convention on the remote storage seems to align with the uuid of the VM. In the DR scenario does the directory structure on the remote have to get recreated to restore a VM from the VHD file?