For those finding this post and are using ronivay's XenOrchestraInstallerUpdater, the solution is as follows...

Edit xo-install.cfg with your favourite editor.

Look for the line...

BRANCH="master"

Comment it out and and on a new line add...

#BRANCH="master"
BRANCH="0ccfd4b"

If you then run the script and choose 2) Update, it should work.

Because I was in a hurry, I deleted the /opt/ox directory, put the SSL certs that were in it back, and then ran 1) Install so that I knew that I was certain that there were no conflicts with compiling and older version.

But either method should work.

@rtjdamen Sorry! Just to clarify. When I said a mixture of backup solutions, I was talking about the different backup solutions that XCP-NG backup provides. IE Running a delta backup after running a continuous replication backup.

When running a mixture of XCP-NG incremental backups, there appears to be a bug somewhere that has been causing the fall back to base errors along with a couple of other errors that break the backup process.

@michmoor0725 Sure!

I don't think that it is a ronivay issue as it appears to be occurring with any self-signed certificate. Regardless of what was used to compile it.

That said, if some people check there instead of here, it mays good sense to have something posted there as well.

First of all, sorry for hijacking this old topic but as it was so well documented, it made more sense to expand upon it then to try and recreate everything that was already here.

Second, I generally don't use XCP-ng Center. However, once coming across this post I did open it to verify that what I saw in this post was the same as what my XCP-ng Center was showing.

Third, I am on one of the latest community additions but I have seen this occur a couple of time on previous versions as well.

Okay with that out of the way, I now believe that I know what is causing this to happen...

When you have a VM with multiple VDI's in it and one of those VDI's is both large and fairly busy, if the following exists then XO will not show one or more of the smaller drives despite the hosts still functioning normally while working on that VM:

• The virtual machine has multiple snapshots of all of the VDI's in it. (IE like after several backups)
• For some reason a bunch of the snapshots are consolidated at one.
• All the snapshots that were consolidated are removed from the VM and the snapshots of the VDI's have coalesced but the system is still working on coalescing the large VDI.
• If a backup process tries to work on this VM, it will create a snapshot that XO shows to stay connected to the smaller drives even though the backup is skipped because a new snapshot of the large VDI can't be created due to the older snapshots not being coalesced yet.

Once the large VDI has properly coalesced all it's snapshots, the hosts will continue to treat it as any other VM but that VDI never does show up in XO again.

The only solution that I have found to fix it is to turn off the VM and then create a copy of missing VDI or VDI's and use those copies to replace the missing VDI's.

I know that I am just a member of the community but I hope that this helps everyone to understand what is going when this happens to them.

It happened to me a couple of times before I was able to peace together the whole puzzle.

@olivierlambert @florent If it helps to know... Disabling "Use NBD + CBT to transfer disk if available" on the same backup jobs as I used above works flawlessly. Although on big backup jobs, not having NBD available does add about a quarter of the time to the backup process.

@rtjdamen Sorry! Just to clarify. When I said a mixture of backup solutions, I was talking about the different backup solutions that XCP-NG backup provides. IE Running a delta backup after running a continuous replication backup.

When running a mixture of XCP-NG incremental backups, there appears to be a bug somewhere that has been causing the fall back to base errors along with a couple of other errors that break the backup process.

@rtjdamen Seems to happen when you run a mixture of backup solutions.

To follow up on the backups that break, I have been able to create a reproducible scenario for causing backups to fail with the error message... "can't create a stream from a metadata VDI, fall back to a base"

Hardware:

• 4 XCP-NG hosts (Error is reproducible regardless of which host is hosting the VM.)
• 1 TrueNAS file server providing the NFS share the XCP-NG hosts use for running the VMs.
• 1 TrueNAS file server providing the NFS share for backing up VMs in Continuous Replication mode.
• 1 TrueNAS file server providing the NFS share for backing up VMs in Delta Backups.

Backup Order:

• Full backup in Continuous Replication mode
• Full backup in Delta Backup mode
• Multiple incremental backups in Continuous Replication mode (Not sure of the exact minimum but my servers ran the backup 7 times yesterday.)
• At this point, running an incremental backup of a Delta backup will fail with "can't create a stream from a metadata VDI, fall back to a base"

After this backup has failed, attempting to run any incremental backup will fail. Even the Continuous Replication backups that were working correctly prior to the Delta Backup attempt.

• NDB Connections can be any amount.
• With NBD + CBT = True
• Purge snapshots when using CBP = True

Not purging snapshots generates the same error but also causes problems with the VDI chain when it happens.

All retention policies are set to 1.

This was tested on the current latest version of XCP-NG on all hosts and with commit 0a28a and multiple commits prior to it of the XO Community Edition.

@michmoor0725 Sure!

I don't think that it is a ronivay issue as it appears to be occurring with any self-signed certificate. Regardless of what was used to compile it.

That said, if some people check there instead of here, it mays good sense to have something posted there as well.

For those finding this post and are using ronivay's XenOrchestraInstallerUpdater, the solution is as follows...

Edit xo-install.cfg with your favourite editor.

Look for the line...

BRANCH="master"

Comment it out and and on a new line add...

#BRANCH="master"
BRANCH="0ccfd4b"

If you then run the script and choose 2) Update, it should work.

Because I was in a hurry, I deleted the /opt/ox directory, put the SSL certs that were in it back, and then ran 1) Install so that I knew that I was certain that there were no conflicts with compiling and older version.

But either method should work.

I can confirm that in the community edition, something between 0ccfb4b and 8e5d9 breaks restoring to hosts with self signed certs.

I had to try and restore a couple of VMs after the upgrade to 8e5d9 and despite a couple of attempts to compile previous commits and see if they would work, nothing did until I reached 0ccfb4b.

Once I had rolled it back that far, the restores were no longer stuck at 0% and there were no problems restoring any of the backups that I tied.