Just updating for anyone that has the same issue, we ended up just rebooting the master and like @JamuelStarkey said everything just automatically fell in place. Did have to exit maintenance mode on the master and replug the PBD but everything else went back to normal immediately.

Still frustrating to experience and would really love to know what caused this. If there's any logs I can pull to figure this out do let me know @olivierlambert

@olivierlambert Shouldn't it be a disaster recovery option in case one is unable to spin up another XO instance?

@olivierlambert Neither. A ubuntu VM with backups-cli installed via npm install --global @xen-orchestra/backups-cli

@olivierlambert I was running the clean-vms -m command to try and merge a bunch of delta backups into a single .vhd file but it throws this strange assertion error which makes no sense to me. Looking for suggestions because it should be a relatively straightfoward task..

xo-backups clean-vms -m /vm1
VHD check error {
  path: '/vm1/vdis/e8c6772b-0bab-489d-a24e-b41e07f9298b/7572fba5-2cbb-496a-9349-deb8f0c1af29/.20240914T191707Z.vhd',
  error: AssertionError [ERR_ASSERTION]: footer1 !== footer2
      at VhdFile.readHeaderAndFooter (/usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/vhd-lib/Vhd/VhdFile.js:195:7)
      at async VhdFile.open (/usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/vhd-lib/Vhd/VhdFile.js:95:7)
      at async openVhd (/usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/vhd-lib/openVhd.js:15:14)
      at async Promise.all (index 0)
      at async Set.<anonymous> (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/@xen-orchestra/backups/_cleanVm.mjs:209:7)
      at async Promise.all (index 0)
      at async RemoteAdapter.cleanVm (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/@xen-orchestra/backups/_cleanVm.mjs:203:3)
      at async Disposable.<anonymous> (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/commands/clean-vms.mjs:26:9)
      at async Promise.all (index 0)
      at async cleanVms (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/commands/clean-vms.mjs:23:3) {
    generatedMessage: false,
    code: 'ERR_ASSERTION',
    actual: false,
    expected: true,
    operator: '=='
  }
}

@Danp I don't really need something complex. I see the full & delta vhd files in each VM's folder but trying to mount the delta vhd with guestmount throws an error. There has to be a way to fold the delta vhds into the main vhd for the most up to date vhd..

@Danp I cannot manually call the restore script that XO uses? Or is there some explanation on what steps are taken so I can reproduce it myself?

There's no available SR to restore the VMs to on the old pool as there was a complete failure. I need to rebuild the VDI for export because I am still looking for a place to re-home the VMs and might need to import them into other hypervisors. Can I just zip the largest folder to make a .raw image?

@Danp There is no SR available for the current pool and the new pool is just a make shift temporary solution that is running regular Xen (no XCP-ng). Is there any script that I can run that will reconstruct it as .xva or .raw format?

We had a complete failure of SR and need to restore from backup to a complete new cluster. Is there a way to reconstruct a VDI from delta backups that can be imported straight as a template?

Just updating for anyone that has the same issue, we ended up just rebooting the master and like @JamuelStarkey said everything just automatically fell in place. Did have to exit maintenance mode on the master and replug the PBD but everything else went back to normal immediately.

Still frustrating to experience and would really love to know what caused this. If there's any logs I can pull to figure this out do let me know @olivierlambert

@JamuelStarkey Just want to reconfirm this, when you had the issue, the master was still connected to everything single thing on the network just unable to see slaves?