@olivierlambert Shouldn't it be a disaster recovery option in case one is unable to spin up another XO instance?

@olivierlambert Neither. A ubuntu VM with backups-cli installed via npm install --global @xen-orchestra/backups-cli

@olivierlambert I was running the clean-vms -m command to try and merge a bunch of delta backups into a single .vhd file but it throws this strange assertion error which makes no sense to me. Looking for suggestions because it should be a relatively straightfoward task..

xo-backups clean-vms -m /vm1
VHD check error {
  path: '/vm1/vdis/e8c6772b-0bab-489d-a24e-b41e07f9298b/7572fba5-2cbb-496a-9349-deb8f0c1af29/.20240914T191707Z.vhd',
  error: AssertionError [ERR_ASSERTION]: footer1 !== footer2
      at VhdFile.readHeaderAndFooter (/usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/vhd-lib/Vhd/VhdFile.js:195:7)
      at async VhdFile.open (/usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/vhd-lib/Vhd/VhdFile.js:95:7)
      at async openVhd (/usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/vhd-lib/openVhd.js:15:14)
      at async Promise.all (index 0)
      at async Set.<anonymous> (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/@xen-orchestra/backups/_cleanVm.mjs:209:7)
      at async Promise.all (index 0)
      at async RemoteAdapter.cleanVm (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/node_modules/@xen-orchestra/backups/_cleanVm.mjs:203:3)
      at async Disposable.<anonymous> (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/commands/clean-vms.mjs:26:9)
      at async Promise.all (index 0)
      at async cleanVms (file:///usr/lib/node_modules/@xen-orchestra/backups-cli/commands/clean-vms.mjs:23:3) {
    generatedMessage: false,
    code: 'ERR_ASSERTION',
    actual: false,
    expected: true,
    operator: '=='
  }
}

@Danp I don't really need something complex. I see the full & delta vhd files in each VM's folder but trying to mount the delta vhd with guestmount throws an error. There has to be a way to fold the delta vhds into the main vhd for the most up to date vhd..

@Danp I cannot manually call the restore script that XO uses? Or is there some explanation on what steps are taken so I can reproduce it myself?

There's no available SR to restore the VMs to on the old pool as there was a complete failure. I need to rebuild the VDI for export because I am still looking for a place to re-home the VMs and might need to import them into other hypervisors. Can I just zip the largest folder to make a .raw image?

@Danp There is no SR available for the current pool and the new pool is just a make shift temporary solution that is running regular Xen (no XCP-ng). Is there any script that I can run that will reconstruct it as .xva or .raw format?

We had a complete failure of SR and need to restore from backup to a complete new cluster. Is there a way to reconstruct a VDI from delta backups that can be imported straight as a template?

Just updating for anyone that has the same issue, we ended up just rebooting the master and like @JamuelStarkey said everything just automatically fell in place. Did have to exit maintenance mode on the master and replug the PBD but everything else went back to normal immediately.

Still frustrating to experience and would really love to know what caused this. If there's any logs I can pull to figure this out do let me know @olivierlambert

@JamuelStarkey Just want to reconfirm this, when you had the issue, the master was still connected to everything single thing on the network just unable to see slaves?

@JamuelStarkey Can I just confirm that you had the same network issues where communication between master and slave was severed but master was still connected to the internet? Did you not have to touch the slaves at all?

Hi all,

Overnight our pool went into a weird situation where the the master seemed to see all slaves as offline.
Upon investigation, it seems like all nodes are still online and not in emergency mode.
All nodes still think that they have the same master in the pool.conf file.
Able to SSH into all nodes including the master and access all parts of the network.
No isuses with connectivity with iSCSI storage.
All slaves can ping each other except the master.
VMs that are NOT on the master node seem to be running fine.
VMs on the master node are behaving weird (most have no internet connectivity).
XAPI service is running on all hosts (although master has this extra warning line "Warning: xapi.service changed on disk. Run 'systemctl daemon-reload' to reload units."
XAPI commands seem to hang on slaves (xe sr-list/vm-list/host-list)
Unable to log into slaves on XCP-ng Center because it prompts to log into master and master sees all slaves as offline.

What is the cleanest way to gracefully fix this? Maybe transition one of the slaves into the master?

Thanks!

But the second schedule still applies to both remotes. I'm trying to have one remote with 3 days retention and the other with 5 but using the same snapshot instance.

Where can I find more information about multi schedule? I see the option to add multiple remotes per backup job but under schedule settings the backup retention option seems to apply to the entire backup job or all remotes selected.

Yup. I managed to clear out the session. So the next step would be to disable the backups on the old XO and enable it on the new XO and observe one backup cycle for any issues?

Btw, is there any way to use the same backup job to backup to two different remotes with different retention count (3/5 days for remote 1/2)? Right now it's on two separate jobs but wondering if there's any new methods since I set it up many years ago.

@olivierlambert I spun up a fresh install on XO with the latest commit in master, imported the config from the old XO and now the GUI is stuck in a perpetual reload of "still loading". No error printed in the console.

Edit: Found the issue in the journal, it was trying to auth with the old credentials that was no longer valid with imported config. Might want to consider forcing a relogin after config import

Old XO as in version that is "broken". In any case the xo-server version is 5.52.1

@olivierlambert Will the fresh install play nice with the incremental backups from the old XO?

Is there a reason why SSHFS isn't recommended? Or is there another more secure way to mount a remote than NFS?

@olivierlambert I still can't add the remote though. Under the backup it's empty as well.