XO rebooted, remotes lost, unable to add back
-
Hi,
Our remote is mounted via SSHFS to /mnt/backups via fstab. XO server had a power failure and restarted but the remotes are all missing from the "remotes" tab. The mount is working fine, ls shows all the files and touch works for r/w permissions. Trying to readd the remote via XO throws "invalid status closed, expected open" and a link to logs but the logs page never loads.
Refreshing the logs page shows the last log as:
"VM_SHUTDOWN_TIMEOUT(OpaqueRef:69b3e5a5-873c-4841-b68f-8438dc782b04, 1200.)
This is a XenServer/XCP-ng error".Appreciate any ideas.
Thanks!
-
Hi!
Just refresh the web UI. You probably have a stuck/stalled websocket issue between your browser and XO.
Also, your configuration is not officially tested/supported (local backup repository mounted then internally in SSHFS).
-
@olivierlambert I still can't add the remote though. Under the backup it's empty as well.
-
You have clearly have a problem with your XO installation. Can you try with a fresh XOA fully up to date and see if you can import the configuration from your XO from the sources? (assuming that's what you use, BTW hopefully you are entirely up to date on this one?)
-
@olivierlambert Will the fresh install play nice with the incremental backups from the old XO?
Is there a reason why SSHFS isn't recommended? Or is there another more secure way to mount a remote than NFS?
-
What do you mean by "old XO"? How old is your XO?
-
Old XO as in version that is "broken". In any case the xo-server version is 5.52.1
-
We usually talk in commit number now. Also check the big yellow warning here: https://xen-orchestra.com/docs/community.html#report-a-bug
You should at least install a fresh one on the side, import the config and check the behavior. It's dangerous to keep an old XO version.
-
@olivierlambert I spun up a fresh install on XO with the latest commit in master, imported the config from the old XO and now the GUI is stuck in a perpetual reload of "still loading". No error printed in the console.
Edit: Found the issue in the journal, it was trying to auth with the old credentials that was no longer valid with imported config. Might want to consider forcing a relogin after config import
-
Ah yes, it's something that could happen, in that case, using the browser private mode or removing your cookie helps
-
Yup. I managed to clear out the session. So the next step would be to disable the backups on the old XO and enable it on the new XO and observe one backup cycle for any issues?
Btw, is there any way to use the same backup job to backup to two different remotes with different retention count (3/5 days for remote 1/2)? Right now it's on two separate jobs but wondering if there's any new methods since I set it up many years ago.
-
- You can indeed "resume" the jobs, but remember it will use the "previous" backup format (which is fine).
- Yes, you can have multi remote and multi schedule jobs now.
-
Where can I find more information about multi schedule? I see the option to add multiple remotes per backup job but under schedule settings the backup retention option seems to apply to the entire backup job or all remotes selected.
-
Just create another schedule
There's a "+" button in the schedule card. -
But the second schedule still applies to both remotes. I'm trying to have one remote with 3 days retention and the other with 5 but using the same snapshot instance.
-
So what you need is more something like backup tiering (or secondary backup). This feature is currently a work in progress by @florent
