Error: couldn't instantiate any remote

screame1

Hello!

Recently (just yesterday) installed a fresh copy of XO from sources. I am trying to set up a 'Continuous Replication' backup job to simply copy VMs from one server to another.

Just as soon as I start the backup, I get an error. I've attached a copy of the log below:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1664280752028",
  "jobId": "04dce46f-71fa-41b6-9f4c-197a500f429b",
  "jobName": "FCC-APPDATA (Continuous)",
  "message": "backup",
  "scheduleId": "52f1dd3a-3a8a-47f6-8bcf-cae72ac41ef0",
  "start": 1664280752028,
  "status": "failure",
  "end": 1664280752032,
  "result": {
    "errors": {},
    "message": "couldn't instantiate any remote",
    "name": "Error",
    "stack": "Error: couldn't instantiate any remote\n    at executor (file:///opt/xo/xo-builds/xen-orchestra-202209261451/packages/xo-server/src/xo-mixins/backups-ng/index.mjs:233:27)\n    at file:///opt/xo/xo-builds/xen-orchestra-202209261451/packages/xo-server/src/xo-mixins/jobs/index.mjs:263:30\n    at Jobs._runJob (file:///opt/xo/xo-builds/xen-orchestra-202209261451/packages/xo-server/src/xo-mixins/jobs/index.mjs:292:22)\n    at Jobs.runJobSequence (file:///opt/xo/xo-builds/xen-orchestra-202209261451/packages/xo-server/src/xo-mixins/jobs/index.mjs:332:7)\n    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202209261451/packages/xo-server/src/xo-mixins/api.mjs:394:20)"
  }
}

I've tried searching for this phrase and can't find any mention of it anywhere at all. Can anyone shed some light on a place to begin troubleshooting this error?

Thanks in advance!

AtaxyaNetwork

@screame1 Hi !

You need to configure first a remote, in Settings -> Remote

You can then configure a NFS, SMB or S3 remote

screame1

@AtaxyaNetwork - Thanks so much for offering the advice. Unfortunately, I'm confused. I understand setting up a remote for Delta backups, etc. I actually have one working pushing data to S3.

In this case, for the Continuous Replication backup, you choose the storage repository as another Xen server (which I did). I don't see anywhere that I would choose a remote storage source.

I feel like I'm having trouble describing this properly. Does that make sense?

Thanks again!

AtaxyaNetwork

@screame1 My bad, I read your first message too fast ^^

What is the version of XCP-ng/XEN server on each host?

screame1

@AtaxyaNetwork - No worries about reading the first post too fast! I just appreciate the offer of advice.

Both servers are running:
release/yangtze/master/58

Thanks again!

AtaxyaNetwork

@screame1 Can you show me how you have configured the Continuous Replication ? (with a screenshot, for example, don't forget to hide sensitive data )

screame1

@AtaxyaNetwork - Of course. I've attached it below...

As you can see, it's very straightforward. It backs up a single VM from server 'A' and attempts to make a continuous replication on server 'B'.

Just for reference, the XO VM is also on server 'B'.

I'm really stumped and sure appreciate you trying to help.

AtaxyaNetwork

@screame1
Can you enable "Smart mode" (in the top right, near "VMs to backup"), It's a better view to create backup ^^

Otherwise, it's seem correct for me !

Let me check my created Continuous replication on my infra, maybe I will see a difference

Both servers are running:
release/yangtze/master/58

For debuging it's best to provide XCP-ng/XenServer version. You can see that in Home -> Hosts -> Click on you host. You will see something like this

XCP-ng 8.2.1 is the version

screame1

@AtaxyaNetwork - Thanks again. I checked the version like you asked. It's XCP-ng 8.2.1 (GPLv2) just as you showed in your example.

From my understanding the "Smart Mode" is just a way to choose VMs by tags, etc. correct? I actually turned it off to ensure there wasn't something strange happening there. I've tried to drive this down to a bare minimum backup job in hopes of narrowing down the failure.

Thanks again for the time this morning!

AtaxyaNetwork

@screame1 Hum, i just created a continuous replication similar to yours, and it works well with XOA latest

Are you on the latest commit on XO source ? What kind of storage are you using in your servers (source and destination) ?

screame1

@AtaxyaNetwork - I am on the latest XO (just installed from source yesterday). The disks are straightforward. Simple Dell PowerEdge R730xd server with disks on a SCSI Raid locally mounted via LVM.

Can you think of any other logs or anything I should check?

Thanks!

JamfoFL

I can confirm the same thing has happened to my lab setup as is happening to Screame1.

I have three different servers that run a continuous replication, one every 10 minutes. Everything has been working perfectly for a couple weeks. Yesterday, at around 3:50 PM, I pulled down the latest XO update, 3d3b6, and my continuous replications stopped working from that point forward. The replication prior to the update (that ran at 3:40 PM) worked perfectly... the next one to run at 4:00 PM failed, and all have continued to fail from that point forward.

Like Screame1, I am also running XCP-ng 8.2.1 (GPLv2), with XO from sources update 3d3b6. Prior to updating yesterday, I was on XO update ed78d. It looks like one of the six commits that were pushed out between that version and the latest version has somehow broken the ability to run continuous replications.

I have a mix of hardware involved in the test lab... some are simple single drive workstations acting as hosts, and I have one HP server with standard SATA drives in a RAID 5 array.

Normal backups to an NFS share are working... only the continuous replications between hosts are broken.

JamfoFL

Just another bit of additional information. In my continuous replication jobs, the buttons that allow you to delete or edit the schedule for the jobs are missing, as well. Those same buttons are available for my Delta Backup jobs.
Missing buttons.png

I even deleted and build a new continuous replication job from scratch and have the same issue with the new job as with the originals.

Hopefully this extra info helps. If not, I'll also be happy to provide anything you ask for... hopefully since you now have more than one instance, combining the two will help!

Thanks!

Danp

@JamfoFL The buttons are likely there, just off screen. Can you try expanding the window or scrolling to the right?

JamfoFL

@Danp Yuuuup! I'm a flippin' idiot on that front. Thanks, Danp! I don't know why my screen is now reformat in those windows (it hadn't been like this previously, and doesn't look like this for the Delta Backups), but if it works and only looks funny, I can live with that. You can see how the border that usually surrounds all the fields for the scheduler is now "too short" and doesn't show stuff off of the screen.

Regardless... as I said, I can live with a slightly off-kilter screen. It's the complete loss of the ability to run continuous replications that of the greatest concern. I just want to throw in as much detail as possible.

Thanks again!!

JamfoFL

Just for completeness... here is the log for the failure of one of the continuous replications jobs that had been working for quite some time until after the update:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1664289600023",
  "jobId": "ab9ec0fb-43d5-4489-9f1b-b38e39b80e1a",
  "jobName": "VXCTEST-22-03 Replication",
  "message": "backup",
  "scheduleId": "99e24d76-1455-4e18-9f47-bc569cfe74e4",
  "start": 1664289600023,
  "status": "failure",
  "end": 1664289600028,
  "result": {
    "errors": {},
    "message": "couldn't instantiate any remote",
    "name": "Error",
    "stack": "Error: couldn't instantiate any remote\n    at executor (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng/index.mjs:233:27)\n    at file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.mjs:263:30\n    at Jobs._runJob (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.mjs:292:22)\n    at Jobs.runJobSequence (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.mjs:332:7)"
  }
}

And here is the log for the brand-new continuous replication job I set up just now to see if that would make a difference:

backupNg.runJob
{
  "id": "905e5413-f550-40af-afbb-027cfc2133f7",
  "schedule": "23b7fbac-77f0-4a0a-99cf-bb39ddb514ab"
}
{
  "errors": {},
  "message": "couldn't instantiate any remote",
  "name": "Error",
  "stack": "Error: couldn't instantiate any remote
    at executor (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng/index.mjs:233:27)
    at file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.mjs:263:30
    at Jobs._runJob (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.mjs:292:22)
    at Jobs.runJobSequence (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.mjs:332:7)
    at Api.#callApiMethod (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:394:20)"
}

JamfoFL

One last bit of information to add, then I'll shut up and let the professionals go at it!

As I have been running these jobs successfully for some time before this issue popped up, I did have a number of VMs sitting ready from prior continuous replication jobs. I can confirm that I was able to successfully power on and boot these already existing continuous replication VMs, so whatever the issue is, it is only affecting the ability to complete continuous replication jobs going forward, but has not affected the ability to access and utilize the results of previous jobs.

AtaxyaNetwork

@JamfoFL Nice catch !
i can't help anymore since i'm not a great dev, but @florent or @olivierlambert will be better than me to diagnostic these ! ^^

Danp

@JamfoFL What commit is showing under the XO About screen?

Error is likely related to this commit

JamfoFL

@Danp Current commit is 3d3b6.

xo-server: 5.103.0
xo-web: 5.104.0

Yes... as I had mentioned, everything worked until I updated yesterday at around 3:50 PM. I noticed there were five different commits that were released yesterday, and the one you linked to is one of those. So, as I figured, one of the commits from yesterday "broke" the ability to run continuous replications.

Now that it looks like you've focused on a cause, I'm sure it's only a matter of time until a new commit is published to fix the issue. Once it's released, I'll get it installed and, hopefully, we can put this behind us.

Thanks!!