Correct - through the CLI one could see that the old VM that had been migrated from Host1 to Host2 was still showing under Host1, whilst the migrated copy was showing as running on Host2. Once the Host1 remnant of the migration was removed that cleared things and XO correctly reported the VM as running on Host2 with its disks attached.

TLDR - There were no other conflicts beyond what appeared through XO to be the only version sitting halted on Host1, but through the CLI one could see the halted copy on Host1 and the running copy on Host2. Somehow the running version did not show in XO until the remnant was removed.

Thanks for your help @olivierlambert

Solved.

Thank you - superb deduction skills

Ah, you genius. Yes, disconnecting Host1/Pool1 does indeed have the VM magically appear. So there must be a duplicate on Host1 that 'shields' XO from seeing it - only the dup doesn't show in XO. Will check via CLI - I have enough now to find the issue.

Thank you!!

Correct

It's listed in XO as being halted on Host1 without any disks but (correctly) doesn't show on Host1 under xe list, meanwhile it's listed (correctly) in Host2 xe list as running on there.

Have tried from XO on different boxes (Host1, Host2 and also on a different host) but all report exactly as above - halted on Host1 with no disks.

Have restarted toolstack on Host1 and Host2, have rebooted Host1 and Host2, have rebooted VM from ssh (daren't do it from within XO in case it doesn't come up, similarly dare not stop it from ssh and then try to start it from XO), have tried migrating within XO but does not work, have tried booting from XO (despite it already running) but it does not work.

In summary - in reality it is running on Host2 just fine; xe list is reporting correctly but XO is not.

@olivierlambert Correct

@olivierlambert Thanks for the reply.

Console (xl list) on Host1 does not show VM at all (despite XO thinking it's there without any disks but halted), meanwhile the same on Host2 shows it as booted (although XO does not show it at all on that host).

Toolstack has been restarted several times at both ends to no avail.

I'm a little confused!

Strange one for you:

A running VM was (live) migrated from Host 1 (Pool 1) to Host 2 (Pool 2 - each pool only has a single host). The migration failed (too long ago now for me to have the error to hand - but for us it is not an unusual circumstance for live migrations to fail when the disk is > 250gb) and the VM continued to function BUT it shows in XO as not running but present on Host 1 without a disk attached. However in reality it is running on Host 2 - when shutting down Host 1 the VM continues to operate just fine, when shutting down Host 2 it terminates. .

Starting the allegedly stopped VM in XO fails (starting it on Host1 although as above we know it's running on Host2) it shows as it started for about 10 seconds then goes back to stopped - but no error is given.

Migrating it results in the same outcome - it appears to migrate ok but nothing changes and there are no error messages - but remember that XO has it on the wrong host and without an attache disk. Rebooting the VM from its own console direct via SSH comes up just fine, but the issue endures in Xen Orchestra, and rebooting both hosts also fails to resolve the issue.

Is there a way (without damaging the running VM!) to have it re-registered in XO? As an aside, backups using XO are working for the 'stopped' VM except that as it shows in XO as having no attached disk, they take 2s and are worthless.

[Xen Orchestra, commit 00a17 (although will have been updated a few time since the issue presented itself a month or so ago), both hosts on 8.2.1.]

@florent Can't wait!

As an aside, using the same build I am also getting problems when running CR over a slower WAN, with the following error. Just mentioning it here in case it's related.

"message": "VDI_IO_ERROR(Device I/O errors)",
            "name": "XapiError",
            "stack": "XapiError: VDI_IO_ERROR(Device I/O errors)\n    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/_XapiError.js:26:12)\n    at _default (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/_getTaskResult.js:24:38)\n    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:761:51)\n    at /opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:789:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:774:12)\n    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202105081131/packages/xen-api/dist/index.js:931:14)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
          }
        }
      ],
      "end": 1620864312824,
      "result": {
        "message": "all targets have failed, step: writer.transfer()",
        "name": "Error",
        "stack": "Error: all targets have failed, step: writer.transfer()\n    at VmBackup._callWriters (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/_VmBackup.js:118:13)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at async VmBackup._copyDelta (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/_VmBackup.js:190:5)\n    at async VmBackup.run (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/_VmBackup.js:371:9)"

CR to a local machine is working fine, the remote one is failing every iteration.

New S3 function (albeit using Backblaze) seems to be working really well for small backups; I'm now seeing almost 100% success.

However larger backups are consistently failing with

"message": "Error calling AWS.S3.upload: aborted",
            "name": "Error",
            "stack": "Error: Error calling AWS.S3.upload: aborted\n    at rethrow (/opt/xo/xo-builds/xen-orchestra-202105081131/node_modules/@sullux/aws-sdk/index.js:254:24)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at async S3Handler._outputStream (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/fs/dist/s3.js:100:5)\n    at async S3Handler.outputStream (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/fs/dist/abstract.js:250:5)\n    at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/RemoteAdapter.js:509:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202105081131/@xen-orchestra/backups/writers/FullBackupWriter.js:69:7"

Running with xo-server 5.79.3 , xo-web 5.81.0

I'm having the same issue across different buckets in B2 and from different servers.

Any ideas?

@nraynaud Getting the "not implemented" error on all branches at the moment - is there a functioning, albeit not perfect branch that we can run with as a temporary measure? Thanks chap

@nraynaud Bit of a hiccup I'm afraid ; errors out with "Not Implemented":

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1615570162008",
  "jobId": "b947a31c-35f7-45d9-af88-628ec027c71e",
  "jobName": "B2",
  "message": "backup",
  "scheduleId": "8481e1b8-3d22-4502-abb7-9f0413aca0a3",
  "start": 1615570162008,
  "status": "pending",
  "infos": [
    {
      "data": {
        "vms": [
          "25a4e2a6-b116-7cee-b94b-f3553fe001f2",
          "24103ce1-e47b-fe12-4029-d643e0382f08",
          "60ac886d-8adf-9e21-4baa-b14e0fc1b2bb"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "25a4e2a6-b116-7cee-b94b-f3553fe001f2"
      },
      "id": "1615570162050",
      "message": "backup VM",
      "start": 1615570162050,
      "status": "pending",
      "tasks": [
        {
          "id": "1615570162064",
          "message": "snapshot",
          "start": 1615570162064,
          "status": "success",
          "end": 1615570163265,
          "result": "294d65fc-ddc7-b8ad-68b8-d14e36b6838e"
        },
        {
          "data": {
            "id": "df07dd6b-753e-40f2-89be-75404bac2c1e",
            "type": "remote",
            "isFull": true
          },
          "id": "1615570163286",
          "message": "export",
          "start": 1615570163286,
          "status": "failure",
          "tasks": [
            {
              "id": "1615570164234",
              "message": "transfer",
              "start": 1615570164234,
              "status": "failure",
              "end": 1615570164257,
              "result": {
                "message": "Not implemented",
                "name": "Error",
                "stack": "Error: Not implemented\n    at S3Handler._createWriteStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/fs/src/abstract.js:428:11)\n    at S3Handler._createOutputStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/fs/src/abstract.js:412:25)\n    at S3Handler.createOutputStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/fs/src/abstract.js:129:12)\n    at RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/RemoteAdapter.js:513:34)\n    at /opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/_FullBackupWriter.js:70:7\n    at FullBackupWriter.run (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/_FullBackupWriter.js:69:5)\n    at Array.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/_VmBackup.js:209:9)"
              }
            }
          ],
          "end": 1615570164258,
          "result": {
            "message": "Not implemented",
            "name": "Error",
            "stack": "Error: Not implemented\n    at S3Handler._createWriteStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/fs/src/abstract.js:428:11)\n    at S3Handler._createOutputStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/fs/src/abstract.js:412:25)\n    at S3Handler.createOutputStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/fs/src/abstract.js:129:12)\n    at RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/RemoteAdapter.js:513:34)\n    at /opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/_FullBackupWriter.js:70:7\n    at FullBackupWriter.run (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/_FullBackupWriter.js:69:5)\n    at Array.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202103121606/@xen-orchestra/backups/_VmBackup.js:209:9)"
          }
        }
      ]
    }
  ]
}

I'll try over the weekend with a fresh install. Backing up to Backblaze B2 rather than S3 - likely to be the issue?

@olivierlambert Ahhh, ok. Was focusing on XO; that's pointed me in the right direction. Thanks...and have a great weekend!

@nraynaud Building XO now; will run a few backups overnight.

Anything in particular you'd like me to look for / test ?

Having repeated errors migrating VMs off secondary server back to the primary that they originated from. Same error message for all 17 VMs - all linux unlike the Win machines described in other similar posts.

Any thoughts?

vm.migrate
{
  "vm": "761e6088-2c1c-2113-6f39-214cde1026e1",
  "mapVifsNetworks": {
    "60bcd3b6-4800-631e-7e74-8af907ff8a25": "11fb32b0-ac18-cc90-3b01-d8ec76471db2"
  },
  "migrationNetwork": "1d23ef9c-7d04-bbe9-a6a6-505e3b4dd1f5",
  "sr": "31735171-91da-832f-a335-060227e19339",
  "targetHost": "8b7ead55-4a9f-46e5-99f7-da1ba170e3bd"
}
{
  "code": 21,
  "data": {
    "objectId": "761e6088-2c1c-2113-6f39-214cde1026e1",
    "code": "INTERNAL_ERROR"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at factory (/home/node/xen-orchestra/packages/xo-common/src/api-errors.js:21:32)
    at /home/node/xen-orchestra/packages/xo-server/src/api/vm.js:487:15
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at runNextTicks (internal/process/task_queues.js:62:3)
    at processImmediate (internal/timers.js:431:9)
    at Object.migrate (/home/node/xen-orchestra/packages/xo-server/src/api/vm.js:474:3)
    at Api.callApiMethod (/home/node/xen-orchestra/packages/xo-server/src/xo-mixins/api.js:304:20)"
}

xo-web 5.77.0
xo-server 5.74.1
Latest XCP-ng fully patched.

Thanks for any thoughts!

@danp Sorry, since it seems to be a design item rather than a bug I didn't think to put the logs in. Here you go:

vm.migrate
{
  "vm": "24103ce1-e47b-fe12-4029-d643e0382f08",
  "mapVifsNetworks": {
    "7457d175-8d01-613e-7b47-fb1714693074": "b62c7a9a-222a-e8e9-754e-982839e00d0e"
  },
  "migrationNetwork": "ef24440c-fda5-d88b-ce4a-fd12b7ad1d4d",
  "sr": "cf2dbaa3-21f3-903b-0fd1-fbe68539f897",
  "targetHost": "98da99c3-4ec2-4db8-ab1b-a1cb6ffd329a"
}
{
  "code": 21,
  "data": {
    "objectId": "24103ce1-e47b-fe12-4029-d643e0382f08",
    "code": "DUPLICATE_VM"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at factory (/opt/xo/xo-builds/xen-orchestra-202102171611/packages/xo-common/src/api-errors.js:21:32)
    at /opt/xo/xo-builds/xen-orchestra-202102171611/packages/xo-server/src/api/vm.js:487:15
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at runNextTicks (internal/process/task_queues.js:66:3)
    at processImmediate (internal/timers.js:434:9)
    at Object.migrate (/opt/xo/xo-builds/xen-orchestra-202102171611/packages/xo-server/src/api/vm.js:474:3)
    at Api.callApiMethod (/opt/xo/xo-builds/xen-orchestra-202102171611/packages/xo-server/src/xo-mixins/api.js:304:20)"
} 

@jedimarcus That is correct; issue ceases to be a problem with shared storage.