Right, so yeah I did just that - disabled HA and have been keeping an eye on the logs as well as general performance in the env. Glad to know my actions align with your recommendations at least

There are a couple of hosts that get a lot of these sorts of messages in the kern.log:

Apr 25 08:15:33 oryx kernel: [276757.645457] vif vif-21-3 vif21.3: Guest Rx ready
Apr 25 08:15:54 oryx kernel: [276778.735509] vif vif-22-3 vif22.3: Guest Rx stalled
Apr 25 08:15:54 oryx kernel: [276778.735522] vif vif-21-3 vif21.3: Guest Rx stalled
Apr 25 08:16:04 oryx kernel: [276788.780828] vif vif-21-3 vif21.3: Guest Rx ready
Apr 25 08:16:04 oryx kernel: [276788.780836] vif vif-22-3 vif22.3: Guest Rx ready

Am I wrong to attribute this to issues within specific VMs (i.e. not a hv performance issue)?

I know one of the VMs that causes these is a very old centos 5 testing VM one of my devs use and the messages stop when it's powered down.

Is there any way to easily associate those vifs with the actual VMs they are attached to? My google-foo failed me for that.

Other than that, I noticed my nic firmware is a bit old on the X710-da2's I use so I'm going through and upgrading those with no noticeable changes. I'm fairly hesitant to re-enable HA without tracking down the root cause.

On Sunday I did pool updates for the latest patches and everything seemed to go fine for the most part. However, the last two nights I have had hosts suddenly decide they needed to be fenced and reboot. On Monday morning it was 2 of them, and then this morning 4 of them that rebooted (it's a 5 node pool).

Did any of the most recent updates make HA a bit more punchy or is that coincidental?

Interpreting the xha log is pretty rough for me at this point so I'm having a trouble determining the exact source of the problem - it certainly seems to be a state file disagreement. I'm thinking maybe the backups task, which kicks off at midnight, is having a negative impact to the storage performance enough to cause this - but that's only a guess.

Any insight would be appreciated!

xha.log:

Apr 23 00:26:41 CDT 2024 [debug] WD: now - settime = 38038 timeout = 75000.
Apr 23 00:26:41 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:44 CDT 2024 [debug] WD: now - settime = 41040 timeout = 75000.
Apr 23 00:26:44 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:47 CDT 2024 [debug] WD: now - settime = 44042 timeout = 75000.
Apr 23 00:26:47 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:50 CDT 2024 [debug] WD: now - settime = 47044 timeout = 75000.
Apr 23 00:26:50 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:53 CDT 2024 [debug] WD: now - settime = 50047 timeout = 75000.
Apr 23 00:26:53 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:55 CDT 2024 [debug] FH: Start fault handler.
Apr 23 00:26:55 CDT 2024 [debug] FH: waiting for SF from host (0), time since last SF update = 56037.
Apr 23 00:26:55 CDT 2024 [debug] FH: waiting for SF from host (1), time since last SF update = 55969.
Apr 23 00:26:55 CDT 2024 [debug] FH: waiting for SF from host (2), time since last SF update = 55825.
Apr 23 00:26:55 CDT 2024 [debug] FH: waiting for SF from host (3), time since last SF update = 55606.
Apr 23 00:26:55 CDT 2024 [debug] FH: waiting for SF from host (4), time since last SF update = 55390.
Apr 23 00:26:55 CDT 2024 [debug] FH: waiting for SF from host (5), time since last SF update = 55194.
Apr 23 00:26:56 CDT 2024 [debug] WD: now - settime = 53049 timeout = 75000.
Apr 23 00:26:56 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:59 CDT 2024 [debug] WD: now - settime = 56050 timeout = 75000.
Apr 23 00:26:59 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:26:59 CDT 2024 [debug] SM: SF domain is updated [sfdomain = (_11111)].
Apr 23 00:27:00 CDT 2024 [debug] SM: SF domain is updated [sfdomain = (0_1111)].
Apr 23 00:27:00 CDT 2024 [debug] SM: SF domain is updated [sfdomain = (00_111)].
Apr 23 00:27:00 CDT 2024 [debug] SM: SF domain is updated [sfdomain = (000_11)].
Apr 23 00:27:00 CDT 2024 [debug] SM: SF domain is updated [sfdomain = (0000_1)].
Apr 23 00:27:00 CDT 2024 [debug] SM: SF domain is updated [sfdomain = (00000_)].
Apr 23 00:27:00 CDT 2024 [debug] FH: HB/SF state has become stable.
Apr 23 00:27:00 CDT 2024 [debug] FH: weight value [1] is commited.
Apr 23 00:27:01 CDT 2024 [warn] SC: (script_service_do_query_liveset) reporting "State-File approaching timeout". host[0].time_since_last_update_on_sf=61256.
Apr 23 00:27:01 CDT 2024 [warn] SC: (script_service_do_query_liveset) reporting "State-File approaching timeout". host[1].time_since_last_update_on_sf=61188.
Apr 23 00:27:01 CDT 2024 [warn] SC: (script_service_do_query_liveset) reporting "State-File approaching timeout". host[2].time_since_last_update_on_sf=61044.
Apr 23 00:27:01 CDT 2024 [warn] SC: (script_service_do_query_liveset) reporting "State-File approaching timeout". host[3].time_since_last_update_on_sf=60825.
Apr 23 00:27:01 CDT 2024 [warn] SC: (script_service_do_query_liveset) reporting "State-File approaching timeout". host[4].time_since_last_update_on_sf=60609.
Apr 23 00:27:01 CDT 2024 [warn] SC: (script_service_do_query_liveset) reporting "State-File approaching timeout". host[5].time_since_last_update_on_sf=60413.
Apr 23 00:27:02 CDT 2024 [debug] WD: now - settime = 58829 timeout = 75000.
Apr 23 00:27:02 CDT 2024 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Apr 23 00:27:03 CDT 2024 [debug] FH: waiting for consistent view...
Apr 23 00:27:41 CDT 2024 [info] SC: (script_service_do_query_liveset) "State-file approaching timeout" turned FALSE 
Apr 23 00:28:03 CDT 2024 [warn] Host (1) and the local host do not agre on the view to the pool membership.
Apr 23 00:28:03 CDT 2024 [warn]         local HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         local SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         remote HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         remote SF domain = (011111)
Apr 23 00:28:03 CDT 2024 [warn]         remote HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         remote SF domain on HB = (011111)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (011111)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (011111)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         weight[0] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[1] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[2] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[3] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[4] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[5] = 1
Apr 23 00:28:03 CDT 2024 [warn] after merger:
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (011111)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (011111)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain = (111111)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other HB domain on SF = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         other SF domain on HB = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         weight[0] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[1] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[2] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[3] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[4] = 1
Apr 23 00:28:03 CDT 2024 [warn]         weight[5] = 1
Apr 23 00:28:03 CDT 2024 [warn]         merged HB domain = (000000)
Apr 23 00:28:03 CDT 2024 [warn]         merged SF domain = (000000)
Apr 23 00:28:03 CDT 2024 [debug] FH: All hosts now have consistent view to the pool membership.
Apr 23 00:28:06 CDT 2024 [warn] SM: partition_size[0] = 0 0
Apr 23 00:28:06 CDT 2024 [warn] SM: partition_size[1] = 0 0
Apr 23 00:28:06 CDT 2024 [warn] SM: partition_size[2] = 0 0
Apr 23 00:28:06 CDT 2024 [warn] SM: partition_size[3] = 0 0
Apr 23 00:28:06 CDT 2024 [warn] SM: partition_size[4] = 0 0
Apr 23 00:28:06 CDT 2024 [warn] SM: partition_size[5] = 0 0
Apr 23 00:28:06 CDT 2024 [warn] SM: winner_index = -1
Apr 23 00:28:06 CDT 2024 [debug] FH: I have lost.
Apr 23 00:28:06 CDT 2024 [err] Survival rule failed (1905) FH: Survival Rule is not met for the local host.  - Self-Fence.
Apr 23 00:28:06 CDT 2024 [info] watchdog_selffence.

I resolved the host missing stats by ejecting it from the pool and re-adding it.

Is it possible the stacking might be caused by the usage-report plugin?
I disabled that one too and it seems like it may have subsided (at least so far)

@jasonmap and might be unrelated but I'm not sure: one of my nodes rebooted without stats working either (not the master). Just says "No stats" on the tab. The others seem fine.

Wanted to chime in here to say I updated my 5 node cluster to the latest 8.2.1 and xo from source to 7b084. Definitely getting the hanging "Xapi#getResource /rrd_updates" tasks even with the perf-alert plugin disabled.

@florent Nice! This latest change allowed my migration to complete successfully. Seems like the peak transfer speed was about 70Mbps. 4.77GB in 5 minutes. I'm guessing the thin/zeros made this so fast?

@florent

Over the weekend I spun up an XO instance from source. This morning I changed to 'fix_xva_import_thin' after your post. Unfortunately, still the same failure for me. Only notable difference I see is that your ${str} addition for the log now comes back as " undefined"

Here are my logs:

From XO:

vm.importMultipleFromEsxi
{
  "concurrency": 2,
  "host": "vsphere.nest.local",
  "network": "7f7d2fcc-c78b-b1c9-101a-0ca9570e3462",
  "password": "* obfuscated *",
  "sr": "50d8f945-8ae4-dd87-0149-e6054a10d51f",
  "sslVerify": false,
  "stopOnError": true,
  "stopSource": true,
  "user": "administrator@vsphere.local",
  "vms": [
    "vm-2427"
  ]
}
{
  "succeeded": {},
  "message": "no opaque ref found in  undefined",
  "name": "Error",
  "stack": "Error: no opaque ref found in  undefined
    at importVm (file:///opt/xo/xo-builds/xen-orchestra-202402050455/@xen-orchestra/xva/importVm.mjs:28:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at importVdi (file:///opt/xo/xo-builds/xen-orchestra-202402050455/@xen-orchestra/xva/importVdi.mjs:6:17)
    at file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xo-server/src/xo-mixins/migrate-vm.mjs:260:21
    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202402050455/@vates/task/index.js:158:22)
    at Task.run (/opt/xo/xo-builds/xen-orchestra-202402050455/@vates/task/index.js:141:20)"
}

and from journalctl:

Feb 05 05:12:04 xoa-fs xo-server[32410]: 2024-02-05T10:12:04.864Z xo:xo-server WARN possibly unhandled rejection {
Feb 05 05:12:04 xoa-fs xo-server[32410]:   error: Error: already finalized or destroyed
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at Pack.entry (/opt/xo/xo-builds/xen-orchestra-202402050455/node_modules/tar-stream/pack.js:138:51)
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at Pack.resolver (/opt/xo/xo-builds/xen-orchestra-202402050455/node_modules/promise-toolbox/fromCallback.js:5:6)
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at Promise._execute (/opt/xo/xo-builds/xen-orchestra-202402050455/node_modules/bluebird/js/release/debuggability.js:384:9)
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at Promise._resolveFromExecutor (/opt/xo/xo-builds/xen-orchestra-202402050455/node_modules/bluebird/js/release/promise.js:518:18)
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at new Promise (/opt/xo/xo-builds/xen-orchestra-202402050455/node_modules/bluebird/js/release/promise.js:103:10)
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at Pack.fromCallback (/opt/xo/xo-builds/xen-orchestra-202402050455/node_modules/promise-toolbox/fromCallback.js:9:10)
Feb 05 05:12:04 xoa-fs xo-server[32410]:       at writeBlock (file:///opt/xo/xo-builds/xen-orchestra-202402050455/@xen-orchestra/xva/_writeDisk.mjs:15:22)
Feb 05 05:12:04 xoa-fs xo-server[32410]: }
Feb 05 05:12:06 xoa-fs xo-server[32410]: root@10.96.22.111 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xen-api/_XapiError.mjs:16:12)
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xen-api/_getTaskResult.mjs:11:29)
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xen-api/index.mjs:1006:24)
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xen-api/index.mjs:1040:14
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at Array.forEach (<anonymous>)
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xen-api/index.mjs:1030:12)
Feb 05 05:12:06 xoa-fs xo-server[32410]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202402050455/packages/xen-api/index.mjs:1203:14) {
Feb 05 05:12:06 xoa-fs xo-server[32410]:   code: 'IMPORT_ERROR',
Feb 05 05:12:06 xoa-fs xo-server[32410]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
Feb 05 05:12:06 xoa-fs xo-server[32410]:   call: undefined,
Feb 05 05:12:06 xoa-fs xo-server[32410]:   url: undefined,
Feb 05 05:12:06 xoa-fs xo-server[32410]:   task: task {
Feb 05 05:12:06 xoa-fs xo-server[32410]:     uuid: '0f812914-46c0-fe29-d563-1af7bca72d96',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     name_label: '[XO] VM import',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     name_description: '',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     allowed_operations: [],
Feb 05 05:12:06 xoa-fs xo-server[32410]:     current_operations: {},
Feb 05 05:12:06 xoa-fs xo-server[32410]:     created: '20240205T10:07:04Z',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     finished: '20240205T10:12:06Z',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     status: 'failure',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     resident_on: 'OpaqueRef:85a049dc-296e-4ef0-bdbc-82e2845ecd68',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     progress: 1,
Feb 05 05:12:06 xoa-fs xo-server[32410]:     type: '<none/>',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     result: '',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     error_info: [
Feb 05 05:12:06 xoa-fs xo-server[32410]:       'IMPORT_ERROR',
Feb 05 05:12:06 xoa-fs xo-server[32410]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
Feb 05 05:12:06 xoa-fs xo-server[32410]:     ],
Feb 05 05:12:06 xoa-fs xo-server[32410]:     other_config: { object_creation: 'complete' },
Feb 05 05:12:06 xoa-fs xo-server[32410]:     subtask_of: 'OpaqueRef:NULL',
Feb 05 05:12:06 xoa-fs xo-server[32410]:     subtasks: [],
Feb 05 05:12:06 xoa-fs xo-server[32410]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((process xapi)(filename ocaml/xapi/server_>
Feb 05 05:12:06 xoa-fs xo-server[32410]:   }
Feb 05 05:12:06 xoa-fs xo-server[32410]: }
Feb 05 05:12:06 xoa-fs xo-server[32410]: 2024-02-05T10:12:06.930Z xo:api WARN admin@admin.net | vm.importMultipleFromEsxi(...) [5m] =!> Error: no opaque ref found in  undefined

@florent tunnel is open!

@florent

I would be happy to test - but this is where that "hand holding" would come into play.

My VM doesn't have multiple vmdks, just the primary 16GB thin provisioned OS drive. Its actual size is only 4.77GB.

1vCPU
2GB RAM
16GB Disk
1 NIC
No dangling snapshots
powered off

truly nothing out of the ordinary.

@olivierlambert

[12:25 01] xoa:log$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            954M     0  954M   0% /dev
tmpfs           195M  528K  195M   1% /run
/dev/xvda1       19G  3.1G   15G  18% /
tmpfs           973M     0  973M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           195M     0  195M   0% /run/user/1000

This means you likely have a disk space problem on your Dom0. Can you do a df -h in your XCP-ng host?

Sure, it seems ok to me:

[12:21 roentgenium ~]# df -h
Filesystem                                                            Size  Used Avail Use% Mounted on
devtmpfs                                                              2.0G   28K  2.0G   1% /dev
tmpfs                                                                 2.1G  344K  2.1G   1% /dev/shm
tmpfs                                                                 2.1G  9.4M  2.0G   1% /run
tmpfs                                                                 2.1G     0  2.1G   0% /sys/fs/cgroup
/dev/sda1                                                              18G  2.9G   14G  18% /
xenstore                                                              2.1G     0  2.1G   0% /var/lib/xenstored
/dev/sda5                                                             3.9G  302M  3.4G   9% /var/log
10.96.25.20:/mnt/seaborgium/XCP/41b662ff-93bd-c386-9dd0-5d08daa3f4e3   39T  1.0M   39T   1% /run/sr-mount/41b662ff-93bd-c386-9dd0-5d08daa3f4e3
10.96.22.19:/mnt/array1/Isos                                           29T   14T   16T  47% /run/sr-mount/f60530ef-04e2-daf4-36df-c24527814e97
10.96.35.31:/mnt/strontium/XCP/50d8f945-8ae4-dd87-0149-e6054a10d51f    52T   53G   52T   1% /run/sr-mount/50d8f945-8ae4-dd87-0149-e6054a10d51f
tmpfs                                                                 412M     0  412M   0% /run/user/0

I'm running the default XOA image that was auto-deployed from my XCP-ng node on 'latest' (obv) - so it's debian.

Here's the error block from journalctl:

Feb 01 11:50:18 xoa xo-server[107155]: 2024-02-01T16:50:18.569Z xo:xo-server WARN possibly unhandled rejection {
Feb 01 11:50:18 xoa xo-server[107155]:   error: Error: already finalized or destroyed
Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51)
Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6)
Feb 01 11:50:18 xoa xo-server[107155]:       at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9)
Feb 01 11:50:18 xoa xo-server[107155]:       at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18)
Feb 01 11:50:18 xoa xo-server[107155]:       at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10)
Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10)
Feb 01 11:50:18 xoa xo-server[107155]:       at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22)
Feb 01 11:50:18 xoa xo-server[107155]: }
Feb 01 11:50:20 xoa xo-server[107155]: root@10.96.22.111 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
Feb 01 11:50:20 xoa xo-server[107155]:     at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Feb 01 11:50:20 xoa xo-server[107155]:     at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29)
Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24)
Feb 01 11:50:20 xoa xo-server[107155]:     at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14
Feb 01 11:50:20 xoa xo-server[107155]:     at Array.forEach (<anonymous>)
Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12)
Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) {
Feb 01 11:50:20 xoa xo-server[107155]:   code: 'IMPORT_ERROR',
Feb 01 11:50:20 xoa xo-server[107155]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
Feb 01 11:50:20 xoa xo-server[107155]:   call: undefined,
Feb 01 11:50:20 xoa xo-server[107155]:   url: undefined,
Feb 01 11:50:20 xoa xo-server[107155]:   task: task {
Feb 01 11:50:20 xoa xo-server[107155]:     uuid: 'd4267c54-3480-a7fe-4d6d-6cc18eef1dc9',
Feb 01 11:50:20 xoa xo-server[107155]:     name_label: '[XO] VM import',
Feb 01 11:50:20 xoa xo-server[107155]:     name_description: '',
Feb 01 11:50:20 xoa xo-server[107155]:     allowed_operations: [],
Feb 01 11:50:20 xoa xo-server[107155]:     current_operations: {},
Feb 01 11:50:20 xoa xo-server[107155]:     created: '20240201T16:44:17Z',
Feb 01 11:50:20 xoa xo-server[107155]:     finished: '20240201T16:50:23Z',
Feb 01 11:50:20 xoa xo-server[107155]:     status: 'failure',
Feb 01 11:50:20 xoa xo-server[107155]:     resident_on: 'OpaqueRef:85a049dc-296e-4ef0-bdbc-82e2845ecd68',
Feb 01 11:50:20 xoa xo-server[107155]:     progress: 1,
Feb 01 11:50:20 xoa xo-server[107155]:     type: '<none/>',
Feb 01 11:50:20 xoa xo-server[107155]:     result: '',
Feb 01 11:50:20 xoa xo-server[107155]:     error_info: [
Feb 01 11:50:20 xoa xo-server[107155]:       'IMPORT_ERROR',
Feb 01 11:50:20 xoa xo-server[107155]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
Feb 01 11:50:20 xoa xo-server[107155]:     ],
Feb 01 11:50:20 xoa xo-server[107155]:     other_config: { object_creation: 'complete' },
Feb 01 11:50:20 xoa xo-server[107155]:     subtask_of: 'OpaqueRef:NULL',
Feb 01 11:50:20 xoa xo-server[107155]:     subtasks: [],
Feb 01 11:50:20 xoa xo-server[107155]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((proces>
Feb 01 11:50:20 xoa xo-server[107155]:   }
Feb 01 11:50:20 xoa xo-server[107155]: }
Feb 01 11:50:20 xoa xo-server[107155]: 2024-02-01T16:50:20.571Z xo:api WARN admin | vm.importMultipleFromEsxi(...) [6m] =!> Error: no opaque ref found

Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?

No? The VM doesn't show up in XOA if that's what you're asking.

I might need a little hand-holding on doing this part as I'm pretty new to the platform.

@florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.

So I was testing importing from vsphere and it had been going pretty well (a little slow), and then I saw the news blast with the improvements in 5.91 so naturally I excitedly upgraded. Now it's failing with the 'no opague ref found'. I've seen this in the forums but couldn't find any solution to it. Nothing in the environment really changed except for the XOA update.

vm.importMultipleFromEsxi
{
  "concurrency": 2,
  "host": "vsphere.nest.local",
  "network": "7f7d2fcc-c78b-b1c9-101a-0ca9570e3462",
  "password": "* obfuscated *",
  "sr": "50d8f945-8ae4-dd87-0149-e6054a10d51f",
  "sslVerify": false,
  "stopOnError": true,
  "stopSource": true,
  "user": "administrator@vsphere.local",
  "vms": [
    "vm-2427"
  ]
}
{
  "succeeded": {},
  "message": "no opaque ref found",
  "name": "Error",
  "stack": "Error: no opaque ref found
    at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17)
    at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21
    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22)
    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)"
}

Any ideas on what might be the problem?
vSphere uses iSCSI datastores and XCP-ng is connecting to the SR with NFS.