A lot of "host.getMdadmHealth" errors in logs

puffymob

Re: Lots of "host.getMdadmHealth" Failure Logs

as described in the related post, I see a lot of errors both in xoa web gui and journalctl xo-server logs:

Oct 23 09:27:01 xoa xo-server[83622]:     return func(*args, **kwds)
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/raid.py", line 21, in check_raid_pool
Oct 23 09:27:01 xoa xo-server[83622]:     result = run_command(['mdadm', '--detail', device])
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 70, in run_command
Oct 23 09:27:01 xoa xo-server[83622]:     raise subprocess.CalledProcessError(process.returncode, command, None)
Oct 23 09:27:01 xoa xo-server[83622]: CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
Oct 23 09:27:01 xoa xo-server[83622]: )
Oct 23 09:27:01 xoa xo-server[83622]: 2025-10-23T07:27:01.619Z xo:api WARN admin@admin.net | host.getMdadmHealth(...) [229ms] =!> XapiError: -1(Command '['mdadm', '--detail', '/dev/md>
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 101, in wrapper
Oct 23 09:27:01 xoa xo-server[83622]:     return func(*args, **kwds)
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/raid.py", line 21, in check_raid_pool
Oct 23 09:27:01 xoa xo-server[83622]:     result = run_command(['mdadm', '--detail', device])
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 70, in run_command
Oct 23 09:27:01 xoa xo-server[83622]:     raise subprocess.CalledProcessError(process.returncode, command, None)
Oct 23 09:27:01 xoa xo-server[83622]: CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
Oct 23 09:27:01 xoa xo-server[83622]: )
Oct 23 09:27:01 xoa xo-server[83622]: 2025-10-23T07:27:01.640Z xo:api WARN admin@admin.net | host.getMdadmHealth(...) [248ms] =!> XapiError: -1(Command '['mdadm', '--detail', '/dev/md>
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 101, in wrapper
Oct 23 09:27:01 xoa xo-server[83622]:     return func(*args, **kwds)
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/raid.py", line 21, in check_raid_pool
Oct 23 09:27:01 xoa xo-server[83622]:     result = run_command(['mdadm', '--detail', device])
Oct 23 09:27:01 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 70, in run_command
Oct 23 09:27:01 xoa xo-server[83622]:     raise subprocess.CalledProcessError(process.returncode, command, None)
Oct 23 09:27:01 xoa xo-server[83622]: CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
Oct 23 09:27:01 xoa xo-server[83622]: )
Oct 23 09:27:04 xoa xo-server[83622]: 2025-10-23T07:27:04.669Z xo:api WARN admin@admin.net | host.getIpmiSensors(...) [71ms] =!> XapiError: XENAPI_MISSING_PLUGIN(ipmitool.py)
Oct 23 09:27:17 xoa xo-server[83622]: 2025-10-23T07:27:17.656Z xo:api WARN admin@admin.net | host.getIpmiSensors(...) [1ms] =!> XapiError: XENAPI_MISSING_PLUGIN(ipmitool.py)
Oct 23 09:27:20 xoa xo-server[83622]: 2025-10-23T07:27:20.554Z xo:api WARN admin@admin.net | host.getMdadmHealth(...) [1ms] =!> XapiError: -1(Command '['mdadm', '--detail', '/dev/md12>
Oct 23 09:27:20 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 101, in wrapper
Oct 23 09:27:20 xoa xo-server[83622]:     return func(*args, **kwds)
Oct 23 09:27:20 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/raid.py", line 21, in check_raid_pool
Oct 23 09:27:20 xoa xo-server[83622]:     result = run_command(['mdadm', '--detail', device])
Oct 23 09:27:20 xoa xo-server[83622]:   File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 70, in run_command
Oct 23 09:27:20 xoa xo-server[83622]:     raise subprocess.CalledProcessError(process.returncode, command, None)
Oct 23 09:27:20 xoa xo-server[83622]: CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
Oct 23 09:27:20 xoa xo-server[83622]: )

xoa web log from "Tasks":

{
  "id": "0mh36s2bu",
  "properties": {
    "method": "host.getMdadmHealth",
    "params": {
      "id": "c3a00fc0-4290-45b6-a18a-22311585cddd"
    },
    "name": "API call: host.getMdadmHealth",
    "userId": "7d716e60-e7c7-49a4-xxxx-3d0b78f8b1fd",
    "type": "api.call"
  },
  "start": 1761209695434,
  "status": "failure",
  "updatedAt": 1761209695669,
  "end": 1761209695669,
  "result": {
    "code": "-1",
    "params": [
      "Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1",
      "",
      "Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool\n    result = run_command(['mdadm', '--detail', device])\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command\n    raise subprocess.CalledProcessError(process.returncode, command, None)\nCalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1\n"
    ],
    "task": {
      "uuid": "042754c3-5c3d-c42c-7783-122fd342bd5a",
      "name_label": "Async.host.call_plugin",
      "name_description": "",
      "allowed_operations": [],
      "current_operations": {},
      "created": "20251023T08:54:55Z",
      "finished": "20251023T08:54:55Z",
      "status": "failure",
      "resident_on": "OpaqueRef:fb8b1171-21df-4c6f-a688-fd128550a4a0",
      "progress": 1,
      "type": "<none/>",
      "result": "",
      "error_info": [
        "-1",
        "Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1",
        "",
        "Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool\n    result = run_command(['mdadm', '--detail', device])\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command\n    raise subprocess.CalledProcessError(process.returncode, command, None)\nCalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1\n"
      ],
      "other_config": {},
      "subtask_of": "OpaqueRef:NULL",
      "subtasks": [],
      "backtrace": "(((process xapi)(filename ocaml/xapi/rbac.ml)(line 233))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 104)))"
    },
    "message": "-1(Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1, , Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool\n    result = run_command(['mdadm', '--detail', device])\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command\n    raise subprocess.CalledProcessError(process.returncode, command, None)\nCalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1\n)",
    "name": "XapiError",
    "stack": "XapiError: -1(Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1, , Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool\n    result = run_command(['mdadm', '--detail', device])\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command\n    raise subprocess.CalledProcessError(process.returncode, command, None)\nCalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1\n)\n    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1073:24)\n    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1107:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1097:12)\n    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1270:14)"
  }
}

I have hardware RAID5 from 5 SSDs on Dell PERC H965i controller and never used software RAID.
This error started to be noticeable since Feb 2025 and until now it is still there.
During that period of time I more than once updated XCP-ng 8.3 packages, updated XOA, updated firmware of the hardware, BIOS etc.

Also to note: this error is being recorded for for users with "admin" permissions in XOA, regular users doesnt seem to trigger it.

olivierlambert

Hi,

For getting efficient to help you, we need to know about the XO version you are using: XOA on stable or latest?

puffymob

@olivierlambert said in A lot of "host.getMdadmHealth" errors in logs:

Hi,

For getting efficient to help you, we need to know about the XO version you are using: XOA on stable or latest?

Hi, I am using XO built from sources.
Commit: af569

olivierlambert

Ping @mathieuRA about this