Lots of "host.getMdadmHealth" Failure Logs

DustyArmstrong

I'm getting tons of Mdadm errors from Xen Orchestra, but not really sure why.

host.getMdadmHealth
{
  "id": "d2de9e76-ffbf-4640-9d68-43178c7c4006"
}
{
  "code": "-1",
  "params": [
    "Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1",
    "",
    "Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool
    result = run_command(['mdadm', '--detail', device])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command
    raise subprocess.CalledProcessError(process.returncode, command, None)
CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
"
  ],
  "task": {
    "uuid": "34429da6-56ee-9b5c-c465-b0493920b3f4",
    "name_label": "Async.host.call_plugin",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250117T09:42:09Z",
    "finished": "20250117T09:42:09Z",
    "status": "failure",
    "resident_on": "OpaqueRef:f0015d71-0ac1-4a79-bf0d-3700f79ba394",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "-1",
      "Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1",
      "",
      "Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool
    result = run_command(['mdadm', '--detail', device])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command
    raise subprocess.CalledProcessError(process.returncode, command, None)
CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 8780))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "-1(Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1, , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool
    result = run_command(['mdadm', '--detail', device])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command
    raise subprocess.CalledProcessError(process.returncode, command, None)
CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
)",
  "name": "XapiError",
  "stack": "XapiError: -1(Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1, , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 101, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/raid.py\", line 21, in check_raid_pool
    result = run_command(['mdadm', '--detail', device])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 70, in run_command
    raise subprocess.CalledProcessError(process.returncode, command, None)
CalledProcessError: Command '['mdadm', '--detail', '/dev/md127']' returned non-zero exit status 1
)
    at Function.wrap (file:///home/node/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///home/node/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///home/node/xen-orchestra/packages/xen-api/index.mjs:1068:24)
    at file:///home/node/xen-orchestra/packages/xen-api/index.mjs:1102:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///home/node/xen-orchestra/packages/xen-api/index.mjs:1092:12)
    at Xapi._watchEvents (file:///home/node/xen-orchestra/packages/xen-api/index.mjs:1265:14)"
}

Neither host with ID 2de9e76-ffbf-4640-9d68-43178c7c4006 or f0015d71-0ac1-4a79-bf0d-3700f79ba394 are using a software RAID. It may be because I haven't updated the hosts in quite some time. There is no output on either host for cat /proc/mdstat.

Is there a way I can just turn off this check?

olivierlambert

Adding @stormi and @MathieuRA in the loop, it's not normal indeed.

stormi

What does mdadm --detail /dev/md127 output on either hosts?

DustyArmstrong

@stormi thanks for the reply, the output is (on both hosts):

mdadm: cannot open /dev/md127

I do have a 3rd host that does make use of a software RAID, but that also outputs nothing for /dev/md127.

I am updating the hosts today so it's possible they're just so far behind.

DustyArmstrong

Updated all my hosts but ended up with a bunch of stuck tasks for API host calls, didn't seem too healthy! It looks like they were stuck, kept seeing a host unhealthy power state repeatedly pop up and disappear.

I opted to select all tasks and delete, same with my logs (I monitor externally anyway) which appears to have resolved this for the moment. I no longer see these mdadm logs being generated and everything appears normal.