Power on function

dsiminiuk

Async.host.power_on  (on xcpnghost1) 0%

Persists in the web UI Tasks panel no matter what I do.
Restarted toolstack,
Shut down the entire environment and restarted.
No way to cancel it or delete it.

olivierlambert

Are you sure it's not a XO refresh issue?

dsiminiuk

@olivierlambert It's been there since I played with power on function that failed.

Where is this task event stored? On the master? On XO?

How can I query it from a command line?

olivierlambert

xe task-list on any pool member

dsiminiuk

@olivierlambert said in Power on function:

xe task-list

Oh, that was too easy. There it is.

uuid ( RO)                : e1896255-fbec-76d6-c2f4-9c2829c1f72b
          name-label ( RO): Async.host.power_on
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 0.000

xe task-cancel uuid=e1896255-fbec-76d6-c2f4-9c2829c1f72b

Doesn't cancel it.

olivierlambert

Okay and a toolstack restart on the master doesn't clean it?

dsiminiuk

@olivierlambert It did now but never did before

I wonder if it is because the same host the command was run against was the master at the time and is now. Perhaps server 2 as master previously failed when the command was on server 1.

Makes no sense to me. All good for now. Thank you.

dsiminiuk

@olivierlambert Nope, I didn't wait long enough. It's back.

uuid ( RO)                : 4d1971f4-4895-78c8-2856-d679d457b75e
          name-label ( RO): Async.host.power_on
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 0.000

olivierlambert

It's not the same task (new UUID)

dsiminiuk

@olivierlambert correct, which tells me it has been added again by something else. XO? Where I initated the command originally?

olivierlambert

You can check who did this in the audit.log file of your master

dsiminiuk

@olivierlambert

Jul  8 10:58:09 xcpng01 xapi: [20200708T15:58:09.775Z|audit||104166 INET :::80|task.destroy D:b3c0abbbd938|audit] ('trackid=68849bbe6e0a22d9b0b024c21c2b6e36' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.host.power_on' '3450e4eb-9a11-24ba-5e0c-699ff352028b' 'OpaqueRef:99fd01c3-535d-4334-bb92-c4cb4f84f16c')))

Jul  8 11:51:16 xcpng01 xapi: [20200708T16:51:16.956Z|audit||108793 UNIX /var/lib/xcp/xapi|task.cancel D:05f4e18101a6|audit] ('trackid=e49c5c3a24fed047e0aa1a3ed7b47ab3' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.cancel' (('task' 'Async.host.power_on' 'e1896255-fbec-76d6-c2f4-9c2829c1f72b' 'OpaqueRef:083c7658-c1e5-46d7-bf9c-3b809be635d6')))

olivierlambert

This is for task destroy, not a task creation.

dsiminiuk

@olivierlambert it is all I see from:

fgrep Async.host.power_on /var/log/audit.log

olivierlambert

Grep on host.power_on to see if it's better.

dsiminiuk

@olivierlambert

fgrep host.power_on audit.log.1 | fgrep -v destroy | fgrep -v cancel

Jul  7 23:58:02 xcpng01 xapi: [20200708T04:58:02.118Z|audit||36922 |Async.host.power_on R:0af8abcf2f46|audit] ('trackid=68849bbe6e0a22d9b0b024c21c2b6e36' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:INTERNAL_ERROR: [ (Failure \"The host failed to power on.\") ]' 'API' 'host.power_on' (('host' 'xcpng02' '1139bfbc-c487-4311-8206-bbffe96ed6b1' 'OpaqueRef:426e950f-f926-490f-b598-328314aafbdc')))

Jul  8 00:58:02 xcpng01 xapi: [20200708T05:58:02.803Z|audit||42462 |Async.host.power_on R:2733002e2e58|audit] ('trackid=68849bbe6e0a22d9b0b024c21c2b6e36' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:INTERNAL_ERROR: [ (Failure \"The host failed to power on.\") ]' 'API' 'host.power_on' (('host' 'xcpng02' '1139bfbc-c487-4311-8206-bbffe96ed6b1' 'OpaqueRef:426e950f-f926-490f-b598-328314aafbdc')))

Jul  8 01:58:03 xcpng01 xapi: [20200708T06:58:03.488Z|audit||48005 |Async.host.power_on R:e74dd3d29556|audit] ('trackid=68849bbe6e0a22d9b0b024c21c2b6e36' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:INTERNAL_ERROR: [ (Failure \"The host failed to power on.\") ]' 'API' 'host.power_on' (('host' 'xcpng02' '1139bfbc-c487-4311-8206-bbffe96ed6b1' 'OpaqueRef:426e950f-f926-490f-b598-328314aafbdc')))

Jul  8 02:58:04 xcpng01 xapi: [20200708T07:58:04.229Z|audit||53358 |Async.host.power_on R:e730dacdf348|audit] ('trackid=68849bbe6e0a22d9b0b024c21c2b6e36' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:INTERNAL_ERROR: [ (Failure \"The host failed to power on.\") ]' 'API' 'host.power_on' (('host' 'xcpng02' '1139bfbc-c487-4311-8206-bbffe96ed6b1' 'OpaqueRef:426e950f-f926-490f-b598-328314aafbdc')))

Jul  8 03:58:04 xcpng01 xapi: [20200708T08:58:04.897Z|audit||58846 |Async.host.power_on R:19f454d900b0|audit] ('trackid=68849bbe6e0a22d9b0b024c21c2b6e36' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:INTERNAL_ERROR: [ (Failure \"The host failed to power on.\") ]' 'API' 'host.power_on' (('host' 'xcpng02' '1139bfbc-c487-4311-8206-bbffe96ed6b1' 'OpaqueRef:426e950f-f926-490f-b598-328314aafbdc')))

Now going back in time to audit.log.[2-7].gz

olivierlambert

It's possible that XAPI recorder the host.power_on and loop on trying to do that, but to be fair, I never saw that before

Can you power on that host to see if it stops?

dsiminiuk

@olivierlambert

Going back as far as I can to audit.log.7.gz only takes me back to June 30 and this has been a problem for more than a month so I don't think I can find the original request.

dsiminiuk

@olivierlambert said in Power on function:

Can you power on that host to see if it stops?

I had the whole rack powered off yesterday, 2 - XCP-NG 8.1 servers (DL380 Gen9), 1 Freenas (DL380 Gen9) for iSCSI VDIs. It still came back.

olivierlambert

So now everything is back online but despite that you got the task?