The HA doesn't work

sixela

Hello,

We had a DOM0 crash last night and in the HA I have the machines restarted if possible but the vms have not restarted, here are the errors:

Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] sm_exec D:7818195f2572 failed with exception Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_46);[S();S(The VDI is not available [opterr=['HOST_OFFLINE', 'OpaqueRef:e67d5aed-ae13-497e-ac16-29882c317ef3']]);S()]]])
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] Raised Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_46);[S();S(The VDI is not available [opterr=['HOST_OFFLINE', 'OpaqueRef:e67d5aed-ae13-497e-ac16-29882c317ef3']]);S()]]])
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 1/8 xapi Raised at file ocaml/xapi/sm_exec.ml, line 377
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 3/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 95
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 7/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 8/8 xapi Called from file lib/backtrace.ml, line 177
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace]
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] VDI.activate D:7781cbf53aa1 failed with exception Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_46);[S();S(The VDI is not available [opterr=['HOST_OFFLINE', 'OpaqueRef:e67d5aed-ae13-497e-ac16-29882c317ef3']]);S()]]])
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] Raised Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_46);[S();S(The VDI is not available [opterr=['HOST_OFFLINE', 'OpaqueRef:e67d5aed-ae13-497e-ac16-29882c317ef3']]);S()]]])
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] 1/1 xapi Raised at file (Thread 14746299 has no backtrace table. Was with_backtraces called?, line 0
Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace]

I have the impression that the vdi remains locked on DOM0 until it restarts.

Because around 00:25 I rebooted by hand and it works.

I would have preferred the HA to do what was necessary.

Thanks for your help

Have a nice day.

Danp

Hi @sixela

Is the VDI sitting on shared or local storage?

Dan

sixela

@Danp shared with multiple host (LUN) iscsi

sixela

This post is deleted!

Danp

Have you confirmed that HA is enabled on the pool's Advanced tab? Also, did one of the other hosts automatically take the role of pool master?

sixela

@Danp yesss ha is enable, the master has not changed

Danp

Can you describe in more detail this dom0 crash? You should investigate in the logs for why HA didn't kick in and promote a new pool master.

sixela

@Danp Hello,

I think he crashed due to a hardware problem.

The ha started but I got the above error so I had to start the server by hand afterwards but the dom0 that had crashed was UP again

Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] Raised Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_46);[S();S(The VDI is not available [opterr=['HOST_OFFLINE', 'OpaqueRef:e67d5aed-ae13-497e-ac16-29882c317ef3']]]);S()]])

sixela

@Danp Hello,

In addition :

28 machines impacted, 15 left ok, and 13 with the error msg. the HA did try, but there was a problem.

Danp

@sixela There's likely more information in the logs that would explain why a new pool master wasn't designated.

sixela

@Danp I'm not talking about a new master in my problem... but that my vm's all restart with the ha on another host in the same cluster with the restart if possible parameter that it tries but it's still lock with the following error:

Feb 27 00:03:43DOM0 xapi: [error||14746299 ||backtrace] Raised Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_46);[S();S(The VDI is not available [opterr=['HOST_OFFLINE', 'OpaqueRef:e67d5aed-ae13-497e-ac16-29882c317ef3']]]);S()]])

Translated with DeepL.com (free version)

Danp

@sixela I understand. I'm not an expert on HA functionality, but I suspect that the new pool master would need to be designated as a first step. That is why I am suggesting that you investigate why this didn't automatically occur.

tjkreidl

How many hosts in your pool? For HA to work out of the box, you need at least three hosts in a pool. Also, are all your hosts properly time synchronized to the same time source?
They need to be very close in time to each other for HA to work properly. Note that when HA is first enabled on a given host, it has to be rebooted for HA to function.

sixela

@tjkreidl Hello,

We have 17 hosts in the pool and they are well synchronized with a time server

tjkreidl

@sixela Hmmm ... that SR backend error makes me wonder if the place where you designate HA info to be stored (the so-called "heartbeat SR") might be corrupted or such?

sixela

@tjkreidl Hello,

Didn't I get half of ok too?

28 machines impacted, 15 left ok, and 13 with the error msg