Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates

Great! I think this is really the best possible approach to make it clean

Hi,

The best approach is first to unplug the PBD (or disable the remote) while the storage is UP, and then shutdown the target/storage node itself.

The opposite is 100% causing issues due to the nature itself of a network storage.

You can use the REST or JSON-RPC API to disable the remote and then send the shutdown order. This way, no problem.

Thanks for the logs @ph7, that helps narrow it down.

Some context on what this error actually means: XO's XAPI client kills the connection if it receives nothing from the pool master for 5 minutes during the transfer. So the BodyTimeoutError on a metadata backup means the /pool/xmldbdump response from your master went silent. Either xapi stalled while producing the dump, or the connection died silently somewhere in between (a NAT/firewall/VPN dropping the connection state would look exactly like this). Note that your XO config backup succeeding right before doesn't rule out a network issue: the config backup only talks to the remote, it never touches the pool master.

To figure out which side is failing, could you check a few things?

1. On your backup remote, look at the file left behind by the failed run: xo-pool-metadata-backups/<scheduleId>/<poolUuid>/<date>/data. What size is it, compared to the same file from a successful run? If it's empty or tiny, the master stalled. If it looks complete, the data actually arrived and only the end-of-connection signal got lost, which would point at the network path.
2. On the pool master, grab /var/log/xensource.log around the failure time. There should be a task named "Export pool metadata" and the xmldbdump HTTP handler. We'd like to know if it completed on the xapi side.
3. Is there anything between your XO and the pool master, like NAT, a firewall or a VPN?
4. Does it fail every run, or intermittently?

@Tristis-Oris the same questions apply to your setup. The fact that it hits different pools one after the other makes me suspect something shared in the network path.

Hi,

Would be nice to have the entire log

Could it be possible that the master or the pool was not reachable at the time of the metadata backup?

Hi,

I think it's a known problem, let me ping @poddingue so he gives you a quick recap on the situation (or someone from the team storage)

Ping @julienXOVates

Ping @julienXOVates

Thanks, I made the feedback to the XO team, this will be fixed ASAP.

I made 2 PRs upstream, let me ping @Team-XAPI-Network about it.

• https://github.com/xapi-project/xen-api/pull/7145
• https://github.com/xapi-project/xen-api/pull/7146

olivierlambert opened this pull request in xapi-project/xen-api

closed xapi: report a clear error when host evacuation is blocked by unprotected VMs #7145

olivierlambert opened this pull request in xapi-project/xen-api

draft xapi: evacuate non-HA-protected VMs when no host failures are tolerated #7146

Hi,

It's related to an existing bug: https://github.com/xapi-project/xen-api/issues/4323

Quick answer so you can move forward:

1. Set the VMs' HA restart priority to "Restart" (ha_restart_priority = restart), then they're "protected" and get a real evacuation plan. In a 2-node pool with ha-host-failures-to-tolerate=0 this is somewhat cosmetic but it satisfies the partition check.
2. Or temporarily disable HA, evacuate / restart the host, re-enable HA. Cleanest for maintenance on a 2-node pool, since failures-to-tolerate is forced to 0 anyway.

I will see to accelerate for a fix upstream.

olivierlambert created this issue in xapi-project/xen-api

closed Mysterious failure in HA with HOST_NOT_ENOUGH_FREE_MEMORY #4323

For your storage question, it's fully explained in the doc: https://docs.xcp-ng.org/storage/#-how-to-modify-an-existing-sr-connection

And yes, it's planned to get the complete error visible in XO, sadly, it's not "obvious" since the error message isn't returned by XAPI when you try to start but by another method we need to call after it fails ("assert can be started here" from the top of my head).

Let me ping @julienXOvates

Start the VM with xe vm-start uuid=<VM UUID> and see the error message there.

Ping @ataxyanetwork to the rescue

Hi,

So it seems you are in a case you don't have a pool to avoid a VM reboot with your host reboot.

In that case, you need to have auto power on enabled to make sure after host reboot, they all start.

I'm not sure it's a great idea, because even 64 hosts is huge in terms of VMs and fallout if you have a problem on your pool DB. Even if you drastically improve the current mechanism, the impact of a problem pool wide is far bigger with 64 hosts than 24 for example.

It's more than purely tech, it's also a tech design/choice.

Even with existing customer with that many VMs or host, absolutely nobody will do a giant pool. And in the ROBO/edge world, it's mostly 2 or 3 machines per shop.

Even large DC deployments tend to use around 10 hosts per pool (a good sweetspot between convenience and fallout protection in case you have a problem on a pool).

I won't be against it obviously. But I'm not the one making the choice

There are many alternative solutions, not just us

Why, any hint in there that would be XCP-ng?