@olivierlambert Thank you! Updated to latest version of Xen Orchestra, commit b0e2e, and started backup again. Will update once it finishes. One thing that scares me is that now XOA is saying that XVA might be broken.

Hello @olivierlambert, sure!
Xen Orchestra, commit fd2c7

Hello,
I am experiencing the issue that is similiar to this one from 2021.

3 out of 21 VM backup fails every time with the following warning:

the writer DeltaBackupWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution.

and error:

Error: Lock file is already being held

This is a backup log for the first job

On the same job 2 hours later I got the same errors, but on different VMs. Backup log

It is always same 3 VMs that are failing during the first Job and then same 2 VMs during the second one

I am not sure if there is a specific issue or I have to delete the orphaned logs.

xo-server 5.113.0
xo-web 5.116.1

@olivierlambert Thank you!

@olivierlambert

By looking into the source of the plugin, it looks like all the currently implemented behavior is observed.

It calculates/simulates the best plan, moves VMs and tries to shut down the host. Nothing about waking the hosts up.

And that makes sense, as the feature of spinning hosts back up should be implemented elsewhere (in context of density plan) - when starting VM, instead of providing the error message "No hosts that are able to run this VM are online", there should/could/may be:

1. Evaluation of which host is the smallest one which would be able to host the desired VM (partially implemented in density-plan) and WoL/iDrac/iLO packet send to it(implemented in performance plan)
2. Some timeout/waiting for boot pop-up/dialog/having this thing in queue
3. After set time (let's say 5 min), call the currently implemented logic of starting VM in a pool.

P.S. Would be nice to be able to set, how often the plan is evaluated (currently 1 minute), so until "spinning hosts back up" functionality is implemented, at least host started manually won't be shut down by plan immediately.

@olivierlambert said in Xen Orchestra Load Balancer - turning on hosts:

Anyway, back to the topic, I wonder if the behavior is different between NFS and SMB shared SR I do remember seeing those limitation with "half powered on" pool, but I don't remember which operations exactly.

State right now is that SR is "Broken" because it is Unplugged from hosts... because, you know... the hosts are down.

Snapshotting works, migration works as well (when hosts in pool are down). Also, IIRC there are no issues when hosts in a pool are down and VMs are hosted on the SR storage. That is something that happened quite often in many previous projects.

So my issue with LB plugin is deterministic way of knowing under which conditions will it spin hosts back up.

@olivierlambert said in Xen Orchestra Load Balancer - turning on hosts:

As it will be possible with XO Lite

That is awesome, no more Windows for me

Live migrate VM from one host to another? That was working yesterday, as long as I exclude the host from LB. I can do that if you come up with a test-case

You may be allergic to XCP-ng, but sometimes it is the single reason to have windows installed There are some things that are easier to find (like WoL settings, I found it by mistake in XCP-ng, no idea where it is in Orchestra), but most important thing is that Center is outside the pool, so in case of screw-up I can control the pool and start Orchestra

@olivierlambert I see what you did there

1. My mistake - currently I use SMB shared storage, not NFS.

2. Booted up the pool, waited for LB to shut down everything except master. Then decided to snapshot the XOA itself using XCP-ng center.
   

No issues whatsoever. Just disks, with memory, everything is fine. The XOA is on SMB SR.

3. Snapshoted the XOA using... XOA.
   

Also no issues.

Well, the hosts are shutting down with no issues.

What is happening is that usually LB will migrate all the VMs to the pool master and then shut down all other hosts. My issue is with LB sending (or in my case not sending) WoL packets when I try to start/deploy new VM that doesn't fit on pool master anymore.

@olivierlambert thanks!

I, unfortunately, use shared NFS storage to store all VMs across the pool, as it is simpler and makes migration between hosts way faster.

So to work correctly, that would require dividing current pool into chunks for each host? And if I do so, under which condition will LB spin-up host back online?

Also, any plans to switch from SMAPIv1 to something else, so that use-case is doable?