XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    [Bug?] Load Balancer: Performance mode fails to power on hosts previously shut down by Density mode

    Scheduled Pinned Locked Moved Xen Orchestra
    1 Posts 1 Posters 13 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      frb94
      last edited by

      [Bug?] Load Balancer: Performance mode fails to power on hosts previously shut down by Density mode

      Hi everyone,

      I'm experiencing a possible bug with the Xen Orchestra Load Balancer plugin when switching from Density to Performance mode.

      Density successfully consolidates VMs and shuts down an unused host, but switching back to Performance does not automatically power that host on, even though Wake-on-LAN is correctly configured and works manually.

      After investigating the source code, I suspect that the host filtering logic prevents Performance mode from detecting hosts that were disabled during shutdown.

      Environment

      • Xen Orchestra: Built from source
      • Build: (November 10, 2026) Xen Orchestra, commit 03e93 | Master, commit 83c3e
      • Plugin: xo-server-load-balancer
      • Storage: Shared storage, live VM migrations working
      • Power-on method: Wake-on-LAN

      The pool contains three hosts:

      Host Role Load Balancer configuration
      XCPNG1 Additional host Excluded from the plan
      XCPNG2 Pool Master Always running
      XCPNG3 Secondary host Managed by Load Balancer

      Xen Orchestra itself runs as a VM on the Pool Master (XCPNG2).

      Steps to reproduce

      1. Start XCPNG2 and XCPNG3.
      2. Set the Load Balancer plan to Performance.
      3. Plugin distributes VMs between both hosts (approximately 50/50).
      4. Switch the plan to Density.
      5. Density successfully migrates the VMs from XCPNG3 to XCPNG2 and automatically shuts down XCPNG3.
      6. Switch the same plan back to Performance.
      7. Save the configuration and wait for multiple Load Balancer execution cycles.

      Expected behavior

      According to the official Xen Orchestra Load Balancing documentation, Performance mode should automatically power on halted hosts when a power-on mode is configured.

      XCPNG3 should therefore power on automatically.

      Actual behavior

      XCPNG3 remains powered off.

      No automatic power-on occurs.

      However, clicking the Play / Power on button for XCPNG3 in the Xen Orchestra UI successfully starts the host.

      The issue does not appear to be related to Wake-on-LAN connectivity.

      Load Balancer configuration

      • Mode: Performance
      • Performance behavior: Conservative
      • CPU critical threshold: 90%
      • RAM free critical threshold: 10000 MB
      • Migration cooldown: 1 minute
      • Maximum concurrent migrations: 1
      • Excluded hosts: XCPNG1

      The same pool and plan work correctly in Density mode.

      Investigation

      1. XAPI power-on mode is configured

      Running the following command on the Pool Master:

      xe host-list params=name-label,uuid,power-on-mode
      

      Returns:

      XCPNG3
      power-on-mode: wake-on-lan
      
      XCPNG1
      power-on-mode: wake-on-lan
      
      XCPNG2
      power-on-mode: wake-on-lan
      

      2. XCPNG3 is disabled after Density shuts it down

      Running:

      xe host-list \
        uuid=dd2cec0b-7849-412e-a040-e2826c35c6cc \
        params=name-label,enabled,power-on-mode
      

      Returns:

      name-label    : XCPNG3
      enabled       : false
      power-on-mode : wake-on-lan
      

      3. Performance mode attempts to power on halted hosts

      In:

      packages/xo-server-load-balancer/src/performance-plan.js

      The execute() method begins with:

      async execute() {
        // Try to power on a hosts set.
        try {
          await Promise.all(
            filter(
              this._getHosts({ powerState: 'Halted' }),
              host => host.powerOnMode !== ''
            ).map(host => {
              const { id } = host
              return this.xo.getXapi(id).powerOnHost(id)
            })
          )
        } catch (error) {
          console.error(error)
        }
      }
      

      This suggests that Performance should attempt to power on halted hosts independently of CPU/RAM thresholds or migration cooldown.

      4. Potential issue in _getHosts()

      In:

      packages/xo-server-load-balancer/src/plan.js

      Lines 256–265:

      _getHosts({ powerState = 'Running' } = {}) {
        return filter(
          this.xo.getObjects(),
          object =>
            object.type === 'host' &&
            includes(this._poolIds, object.$poolId) &&
            object.power_state === powerState &&
            !includes(this._excludedHosts, object.id) &&
            object.enabled === true
        )
      }
      
      object.enabled === true
      

      If a halted host has enabled: false, _getHosts({ powerState: 'Halted' }) appears to exclude it before Performance can call powerOnHost().

      5. Density uses shutdownHost()

      In:

      packages/xo-server-load-balancer/src/density-plan.js

      debug(`Shutdown Host (${fmtSrcHost}).`)
      
      try {
        await xapiSrc.shutdownHost(srcHost.id)
      } catch (error) {
        debug(`Unable to shutdown Host (${fmtSrcHost}).`, { error })
      }
      

      The underlying implementation in:

      packages/xo-server/src/xapi/index.mjs

      Contains:

      async shutdownHost(hostId, { force = false, bypassEvacuate = false } = {}) {
        const host = this.getObject(hostId)
        if (bypassEvacuate) {
          await this.call('host.disable', host.$ref)
        } else {
          await this.clearHost(host, force)
        }
        await this.callAsync('host.shutdown', host.$ref)
      }
      

      This is consistent with the observation that XCPNG3 ends up with enabled: false after being shut down by Density.

      Suspected root cause

      My hypothesis is:

      1. Density evacuates XCPNG3.
      2. Density calls shutdownHost().
      3. XCPNG3 becomes halted and disabled (enabled: false).
      4. Performance calls _getHosts({ powerState: 'Halted' }).
      5. _getHosts() filters out XCPNG3 because enabled !== true.
      6. powerOnHost() is never called for XCPNG3.

      This would explain why manual power-on through the Xen Orchestra UI works while automatic power-on through the Load Balancer does not.

      Possible fix

      Would it make sense to allow disabled hosts to be returned by _getHosts() only when searching for halted hosts to power on?

      For example, the condition:

      object.enabled === true
      

      could potentially become:

      (powerState === 'Halted' || object.enabled === true)
      

      Questions

      1. Is this a known issue in the current Load Balancer implementation?
      2. Is enabled: false expected after Density shuts down a host?
      3. Should Performance be able to restart a host in this state?
      4. Is the enabled === true filter intentional for halted hosts?
      5. Is there an existing workaround or an upstream fix?

      The goal is to use Density at night to consolidate VMs and shut down unused hosts, then switch to Performance during the day to bring those hosts back online automatically.

      Thanks in advance for any insights!

      1 Reply Last reply
      Reply Quote 0

      Hello! It looks like you're interested in this conversation, but you don't have an account yet.

      Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

      With your input, this post could be even better 💗

      Register Login
      • First post
        Last post