Hi @acebmxer,

I've made some tests with a small infrastructure, which helped me understand the behaviour you encounter.

With the performance plan, the load balancer can trigger migrations in the following cases:

• to better satisfy affinity or anti-affinity constraints
• if a host has a memory or CPU usage exceeds a threshold (85% of the CPU critical threshold, of 1.2 times the free memory critical threshold)
• with vCPU balancing behaviour, if the vCPU/CPU ratio differs too much from one host to another AND at least one host has more vCPUs than CPUs
• with preventive behaviour, if CPU usage differs too much from one host to another AND at least one host has more than 25% CPU usage

After a host restart, your VMs will be unevenly distributed, but this will not trigger a migration if there are no anti-affinity constraints to satisfy, if no memory or CPU usage thresholds are exceeded, and if no host has more CPUs than vCPUs.

If you want migrations to happen after a host restart, you should probably try using the "preventive" behaviour, which can trigger migrations even if thresholds are not reached. However it's based on CPU usage, so if your VMs use a lot of memory but don't use much CPU, this might not be ideal as well.

We've received very few feedback about the "preventive" behaviour, so we'd be happy to have yours.

As we said before, lowering the critical thresholds might also be a solution, but I think it will make the load balancer less effective if you encounter heavy load a some point.

@Greg_E The RPU is supposed to disable the load balancer, but it's possible that when the load balancer restarts at the end of the RPU, it takes into account the host stats during the RPU, which may create some unexpected migrations.

We'll have to investigate on that. Thanks for the feedback.

@acebmxer at the moment I don't know what could cause this behaviour. I'll try to reproduce it during the following days.

I think setting the memory limit to half of the host RAM is fine if you don't expect too much load, but if you're getting a lot of RAM use on your hosts at some point, I'm not sure the load balancer will migrate VMs from a host at 90% RAM use to a host at 60% RAM use, as both exceed the limit.

Also, could you try again to reproduce the bug after changing the "performance plan behaviour" setting to conservative, to see if it changes something? The "vCPU balancing" mode is quite recent, so maybe there's some bug with it that we didn't discover yet.

Hi @acebmxer,

I think the reason for this is a feature we recently added that prevents VMs from moving back-and-forth between hosts. VMs now have a cooldown (default 30min) between 2 load-balancer-triggered migrations

Can you try to set the migration cooldown to 0 and tell us if it fixes this behaviour? (in the "Advanced" section of the load balancer configuration)

Hi @MajorP93,

This PR is only about changing the way we delete old logs (linked to a bigger work of making backups use XO tasks instead of their own task system), it won't fix the issue discussed in this topic.

Hi @champagnecharly ,

On the XO side, it seems that this PCI has an empty string ID, which doesn't enable us to delete it.
We'll have to do some tests to find out how to prevent that.

We might have trouble reproducing the issue, so would you mind helping us with the tests?
You would need to add this piece of code on file xo-server/dist/xapi-object-to-xo.mjs before the line that start with if (isHvm) { (that should be near line 475)

    if ((_vm$attachedPcis = vm.attachedPcis) !== null && _vm$attachedPcis !== void 0 && _vm$attachedPcis.includes('')) {
      warn('Empty string PCI id:', otherConfig.pci);
    }

then restart xo-server and look at the output of journalctl, there should be some lines looking like: 2026-01-30T09:26:17.763Z xo:server:xapi-objects-to-xo WARN Empty string PCI id:

We just merged the delay: https://github.com/vatesfr/xen-orchestra/pull/9400

We increased it to 5s to have a security margin, as the optimal delay may not be the same on different configurations.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed fix(backup-reports): prevent succesful backups being sometimes repor… #9400

We're still carrying a bit of investigations to see if we can find the cause of the problem, but if we don't find it we'll add this delay.

Thanks @Pilow for the tests once again

Ok so 1s is slightly not enough, thanks for the update.

Thanks again @Pilow

I don't think the remotes being S3 changes something here.

@cbaguzman for information, I made some changes on vhd-cli so in the future we'll get a more explanatory error message when a command failed because we passed an incorrect argument: https://github.com/vatesfr/xen-orchestra/pull/9386

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed feat(vhd-cli): prevent using invalid options #9386

Hi @Pilow,

I've done some more testing and looked at the code, and I wasn't able to reproduce this behaviour once. It's also unclear to me why it can happen.

We may just add the delay as you did, but 10s is probably too long. Could you try to replace it by a 1s delay instead, and tell us if it's enough?

Hi @cbaguzman,

I tested on my own and I got the same result as you, but then I realized the AI you used both tricked us into thinking that the --chain was a valid option for the info command (it's not).

I removed this option and the command worked properly.
Can you try the same command without this option?

Hi @Pilow,

Thanks again for the feedback, I think now we have enough data to be sure it's indeed a race condition.

We noticed that the log you sent earlier in this topic is a backup job using a proxy. Could you tell if the backup jobs that ended up with a wrong status in the report were all using a proxy, or not all of them?

Thanks @Pilow for the report, I'll try to reproduce on my side to get a better undertanding of what's creates the fallback.

Thanks @Pilow for the tests.

We'll have to investigate this to fix it more properly than adding a ugly delay.

I agree, let's wait for more runs.

If it's indeed a race condition, we'll still have to figure out a better way to settle this than just adding delay

Hi @pilow,

Currently, I don't know what would cause this or why this would happen more frequently.

Could you test on your own to add some delay before sending the report, to see if it's indeed a race condition?

To do that, you just need to edit the file packages/xo-server-backup-reports/dist/index.js by adding these two lines:

const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
await delay(10000);

at the beginning of the _report function, like this:

  async _report(runJobId, {
    type
  } = {}, force) {
    const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
    await delay(10000);
    if (type === 'call') {
      return;
    }

then just restart xo-server.

We just merged a PR that changes the UI to allow offline backups for delta backups, disaster recovery and continuous replication: https://github.com/vatesfr/xen-orchestra/pull/9228

Now, if your delta backup makes offline backups, you can disable that without having to change the backup type.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed fix(xo-web): allow offline delta backups #9228

Hi @Forza,

I made a fix to prevent the "No new data to upload for this VM" message from appearing if it is only true for a part of the backup jobs of one VM.

It's available here: https://github.com/vatesfr/xen-orchestra/pull/9286

You can test it by switching branch if you're running XO from sources, otherwise you'll need to wait for the next end of month release.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed fix(backups): fix incorrect info on mirror backups #9286