Hi @MajorP93,

This PR is only about changing the way we delete old logs (linked to a bigger work of making backups use XO tasks instead of their own task system), it won't fix the issue discussed in this topic.

Hi @champagnecharly ,

On the XO side, it seems that this PCI has an empty string ID, which doesn't enable us to delete it.
We'll have to do some tests to find out how to prevent that.

We might have trouble reproducing the issue, so would you mind helping us with the tests?
You would need to add this piece of code on file xo-server/dist/xapi-object-to-xo.mjs before the line that start with if (isHvm) { (that should be near line 475)

    if ((_vm$attachedPcis = vm.attachedPcis) !== null && _vm$attachedPcis !== void 0 && _vm$attachedPcis.includes('')) {
      warn('Empty string PCI id:', otherConfig.pci);
    }

then restart xo-server and look at the output of journalctl, there should be some lines looking like: 2026-01-30T09:26:17.763Z xo:server:xapi-objects-to-xo WARN Empty string PCI id:

We just merged the delay: https://github.com/vatesfr/xen-orchestra/pull/9400

We increased it to 5s to have a security margin, as the optimal delay may not be the same on different configurations.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed fix(backup-reports): prevent succesful backups being sometimes repor… #9400

We're still carrying a bit of investigations to see if we can find the cause of the problem, but if we don't find it we'll add this delay.

Thanks @Pilow for the tests once again

Ok so 1s is slightly not enough, thanks for the update.

Thanks again @Pilow

I don't think the remotes being S3 changes something here.

@cbaguzman for information, I made some changes on vhd-cli so in the future we'll get a more explanatory error message when a command failed because we passed an incorrect argument: https://github.com/vatesfr/xen-orchestra/pull/9386

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed feat(vhd-cli): prevent using invalid options #9386

Hi @Pilow,

I've done some more testing and looked at the code, and I wasn't able to reproduce this behaviour once. It's also unclear to me why it can happen.

We may just add the delay as you did, but 10s is probably too long. Could you try to replace it by a 1s delay instead, and tell us if it's enough?

Hi @cbaguzman,

I tested on my own and I got the same result as you, but then I realized the AI you used both tricked us into thinking that the --chain was a valid option for the info command (it's not).

I removed this option and the command worked properly.
Can you try the same command without this option?

Hi @Pilow,

Thanks again for the feedback, I think now we have enough data to be sure it's indeed a race condition.

We noticed that the log you sent earlier in this topic is a backup job using a proxy. Could you tell if the backup jobs that ended up with a wrong status in the report were all using a proxy, or not all of them?

Thanks @Pilow for the report, I'll try to reproduce on my side to get a better undertanding of what's creates the fallback.

Thanks @Pilow for the tests.

We'll have to investigate this to fix it more properly than adding a ugly delay.

I agree, let's wait for more runs.

If it's indeed a race condition, we'll still have to figure out a better way to settle this than just adding delay

Hi @pilow,

Currently, I don't know what would cause this or why this would happen more frequently.

Could you test on your own to add some delay before sending the report, to see if it's indeed a race condition?

To do that, you just need to edit the file packages/xo-server-backup-reports/dist/index.js by adding these two lines:

const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
await delay(10000);

at the beginning of the _report function, like this:

  async _report(runJobId, {
    type
  } = {}, force) {
    const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
    await delay(10000);
    if (type === 'call') {
      return;
    }

then just restart xo-server.

We just merged a PR that changes the UI to allow offline backups for delta backups, disaster recovery and continuous replication: https://github.com/vatesfr/xen-orchestra/pull/9228

Now, if your delta backup makes offline backups, you can disable that without having to change the backup type.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed fix(xo-web): allow offline delta backups #9228

Hi @Forza,

I made a fix to prevent the "No new data to upload for this VM" message from appearing if it is only true for a part of the backup jobs of one VM.

It's available here: https://github.com/vatesfr/xen-orchestra/pull/9286

You can test it by switching branch if you're running XO from sources, otherwise you'll need to wait for the next end of month release.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed fix(backups): fix incorrect info on mirror backups #9286

@Forza

Thanks for the details, I managed to reproduce after a few trials.

I'll try to fix it and keep you informed.

@Forza

As the named are greyed-out, could you confirm that the delta backup job saves the backup to remotes srv04-incremental and srv12-incremental, and the mirror backup job copies the backups from srv12-incremental to srv04-incremental? (or at least confirm these are the same couple of remotes)

Also, could you tell me if the VM with which you got the "No new data to upload for this VM" message was backed up by your backup job "Incremental backup every 4 hours - 8 days retention"?

Thanks @Forza,
I'll try on my own with a similar configuration.

I confirm that this is the current behaviour, as @pilow reported here https://xcp-ng.org/forum/post/99446

We might change it in the future to make it better, but it won't be trivial to change.