FYI, our approach with vectura is only temporary, we have other plans in few months to get rid of this potential cat & mouse game Broadcom could do by slightly change their protocol (without even talking about the potential legal risk with it).

@frb94 Just leaving it on 24/7 winter is here now or so we thought (its 75f outside today) . If there is any update on this i would try again. but as it seems you are having the same issue i did then, it does not appear to be fixed.

@fred974 Hello, Could you please provide the logs produced by XenClean after its execution? If your VM couldn't boot after XenClean, XenBootFix executed from a recovery environment should get your VM working again. Instructions are available here: https://docs.xcp-ng.org/troubleshooting/windows-pv-tools/#windows-fails-to-boot-hangs-inaccessible_boot_device

@acebmxer Thank you.

Hello @ludalm13, Thanks for your contribution and sorry for not giving an answer earlier. We initially planned to review your PR while doing some additional work on the audit plugin, but our planning evolved and that work was postponed. We'll review your PR in the following days.

@poddingue said: While you experiment, do the failing VMs have something in common? The same source SR, for example, or a mirror target keeping a different retention than the source. There are two backup jobs that are failing. One from Ark (local) to Synology, and one from Ark to S3. It seems to be the same VMs failing each night, but interestingly, there are some that fail to Synology but succeed to S3 and vice-versa. There are 9 VMs that fail for both. They do not all share the same SR. The backups on Ark are coming from several delta jobs that run daily, weekly, and monthly. Retention are different for all those.

@poddingue Thanks for looking into this! I checked the bulk "Reconnect to all hosts" action as well, and you're right — it doesn't work either. It only works if I select a single SR, open its page, and use the "Reconnect to all hosts" action from there.

@zorro So with purge off, Veeam stayed clean for a few nights, which fits what I saw in the lab test. If I read the docs right, the cost is that XO's reference snapshot keeps its data on the SR again, which you'd notice more on thick-provisioned storage: https://docs.xen-orchestra.com/backups-and-dr/backup-types/incremental_backups#purge-snapshot-data-cbt. Storage hasn't said yet whether purge breaking another tool's chain is expected, so I'd still call this a workaround rather than the answer. If a CBT warning comes back when you add the other VMs, the Veeam task id and the time would probably help them. And keeping both makes sense to me, since Veeam doesn't do the cross-site replication you need XO for.

@yannsionneau I don't use Ubuntu, but I have a few linux vm's ranging from Debian 13-14 to RHEL 10 and in all of them the passthrough works fine.

Thanks @cocoon, and I owe a correction: when I said in https://xcp-ng.org/forum/post/104865 that 6.4 would make these alerts disappear for good, I overstated it. If I'm reading the code right this time, the XO server sends a ping every 30 seconds stamped with its own clock (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/rest-api/src/events/event.class.mts), and the browser compares that stamp with its own clock and shows the banner once the gap passes 40 seconds (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/web-core/lib/packages/remote-resource/sse.store.ts). So https://github.com/vatesfr/xen-orchestra/pull/9681 widened the margin from 32 to 40 seconds, but a browser whose clock runs ahead of the XO server by more than about 10 seconds would still see it. That might be why @GregBinSD saw it on the Windows PC and not on the tablet, though that's only a guess on my part. Comparing two different clocks looks fragile to me, but @Team-XO-Frontend will know far better than I do whether there's a reason it's done that way.

Thanks for the information, @ph7 . That's a typo, for sure.

@nick.lloyd that's something out of this plugin, that's ceph native. most clusters should run on msgr2 native. which ceph versions are you running?

I'm also investigating on a solution to make XCP-ng opportunistically resize device BAR; in case the firmware didn't do it (or has no support for it).

@irtaza9 What do you see if you run top or xtop, or "xe task-list" when you start the process?

Just as an update on this. Did get the migrations working. Still haven't gotten the backups working. Pool is still stable otherwise. Have managed to get backups by using Veeam so it's not to much of a problem.

Thanks @jr-m4, reporting the small ones is useful too. From what I can see in the code, the VM view of the restore page has that refresh button and the Metadata view only has a link back, so I think your screenshots show a real gap and not a glitch on your side. I honestly don't know whether XO 5 still takes this kind of small UI fix now that most of the work goes into XO 6, so I'll leave that to @Team-XO-Frontend to decide.