@CAPS hey, many thanks for the kind words but you know it's not only me at all, we're a team (or many teams ). Let me know if I can be of any assistance. Yes, I'd like to know what you miss? I've got ideas like more visibility and management for images, volumes, or docker compose, or docker instance not running on XO VMs, or having docker enabled VMs and containers in the tree view, list of containers in all VMs etc. What do you think?

@octopuss - this is a GREAT feature! I also have a small setup...a small 2 Host lab...but that's just enough to test a great deal of functions out. One of them I've tested is that. It's great. You have updates? Go to the Pool > Patches > and click that button. Done. That's literally it! One of several great features in XO imo.

Well, after a little more digging...I find that this is still not a 'clean' solution imo. I mean, if you're big into APIs....I guess. But, I'd like to see something intuitive in the webUI. Would like to see @julienxovates and @olivierlambert make this a UI feature, instead of a CLI function..or separate URL(swagger) function Thanks!

Hello all, Version 9.2.445 of the XCP-ng Windows Guest Tools has been released. Download the latest release here: https://github.com/xcp-ng/win-pv-drivers/releases This release brings important bug fixes to the Windows drivers and installer. Important note for users upgrading from 9.2.350: You must first install the XenTools-fix-9.2.351.msp hotfix to upgrade to 9.2.351 before installing 9.2.445. Otherwise, upgrades from 9.2.350 will be blocked. Reboot is not required after installing the hotfix. The installer downloads include a copy of XenClean and XenBootFix. It is not necessary to run XenClean when upgrading from previous versions of the XCP-ng Windows guest tools, unless prompted by the installer. Remember to carefully read XenClean docs before use. Note: Always download the latest XenClean/XenBootFix from https://github.com/xcp-ng/win-pv-drivers/releases . You can find them in the latest release ZIP/ISO archives. Before installing Thank you for using our Windows PV tools. Please carefully read the instructions below. It is not necessary to run XenClean when upgrading from previous versions of the XCP-ng Windows guest tools, unless prompted by the installer. Not compatible with the "Manage Citrix PV drivers via Windows Update" option. You must disable this option before installing. Changes since 9.2.385 NEW: Group Policy support (see the extras/admx directory in the source code tree). Supported policies include: Hide the tray icon Disable all features that allow controlling the VM from the host Improved: Better network performance. Improved: System reliability improvements on driver upgrade failure. On an upgrade failure, the installer will remove all Xen drivers to ensure that the system will keep operating, and to allow you to safely uninstall the existing driver package. Improved: Remove obsolete Windows Update devices and don't block installation because of them. Fixed: Fix further data corruption bugs during memory snapshots and backups. This driver version significantly narrows the corruption window. However, fully fixing all known corruption issues requires further work in other XCP-ng components. Please keep an eye on the XCP-ng blog for updates. Fixed: Fix loss of VBD connection after storage migration. Fixed: Improve reliability of the clipboard service. Fixed: Improve reliability of the VIF IP configuration feature. Fixed: Reliability fixes on VMs with many vCPUs (>64). Fixed: Many other driver reliability fixes. Help / Community support If you encounter installation/uninstallation errors, please try again with one of the following commands: For installing: msiexec.exe /i XenTools-x64.msi /l*vx install.log For uninstalling: msiexec.exe /x XenTools-x64.msi /l*vx uninstall.log Please include this log along with the file C:\Windows\INF\setupapi.dev.log in your bug report. These files will help us troubleshoot any installation issues. Additionally, please report any errors or BSOD you encounter during testing of this release. Your feedback is very appreciated.

@poddingue Yes, reads pretty much like the same error / root cause.

@poddingue Taking the time to do this it's the bare minium! Here you go: https://github.com/vatesfr/xen-orchestra/issues/10741

Thanks, that helps. Some VMs fail to the Synology but pass to S3, and some the other way round. So I think the backups on Ark are probably fine (they pass health checks, as you said at the start) and it has more to do with what each target already holds, but I could be wrong. I'd also flag the several delta jobs with different retentions all feeding Ark, I just don't know the mirror code well enough to say whether that matters. @Team-XO-Backend, posts 6 and 10 have the log and the pattern if someone gets to this one.

@mdm thanks for checking Reconnect. If I'm reading the code right, both actions have the same mistake, so one fix should cover both. You already wrote the steps out in your first post, so if you're up for it, an issue on https://github.com/vatesfr/xen-orchestra/issues would put it where the frontend team tracks their work. Their bug form asks for the XO version (or commit if you're on sources) plus your Node and XCP-ng versions, which I don't know for your setup.

Hello @ludalm13, Thanks for your contribution and sorry for not giving an answer earlier. We initially planned to review your PR while doing some additional work on the audit plugin, but our planning evolved and that work was postponed. We'll review your PR in the following days.

Just as an update on this. Did get the migrations working. Still haven't gotten the backups working. Pool is still stable otherwise. Have managed to get backups by using Veeam so it's not to much of a problem.

Thanks for the information, @ph7 . That's a typo, for sure.

@nick.lloyd that's something out of this plugin, that's ceph native. most clusters should run on msgr2 native. which ceph versions are you running?

@zorro So with purge off, Veeam stayed clean for a few nights, which fits what I saw in the lab test. If I read the docs right, the cost is that XO's reference snapshot keeps its data on the SR again, which you'd notice more on thick-provisioned storage: https://docs.xen-orchestra.com/backups-and-dr/backup-types/incremental_backups#purge-snapshot-data-cbt. Storage hasn't said yet whether purge breaking another tool's chain is expected, so I'd still call this a workaround rather than the answer. If a CBT warning comes back when you add the other VMs, the Veeam task id and the time would probably help them. And keeping both makes sense to me, since Veeam doesn't do the cross-site replication you need XO for.

@yannsionneau I don't use Ubuntu, but I have a few linux vm's ranging from Debian 13-14 to RHEL 10 and in all of them the passthrough works fine.

@fred974 Hello, Could you please provide the logs produced by XenClean after its execution? If your VM couldn't boot after XenClean, XenBootFix executed from a recovery environment should get your VM working again. Instructions are available here: https://docs.xcp-ng.org/troubleshooting/windows-pv-tools/#windows-fails-to-boot-hangs-inaccessible_boot_device

FYI, our approach with vectura is only temporary, we have other plans in few months to get rid of this potential cat & mouse game Broadcom could do by slightly change their protocol (without even talking about the potential legal risk with it).