@poddingue Okay, we're back here. I was told I have a hung CIFS mount that no longer exists, so a host needs to reboot to clear it, and faulty XOSTOR resources. But I can't reboot the host until I can move VMs off. Is there any indication of how to repair the faulty XOSTOR resources so I can move the VMs and reboot the host? Thanks for taking a look.

@florent said: @jr-m4 you can export the heap memory of the nodeJS process by doing kill -SIGUSER2 <xoserverpid> onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment Do you have somewhere I can upload the heapsnapshot? (148MB) Ping @poddingue as well, for visibility

@coolsport00 Sorry about the VMW need for the Cisco product. SOunds like you have a number of constraints, finances being I'm sure one of them! At least you have time on your hands and the means to experiment. You may, als, end up with a number of different platforms to meet your needs. We ran both Sun Microsystems and Red Hat Linux and Microsoft WIndows servers, each taking on specific duties. It's far from ideal and probably not very cost-effective, but you do what you have to to get stuff to work.

@JB @christopher-petzel Thanks for your patience! We are glad it works well!

Pretty cool, @samuelolavo, thanks for clarifying!

@poddingue Hey, it's just fun building stuff, even if it ends up in the bit-bucket (as unfotunately that's the destiny of all things anyway). Happy Day

@dthenot Thanks - I'll be interested to hear about your findings. I'll hold off on submitting a report until I hear back.

We are fully aware of Broadcom's decision to stop providing the VDDK library to the cloud community or to reserve its download to some authorized partners and customers. As much as we regret this decision, we have to respect it. Because it impacts users who need to make backups, we are evaluating alternative options to address this situation. We will provide further guidance as this evaluation progresses but can't share more information for now.

@florent No explicit assigned migration or backup network set. Backups generate errors somewhat randomly. As one VM won't get the "error" one time. But on the next backup run it will. And the third time it might not. This even happens on VMs that are turned off, and therefore shouldn't have any changed bytes between backups. Host is reachable from XO Network has NBD enabled All VMs use VHD IPv4 Backup type is Delta

Thanks for the feedback, @dvinni !

Hello there, Thanks a lot for raising the question and offering ideas to move forward @caps ! I would like to say that I really like the AI crafted screenshot provided 🤩 But, to be clear, the status as of today regarding Docker or other containers integration in XO/XCP-ng is that there is an item in the DevOps Tools team roadmap. The item is just to look and define what could be done (example update xscontainer, do something totally new, ...). Unfortunately this spike has a low priority and is not yet scheduled. That being said, there is an active project, which is close to what you're asking. It's about a better integration of Kubernetes, including, cluster update, adding nodes, etc. More on this should come in the next months! ️ Also note that I am chatting with different people and teams in Vates about this thread and the Docker integration topic to see if we could do something. Also last thing is that, the community is always welcome to build on top of our open source softwares. I am pretty sure that if someone would contribute a Xen Orchestra plugin to integrate Docker we would welcome this, and we could give tips and guidance. Like @teddyastie said, I'm not sure having Docker, Podman, related features in the guest agent would be something our colleagues would merge. But maybe the docker daemon socket could be kind of shared between VMs (I'm not sure at all this is possible and if possible how to do it). In the meantime exposing the docker daemon over network (with restrictions) to Xen Orchestra would be quick'n easy and I think safe enough if done carefully.

@AtaxyaNetwork Thank you for sharing the latest version. It works very well except I'm getting error when trying to install any plugin. Did you get this as well ? [image: 1788936195627-d8c58b99-0330-4c85-830c-998b37004d59-image.jpeg]

v0.6.0 — log collection is in. This was the piece the first post said was next, and it works end to end now. A new Collect page runs a per-host collection as a background job: it pulls logs.tgz and the XAPI audit trail from Xen Orchestra, keeps the raw copies, and produces a redacted copy of each. On my own XCP-ng 8.3 pool that came to 434 MiB for the bundle and 769 MiB for the audit trail, 2.3 GiB stored across the five files, with 2,542,805 values masked — almost all of them session tokens. The download is streamed a megabyte at a time and never held in memory, and every chunk is a cancellation point, so Cancel actually stops a transfer rather than waiting it out. A failed or cancelled download deletes its partial file instead of leaving a half-written bundle sitting there looking like a collection. Both copies are kept and marked redacted — safe to send or raw — unmasked. Redaction is lossy, and the raw copy is the only thing that can answer "what did that used to say?" afterwards. Each collection writes the same redaction report the preview page produces, so you can see which rules fired — and which were switched off — before anything leaves the box. In the screenshot I had IPv4, MAC, UUIDs and hostnames turned off, and they show as off rather than as zero hits. That is deliberate: a partly-masked bundle should be obvious before you attach it to a ticket. There is also a retention policy. The newest n collections are kept whatever their age, and only what is left is judged on age — a long gap in collecting can't empty the store. It shows you what a cleanup would delete and how much space that returns before you press the button. Expect bugs. This is still very much a work in progress and it has only ever run against my own pool. If you try it in a different environment, assume things will break — please tell me what did. Known issues right now: The red "This connection uses a restricted account" warning on the Collect page shows for every connection, including an administrator one. It is a wrong condition in the template, not a real permission check — ignore it if your account is admin. The screenshot below has it, over a collection that succeeded. The Collect page says to expect "433 MB and 100 seconds per host". The 100 seconds is the download only; a full collection also fetches the audit trail and redacts both copies, which took 203 seconds on my pool. Budget three to four minutes. Two more things worth flagging if you try this. logs.tgz sends no Content-Length, so a reverse proxy that buffers will happily hand you a truncated archive with HTTP 200. Mine did. It now detects that and reports the bundle as cut short rather than as a protocol error, and repacks whatever did arrive instead of throwing the lot away. If you are behind Nginx Proxy Manager, proxy_buffering off; and proxy_max_temp_file_size 0; are what fixed the stall for me. The account requirement from the first post has not changed — the log download needs export:logs on host, and on the instance I measured that is only reachable via Administrator. Inventory still works fine on Read only. Next up: individual log categories pulled out of the cached bundle (~35 MB instead of 434 MiB), date ranges, and then findings. https://github.com/acebmxer/xcp_pulse [image: 1788812029125-screenshot_20260907_160937.png]

install_xen_orchestra v0.7.2 — twelve templates, and the builder can now use XO's own API Since the v0.5.0 post the template library has gone from one distro to twelve, and it picked up a second way of talking to the pool. Everything from v0.5.0 through v0.7.2, in one place. Templates: twelve buildable, three still to come v0.5.0 shipped with Debian 13 and I asked here which distros to do next. That list is now: Debian 12, 13 Ubuntu 22.04, 24.04, 26.04 LTS AlmaLinux 8, 9, 10 CentOS Stream 9, 10 Fedora 43, 44 Rocky Linux 8/9/10 are the only rows still marked Coming Soon... in the menu. Every one of these is a row in the catalogue table rather than new build code, which is what I said in the v0.5.0 post I was aiming for. Each image did get read individually rather than assumed to match its neighbour — default user, disk size, kernel type, partition table — because those differ inside a family. Ubuntu 22.04 expands to 2.2 GiB and 24.04 to 3.5 GiB, so copying one figure across the family would have inflated every VM cloned from the smaller one. Two things that had to be solved along the way and now apply to every image: qcow2 images import. Debian publishes raw; almost everyone else publishes qcow2, so the builder converts before importing. The shipped password actually works. The RHEL-family and Fedora images declare lock_passwd: True for their default user, and cloud-init re-locks that account on every boot — so the password the build set was already locked by the time a clone came up. The build now drops a file in /etc/cloud/cloud.cfg.d turning that off. Delete it once you've put your own SSH keys on. Key auth was never affected. Arrow keys in the template menu now skip the Coming Soon rows instead of stepping through them. The builder can now go through XO's API instead of SSH to dom0 Template building has always driven xe over SSH to the pool master, which means having the pool master's root password to hand. Everything it does that way, XO exposes over its own API — create the VM, import the disk, build the cloud-init drive — so v0.7.0 added that as a second path. Which one makes sense depends on where you run it from. If you installed with --deploy, this repo is sitting inside the XO VM and that's where you run --update from, so XO is on localhost and you already have an API token in xo-config.cfg for the pre-update task check. Same token, no root password. Run it from your workstation against a pool with no XO on it yet and none of that is true, so SSH stays. TEMPLATE_BUILD_METHOD picks: auto # default — API if it's reachable and authenticated, else SSH, and it says which and why api # API only, no fallback ssh # exactly what it did before The choice is made by a preflight check before anything gets created on the pool, so a fallback costs you nothing but a different password prompt. On the API path the image streams from its mirror straight into XO without being staged to a file, and XO builds the cloud-init drive itself — so no ISO writer needed on the machine you're running from. Two new config keys: XO_API_TOKEN and XO_URL. XO_API_TOKEN replaces XO_TASK_CHECK_TOKEN in name only — the old key is still read, so an existing config keeps working untouched. XO_URL can be left unset on the XO VM itself, and falls back to PUBLIC_URL if you've set that. --deploy fixes The appliance now gets the XCP-ng guest tools. A deployed VM had no guest agent at all, so the XO it was itself hosting couldn't report its IP, memory or disk usage, and couldn't shut it down cleanly. It ran fine while looking half-blank in the very interface it served. It now boots UEFI, not BIOS. Nothing was ever setting the firmware parameter, so a VM from --deploy and a VM cloned from a --build-templates template came out of the same script with different firmware. It's probed from the disk now, same as the builder does. The temporary deployment SSH key is actually removed. It wasn't being deleted, leaving a working key on the VM. Progress bars curl --progress-bar redraws the whole line every update and leaves the cursor sitting in it, which over SSH is a cursor visibly scribbling back and forth instead of a bar filling up. All four transfers now draw a bar that doesn't repaint, including the one that stages the image on the pool master — which is the longest step of a deploy and was the one people would actually be watching. Also Templates get sensible platform settings instead of inheriting them from the scaffolding: vga=std with 16 MiB video memory (the stock 4 MiB cirrus is what leaves the console at 640x480), viridian off (it's Hyper-V enlightenment, meant for Windows), and cores-per-socket matching the vCPU count. Checksum verification works for every origin now. AlmaLinux, CentOS Stream and Fedora would each have 404'd on the wrong filename and imported unverified with a warning. Debian 11 and Ubuntu 22.04 are deprecated with dates on them, and the menu says so rather than the row silently vanishing one day. The README's menu screenshots no longer need a horizontal scrollbar on GitHub. 353 unit tests, and CI smoke-tests the install on 10 distros. Repo: https://github.com/acebmxer/install_xen_orchestra Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md Templates doc: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

@olivierlambert I'm seeing a recurrence of the snapshot_of / hidden-disks issue after patching to xapi-26.1.16-1.2 (both hosts, rebooted). Unusual element this time: snapshot-fixer.py dry-run flags ~260 VDIs across the SR that all cite a single OpaqueRef which resolves to no VDI (xe vdi-list | grep <ref> returns nothing). Holding off on rewrite until I understand whether that single-dangling-ref pattern is safe, given the set appears to include base VDIs of legitimate snapshots. Full technical detail and questions posted on GitHub: https://github.com/xcp-ng/xcp/issues/844.

@poddingue I'm seeing a recurrence of the snapshot_of / hidden-disks issue after patching to xapi-26.1.16-1.2 (both hosts, rebooted). Unusual element this time: snapshot-fixer.py dry-run flags ~260 VDIs across the SR that all cite a single OpaqueRef which resolves to no VDI (xe vdi-list | grep <ref> returns nothing). Holding off on rewrite until I understand whether that single-dangling-ref pattern is safe, given the set appears to include base VDIs of legitimate snapshots. Full technical detail and questions posted on GitHub: https://github.com/xcp-ng/xcp/issues/844.

I have received a very long update back from Veeam on my backup issues.... There appears to be another user with similar setup / issue not sure if that user is the OP this post specifically... Hello, Thank you for your patience. The QA team has finished the analysis, and I am going to outline the details as below: 1. CBT Inconsistency Issue Whenever you see the Warning about CBT showing: 2026-08-16 17:37:27.459 00079 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 291af3d7-28c1-15a9-7f13-c6a1a12283e9 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. We can go to the previous run and see that we get reports that the CBT of the previous snapshot is inconsistent and thus removed by XCP: 2026-08-16 09:54:34.120 00004 ERROR | [XenBackupManager]: Failed to retain the data for the snapshot 5a17fb31-2616-4e72-a85c-e235883c8c91 Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 39c0f25f-e811-ee45-1b0b-e4a9ca9839ae (Async.VDI.data_destroy) failed: . VDI_NO_CBT_METADATA. OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed The team has been reviewing and testing, and this is what they see. We send the request for the snapshot to be created, and it is sent to the coordinator: xcp-ng-vyadytkn Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] vdi_clone: introduced VDI: OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed (5a17fb31-2616-4e72-a85c-e235883c8c91) CBT shows as open and reading fine: Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] ['/usr/sbin/cbt-util', 'set', '-n', '/var/run/sr-mount/7911c9c5-5f20-01e1-8b8d-39c6a98a2704/5a17fb31-2616-4e72-a85c-e235883c8c91.cbtlog', '-f', '1'] Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] pread SUCCESS But look at how on xcp-ng-host2 the CBT gets marked as inconsistent by XCP, even though it's open and reading: Aug 16 09:51:05 xcp-ng-host2 SM: [1683689][MainThread] Changed Block Tracking metadata is inconsistent for disk 5a17fb31-2616-4e72-a85c-e235883c8c91. The Breakdown: xcp-ng-vyadytkn was the coordinator—the one we talk to, who then passes everything around to the hosts. xcp-ng-host2 was the host that the VM resided on at that time. xcp-ng-host2 is marking the CBT as inconsistent and deleting the snapshot CBT log, which means we cannot reference it on the next run. We do not see anything else interacting with the CBT besides that host. This matches exactly what we see with another client running the same setup. The team successfully replicated the environment, which is configured as follows: VM storage is NFS. The VM is running on a host that is not the coordinator. They have been able to reproduce this behavior occasionally, and the working theory is that the VM host keeps its own tracking separate from the coordinator. Part of the backup process requires the VM host to issue a pause/resume via a process called tapdisk. When it resumes, it pushes a data cache (likely inside the NFS cache), overwriting the CBT reference held by the coordinator server. It acts as a race condition—whoever pushes the CBT data last wins. If the VM host pushes last, it breaks what the coordinator is sending. Next Steps for CBT: The team is working to raise this issue directly with Vates so they can address the race condition. We ask that you also open a Vates ticket if possible to help draw more attention to the bug. We are trying to find ways to code around this race condition in the future, but there are currently no ETAs or guarantees. 2. Synthetic Full Failures (Delilah_ArcFS01) In addition to the CBT bug, the team discovered a separate issue. Recently, the synthetic fulls for Delilah_ArcFS01 have been failing. This appears to be related to the NFS repository: [30.08.2026 00:43:22.856] <24> [0007] Error (1) Failed to execute full transform task [30.08.2026 00:43:22.856] <24> [0007] Error (1) Agent: Failed to process method {Transform.CompileFIB}: NfsFileEx was already stopped. File: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]] (Veeam.Backup.Common.CCppComponentException) [30.08.2026 00:43:22.856] <24> [0007] Error (1) in c++: Failed to execute command Command: READ, Offset: 2523136, Data size: 659456, Chunk size: 131072 [30.08.2026 00:43:22.856] <24> [0007] Error (1) in c++: Failed to read file: Offset: 2523136, Block size: 659456, File: Path: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]], Handle: [01000702080061030000000007fffd5f65840fb20000000000000000150061038a2f7c0b0400610379207c0b], Read chunk size: 131072, Write chunk size: 131072, Read only: true Because it continuously fails during the synthetic full, it eventually causes the snapshot to be lost when retries fail. The team will change this logic in a future update. Action Items for the NFS Issue: To help prevent these snapshot loss failures, could you please provide the logs from the repository NFS (192.168.20.91)? Export the logs from the Veeam server and select the repository host. Provide the results of running this command directly on the Repository host: journalctl --since "30 days ago" > journal_repo.log Temporary Workaround: If you can, please temporarily switch to active fulls instead of synthetic fulls to help stabilize the job. Please let me know if you have any questions, and if you are able to raise that ticket with Vates. They also just responded back with this statment... Regarding the second part of the last email with the noticed Synthetic full issue, I actually would like you to also make this registry entry on the Veeam server and keep synthetic fulls enabled to see if it helps with that issue: Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication Name: Nfs3CommandWaitTimeoutSec Type: DWORD Value (In Decimal): 86400