@poddingue Okay, we're back here. I was told I have a hung CIFS mount that no longer exists, so a host needs to reboot to clear it, and faulty XOSTOR resources. But I can't reboot the host until I can move VMs off. Is there any indication of how to repair the faulty XOSTOR resources so I can move the VMs and reboot the host? Thanks for taking a look.

@florent said: @jr-m4 you can export the heap memory of the nodeJS process by doing kill -SIGUSER2 <xoserverpid> onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment Do you have somewhere I can upload the heapsnapshot? (148MB) Ping @poddingue as well, for visibility

@JB @christopher-petzel Thanks for your patience! We are glad it works well!

@poddingue Hey, it's just fun building stuff, even if it ends up in the bit-bucket (as unfotunately that's the destiny of all things anyway). Happy Day

@florent No explicit assigned migration or backup network set. Backups generate errors somewhat randomly. As one VM won't get the "error" one time. But on the next backup run it will. And the third time it might not. This even happens on VMs that are turned off, and therefore shouldn't have any changed bytes between backups. Host is reachable from XO Network has NBD enabled All VMs use VHD IPv4 Backup type is Delta

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

@gskger Understood.. Thanks for clarifying @gskger

We are fully aware of Broadcom's decision to stop providing the VDDK library to the cloud community or to reserve its download to some authorized partners and customers. As much as we regret this decision, we have to respect it. Because it impacts users who need to make backups, we are evaluating alternative options to address this situation. We will provide further guidance as this evaluation progresses but can't share more information for now.

install_xen_orchestra v0.7.2 — twelve templates, and the builder can now use XO's own API Since the v0.5.0 post the template library has gone from one distro to twelve, and it picked up a second way of talking to the pool. Everything from v0.5.0 through v0.7.2, in one place. Templates: twelve buildable, three still to come v0.5.0 shipped with Debian 13 and I asked here which distros to do next. That list is now: Debian 12, 13 Ubuntu 22.04, 24.04, 26.04 LTS AlmaLinux 8, 9, 10 CentOS Stream 9, 10 Fedora 43, 44 Rocky Linux 8/9/10 are the only rows still marked Coming Soon... in the menu. Every one of these is a row in the catalogue table rather than new build code, which is what I said in the v0.5.0 post I was aiming for. Each image did get read individually rather than assumed to match its neighbour — default user, disk size, kernel type, partition table — because those differ inside a family. Ubuntu 22.04 expands to 2.2 GiB and 24.04 to 3.5 GiB, so copying one figure across the family would have inflated every VM cloned from the smaller one. Two things that had to be solved along the way and now apply to every image: qcow2 images import. Debian publishes raw; almost everyone else publishes qcow2, so the builder converts before importing. The shipped password actually works. The RHEL-family and Fedora images declare lock_passwd: True for their default user, and cloud-init re-locks that account on every boot — so the password the build set was already locked by the time a clone came up. The build now drops a file in /etc/cloud/cloud.cfg.d turning that off. Delete it once you've put your own SSH keys on. Key auth was never affected. Arrow keys in the template menu now skip the Coming Soon rows instead of stepping through them. The builder can now go through XO's API instead of SSH to dom0 Template building has always driven xe over SSH to the pool master, which means having the pool master's root password to hand. Everything it does that way, XO exposes over its own API — create the VM, import the disk, build the cloud-init drive — so v0.7.0 added that as a second path. Which one makes sense depends on where you run it from. If you installed with --deploy, this repo is sitting inside the XO VM and that's where you run --update from, so XO is on localhost and you already have an API token in xo-config.cfg for the pre-update task check. Same token, no root password. Run it from your workstation against a pool with no XO on it yet and none of that is true, so SSH stays. TEMPLATE_BUILD_METHOD picks: auto # default — API if it's reachable and authenticated, else SSH, and it says which and why api # API only, no fallback ssh # exactly what it did before The choice is made by a preflight check before anything gets created on the pool, so a fallback costs you nothing but a different password prompt. On the API path the image streams from its mirror straight into XO without being staged to a file, and XO builds the cloud-init drive itself — so no ISO writer needed on the machine you're running from. Two new config keys: XO_API_TOKEN and XO_URL. XO_API_TOKEN replaces XO_TASK_CHECK_TOKEN in name only — the old key is still read, so an existing config keeps working untouched. XO_URL can be left unset on the XO VM itself, and falls back to PUBLIC_URL if you've set that. --deploy fixes The appliance now gets the XCP-ng guest tools. A deployed VM had no guest agent at all, so the XO it was itself hosting couldn't report its IP, memory or disk usage, and couldn't shut it down cleanly. It ran fine while looking half-blank in the very interface it served. It now boots UEFI, not BIOS. Nothing was ever setting the firmware parameter, so a VM from --deploy and a VM cloned from a --build-templates template came out of the same script with different firmware. It's probed from the disk now, same as the builder does. The temporary deployment SSH key is actually removed. It wasn't being deleted, leaving a working key on the VM. Progress bars curl --progress-bar redraws the whole line every update and leaves the cursor sitting in it, which over SSH is a cursor visibly scribbling back and forth instead of a bar filling up. All four transfers now draw a bar that doesn't repaint, including the one that stages the image on the pool master — which is the longest step of a deploy and was the one people would actually be watching. Also Templates get sensible platform settings instead of inheriting them from the scaffolding: vga=std with 16 MiB video memory (the stock 4 MiB cirrus is what leaves the console at 640x480), viridian off (it's Hyper-V enlightenment, meant for Windows), and cores-per-socket matching the vCPU count. Checksum verification works for every origin now. AlmaLinux, CentOS Stream and Fedora would each have 404'd on the wrong filename and imported unverified with a warning. Debian 11 and Ubuntu 22.04 are deprecated with dates on them, and the menu says so rather than the row silently vanishing one day. The README's menu screenshots no longer need a horizontal scrollbar on GitHub. 353 unit tests, and CI smoke-tests the install on 10 distros. Repo: https://github.com/acebmxer/install_xen_orchestra Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md Templates doc: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

@acebmxer we are far more explicit on the real usage of NBD, paing the way to mark a failed NBD connexion as an error in a few month also working on a diagnostic tool to be more efficient on the checks thank you all for your time

@dkidd255 @jamesg I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control). In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

Thanks for the feedback, @dvinni !

@Dan Hello, I can't answer from memory, we'll take a look at this and come back to you Thanks for the report, I guess it could be considered a bug since even it's one of the other, it's not the expected behaviour to have the mode change between runs.

Hello there, Thanks a lot for raising the question and offering ideas to move forward @caps ! I would like to say that I really like the AI crafted screenshot provided 🤩 But, to be clear, the status as of today regarding Docker or other containers integration in XO/XCP-ng is that there is an item in the DevOps Tools team roadmap. The item is just to look and define what could be done (example update xscontainer, do something totally new, ...). Unfortunately this spike has a low priority and is not yet scheduled. That being said, there is an active project, which is close to what you're asking. It's about a better integration of Kubernetes, including, cluster update, adding nodes, etc. More on this should come in the next months! ️ Also note that I am chatting with different people and teams in Vates about this thread and the Docker integration topic to see if we could do something. Also last thing is that, the community is always welcome to build on top of our open source softwares. I am pretty sure that if someone would contribute a Xen Orchestra plugin to integrate Docker we would welcome this, and we could give tips and guidance. Like @teddyastie said, I'm not sure having Docker, Podman, related features in the guest agent would be something our colleagues would merge. But maybe the docker daemon socket could be kind of shared between VMs (I'm not sure at all this is possible and if possible how to do it). In the meantime exposing the docker daemon over network (with restrictions) to Xen Orchestra would be quick'n easy and I think safe enough if done carefully.

@AtaxyaNetwork Thank you for sharing the latest version. It works very well except I'm getting error when trying to install any plugin. Did you get this as well ? [image: 1788936195627-d8c58b99-0330-4c85-830c-998b37004d59-image.jpeg]

v0.6.0 — log collection is in. This was the piece the first post said was next, and it works end to end now. A new Collect page runs a per-host collection as a background job: it pulls logs.tgz and the XAPI audit trail from Xen Orchestra, keeps the raw copies, and produces a redacted copy of each. On my own XCP-ng 8.3 pool that came to 434 MiB for the bundle and 769 MiB for the audit trail, 2.3 GiB stored across the five files, with 2,542,805 values masked — almost all of them session tokens. The download is streamed a megabyte at a time and never held in memory, and every chunk is a cancellation point, so Cancel actually stops a transfer rather than waiting it out. A failed or cancelled download deletes its partial file instead of leaving a half-written bundle sitting there looking like a collection. Both copies are kept and marked redacted — safe to send or raw — unmasked. Redaction is lossy, and the raw copy is the only thing that can answer "what did that used to say?" afterwards. Each collection writes the same redaction report the preview page produces, so you can see which rules fired — and which were switched off — before anything leaves the box. In the screenshot I had IPv4, MAC, UUIDs and hostnames turned off, and they show as off rather than as zero hits. That is deliberate: a partly-masked bundle should be obvious before you attach it to a ticket. There is also a retention policy. The newest n collections are kept whatever their age, and only what is left is judged on age — a long gap in collecting can't empty the store. It shows you what a cleanup would delete and how much space that returns before you press the button. Expect bugs. This is still very much a work in progress and it has only ever run against my own pool. If you try it in a different environment, assume things will break — please tell me what did. Known issues right now: The red "This connection uses a restricted account" warning on the Collect page shows for every connection, including an administrator one. It is a wrong condition in the template, not a real permission check — ignore it if your account is admin. The screenshot below has it, over a collection that succeeded. The Collect page says to expect "433 MB and 100 seconds per host". The 100 seconds is the download only; a full collection also fetches the audit trail and redacts both copies, which took 203 seconds on my pool. Budget three to four minutes. Two more things worth flagging if you try this. logs.tgz sends no Content-Length, so a reverse proxy that buffers will happily hand you a truncated archive with HTTP 200. Mine did. It now detects that and reports the bundle as cut short rather than as a protocol error, and repacks whatever did arrive instead of throwing the lot away. If you are behind Nginx Proxy Manager, proxy_buffering off; and proxy_max_temp_file_size 0; are what fixed the stall for me. The account requirement from the first post has not changed — the log download needs export:logs on host, and on the instance I measured that is only reachable via Administrator. Inventory still works fine on Read only. Next up: individual log categories pulled out of the cached bundle (~35 MB instead of 434 MiB), date ranges, and then findings. https://github.com/acebmxer/xcp_pulse [image: 1788812029125-screenshot_20260907_160937.png]

On one machine in my lab, /etc/rsyslog.d/xenserver.conf defines 17 outchannels, every one of them at 104857600 bytes, and /var/log/messages isn't one of them; it's written by the stock *.info;mail.none;authpriv.none;cron.none line in /etc/rsyslog.conf and rotated by /etc/logrotate.d/syslog, so nightly and with no size trigger. The docs at https://docs.xcp-ng.org/guides/logs#rsyslog describe the 100 MiB rule but don't list which files it actually covers, which is probably why this is hard to work out without reading the config the way you did. Whether messages can realistically outrun a nightly rotation I don't know, and I'd be guessing if I said either way; the one thing in your favour is that /var/log is its own filesystem, 3.9 GB on the box I looked at (small lab machine, I haven't searched in the code, so I don't know where that size is decided), so it can't take the rest of dom0 with it. Whether that omission is deliberate or just inherited (or even something fancier) is really a question for whoever owns that file, so it might be worth a mention to @Team-OS-Platform-Release.