@poddingue Okay, we're back here. I was told I have a hung CIFS mount that no longer exists, so a host needs to reboot to clear it, and faulty XOSTOR resources. But I can't reboot the host until I can move VMs off. Is there any indication of how to repair the faulty XOSTOR resources so I can move the VMs and reboot the host? Thanks for taking a look.

Looking at Git, it doesn't seem like it, so I guess that answers my question.

@afk said: Hi everyone, Some of you may already be aware of this but I learned the news this morning. https://www.virtualizationhowto.com/2026/09/leaving-vmware-just-got-harder-after-broadcom-pulled-vddk-downloads/ Essentially, Broadcom has decided, without any announcement, to remove all download links (and even some documentation it seems) for VDDK. This impacts all migration tools that use the VDDK for copying data of VM disks. If you have any version of VDDK, please keep it somewhere safe and make backups. I only have a x86-64 linux release from september 2025 that I used for testing V2V. Obviously, I can't share any link here but the release has the following sha256 hash: 79d215198f1b8fd1d240a16b27ac2543c521ae2afdc1876444c2acf8945d74ca VMware-vix-disklib-9.0.0.0.24742305.x86_64.tar.gz Apparently, web archival links are also taken down. @olivierlambert With Broadcom removing public access to VDDK downloads, we now have users who either cannot obtain VDDK at all or whose existing VDDK installations (main + backup) have become corrupted. This creates a real need for a fully open‑source fallback path inside V2V, one that still meets the performance requirements normally associated with VDDK. The open‑source community has already demonstrated that VDDK‑level throughput is achievable without VMware’s proprietary stack. Several projects have shown that: • Parallel NBD can saturate modern storage bandwidth when implemented correctly • Highly multithreaded block pipelines can outperform single‑threaded VDDK transfers • A throwaway Linux Proxy Copy VM can act as an efficient intermediary, reading blocks via NBD and streaming them directly to the destination hypervisor. This approach avoids VDDK entirely, remains fully open source, and provides a reliable fallback for users who can no longer access VMware’s proprietary components. Given the direction Broadcom is taking, having this fallback inside V2V would significantly improve resilience for XCP‑ng users and ensure long‑term independence from VMware licensing changes. The Proxy Copy VM must be treated as an ephemeral orchestration component — automatically excluded from the migration set and destroyed once the transfer completes — ensuring it never interferes with HA, LB, or startup/shutdown sequencing. This architecture also gives Vates a clean, open‑source answer to Broadcom’s decision to pull public VDDK access — ensuring V2V remains fully functional even if VMware continues removing or restricting proprietary components.

@florent said: @jr-m4 you can export the heap memory of the nodeJS process by doing kill -SIGUSER2 <xoserverpid> onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment Do you have somewhere I can upload the heapsnapshot? (148MB) Ping @poddingue as well, for visibility

v0.6.0 — log collection is in. This was the piece the first post said was next, and it works end to end now. A new Collect page runs a per-host collection as a background job: it pulls logs.tgz and the XAPI audit trail from Xen Orchestra, keeps the raw copies, and produces a redacted copy of each. On my own XCP-ng 8.3 pool that came to 434 MiB for the bundle and 769 MiB for the audit trail, 2.3 GiB stored across the five files, with 2,542,805 values masked — almost all of them session tokens. The download is streamed a megabyte at a time and never held in memory, and every chunk is a cancellation point, so Cancel actually stops a transfer rather than waiting it out. A failed or cancelled download deletes its partial file instead of leaving a half-written bundle sitting there looking like a collection. Both copies are kept and marked redacted — safe to send or raw — unmasked. Redaction is lossy, and the raw copy is the only thing that can answer "what did that used to say?" afterwards. Each collection writes the same redaction report the preview page produces, so you can see which rules fired — and which were switched off — before anything leaves the box. In the screenshot I had IPv4, MAC, UUIDs and hostnames turned off, and they show as off rather than as zero hits. That is deliberate: a partly-masked bundle should be obvious before you attach it to a ticket. There is also a retention policy. The newest n collections are kept whatever their age, and only what is left is judged on age — a long gap in collecting can't empty the store. It shows you what a cleanup would delete and how much space that returns before you press the button. Expect bugs. This is still very much a work in progress and it has only ever run against my own pool. If you try it in a different environment, assume things will break — please tell me what did. Known issues right now: The red "This connection uses a restricted account" warning on the Collect page shows for every connection, including an administrator one. It is a wrong condition in the template, not a real permission check — ignore it if your account is admin. The screenshot below has it, over a collection that succeeded. The Collect page says to expect "433 MB and 100 seconds per host". The 100 seconds is the download only; a full collection also fetches the audit trail and redacts both copies, which took 203 seconds on my pool. Budget three to four minutes. Two more things worth flagging if you try this. logs.tgz sends no Content-Length, so a reverse proxy that buffers will happily hand you a truncated archive with HTTP 200. Mine did. It now detects that and reports the bundle as cut short rather than as a protocol error, and repacks whatever did arrive instead of throwing the lot away. If you are behind Nginx Proxy Manager, proxy_buffering off; and proxy_max_temp_file_size 0; are what fixed the stall for me. The account requirement from the first post has not changed — the log download needs export:logs on host, and on the instance I measured that is only reachable via Administrator. Inventory still works fine on Read only. Next up: individual log categories pulled out of the cached bundle (~35 MB instead of 434 MiB), date ranges, and then findings. https://github.com/acebmxer/xcp_pulse [image: 1788812029125-screenshot_20260907_160937.png]

@irtaza9 Hello, could you share the content of your logs during the SR creation with us to investigate on our side ? You can produce a tar of logs with https://docs.xcp-ng.org/troubleshooting/log-files/#produce-a-status-report .

install_xen_orchestra v0.7.2 — twelve templates, and the builder can now use XO's own API Since the v0.5.0 post the template library has gone from one distro to twelve, and it picked up a second way of talking to the pool. Everything from v0.5.0 through v0.7.2, in one place. Templates: twelve buildable, three still to come v0.5.0 shipped with Debian 13 and I asked here which distros to do next. That list is now: Debian 12, 13 Ubuntu 22.04, 24.04, 26.04 LTS AlmaLinux 8, 9, 10 CentOS Stream 9, 10 Fedora 43, 44 Rocky Linux 8/9/10 are the only rows still marked Coming Soon... in the menu. Every one of these is a row in the catalogue table rather than new build code, which is what I said in the v0.5.0 post I was aiming for. Each image did get read individually rather than assumed to match its neighbour — default user, disk size, kernel type, partition table — because those differ inside a family. Ubuntu 22.04 expands to 2.2 GiB and 24.04 to 3.5 GiB, so copying one figure across the family would have inflated every VM cloned from the smaller one. Two things that had to be solved along the way and now apply to every image: qcow2 images import. Debian publishes raw; almost everyone else publishes qcow2, so the builder converts before importing. The shipped password actually works. The RHEL-family and Fedora images declare lock_passwd: True for their default user, and cloud-init re-locks that account on every boot — so the password the build set was already locked by the time a clone came up. The build now drops a file in /etc/cloud/cloud.cfg.d turning that off. Delete it once you've put your own SSH keys on. Key auth was never affected. Arrow keys in the template menu now skip the Coming Soon rows instead of stepping through them. The builder can now go through XO's API instead of SSH to dom0 Template building has always driven xe over SSH to the pool master, which means having the pool master's root password to hand. Everything it does that way, XO exposes over its own API — create the VM, import the disk, build the cloud-init drive — so v0.7.0 added that as a second path. Which one makes sense depends on where you run it from. If you installed with --deploy, this repo is sitting inside the XO VM and that's where you run --update from, so XO is on localhost and you already have an API token in xo-config.cfg for the pre-update task check. Same token, no root password. Run it from your workstation against a pool with no XO on it yet and none of that is true, so SSH stays. TEMPLATE_BUILD_METHOD picks: auto # default — API if it's reachable and authenticated, else SSH, and it says which and why api # API only, no fallback ssh # exactly what it did before The choice is made by a preflight check before anything gets created on the pool, so a fallback costs you nothing but a different password prompt. On the API path the image streams from its mirror straight into XO without being staged to a file, and XO builds the cloud-init drive itself — so no ISO writer needed on the machine you're running from. Two new config keys: XO_API_TOKEN and XO_URL. XO_API_TOKEN replaces XO_TASK_CHECK_TOKEN in name only — the old key is still read, so an existing config keeps working untouched. XO_URL can be left unset on the XO VM itself, and falls back to PUBLIC_URL if you've set that. --deploy fixes The appliance now gets the XCP-ng guest tools. A deployed VM had no guest agent at all, so the XO it was itself hosting couldn't report its IP, memory or disk usage, and couldn't shut it down cleanly. It ran fine while looking half-blank in the very interface it served. It now boots UEFI, not BIOS. Nothing was ever setting the firmware parameter, so a VM from --deploy and a VM cloned from a --build-templates template came out of the same script with different firmware. It's probed from the disk now, same as the builder does. The temporary deployment SSH key is actually removed. It wasn't being deleted, leaving a working key on the VM. Progress bars curl --progress-bar redraws the whole line every update and leaves the cursor sitting in it, which over SSH is a cursor visibly scribbling back and forth instead of a bar filling up. All four transfers now draw a bar that doesn't repaint, including the one that stages the image on the pool master — which is the longest step of a deploy and was the one people would actually be watching. Also Templates get sensible platform settings instead of inheriting them from the scaffolding: vga=std with 16 MiB video memory (the stock 4 MiB cirrus is what leaves the console at 640x480), viridian off (it's Hyper-V enlightenment, meant for Windows), and cores-per-socket matching the vCPU count. Checksum verification works for every origin now. AlmaLinux, CentOS Stream and Fedora would each have 404'd on the wrong filename and imported unverified with a warning. Debian 11 and Ubuntu 22.04 are deprecated with dates on them, and the menu says so rather than the row silently vanishing one day. The README's menu screenshots no longer need a horizontal scrollbar on GitHub. 353 unit tests, and CI smoke-tests the install on 10 distros. Repo: https://github.com/acebmxer/install_xen_orchestra Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md Templates doc: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

@poddingue Hey, it's just fun building stuff, even if it ends up in the bit-bucket (as unfotunately that's the destiny of all things anyway). Happy Day

While I think we need to do something about containers in VMs, I'm not convinced this is a good idea adding this to the regular guest agent for various reasons. Docker is one way of running containers, but there are numerous other ones like podman or other runc/containerd based ones. So we can't really make a one size fits all solution. And that doesn't fully answer the maintenance and long-term aspect. Well, actually what you are looking for is some form of control on guest docker runtime, but without using network, that's actually a solved problem, but plumbing is missing. And we don't need a custom docker-aware agent for this, we mostly need to use one of vsock/pv-channel/pvcalls to build a bridge between e.g guest docker socket and Xen Orchestra (which would talk to guest docker and control/get info from it).

On one machine in my lab, /etc/rsyslog.d/xenserver.conf defines 17 outchannels, every one of them at 104857600 bytes, and /var/log/messages isn't one of them; it's written by the stock *.info;mail.none;authpriv.none;cron.none line in /etc/rsyslog.conf and rotated by /etc/logrotate.d/syslog, so nightly and with no size trigger. The docs at https://docs.xcp-ng.org/guides/logs#rsyslog describe the 100 MiB rule but don't list which files it actually covers, which is probably why this is hard to work out without reading the config the way you did. Whether messages can realistically outrun a nightly rotation I don't know, and I'd be guessing if I said either way; the one thing in your favour is that /var/log is its own filesystem, 3.9 GB on the box I looked at (small lab machine, I haven't searched in the code, so I don't know where that size is decided), so it can't take the rest of dom0 with it. Whether that omission is deliberate or just inherited (or even something fancier) is really a question for whoever owns that file, so it might be worth a mention to @Team-OS-Platform-Release.

Hello everyone ! I rebuild a new OME appliance with the latest version (4.7) What i did on the rebuild: Via live CD add xen-blkfront and xen-netfront driver, and rebuild the initrd change root password activate SSH. Via TUI Change admin password Set interface to DHCP Via SSH sed -i -e 's/sda/xvda/g' /opt/dell/omc/utilities/tui/bin/ome_disk_config.sh sed -i -e 's/sda/xvda/g' /opt/dell/mcsi/appliance/scripts/appliance_resources.sh bash /mnt/Linux/install.sh # To install XCP-ng tools As usual, please send a DM if you want the link to the VHD Also, if you want to build it yourself, the driver are here https://cecilemorange.fr/drivers/xen-netfront.ko https://cecilemorange.fr/drivers/xen-blkfront.ko Feedback is welcome ! (Disclamer: This build is something I create on my free time and to help people who have difficulty running Dell appliances on XCP-ng. I do not guarantee a fully functional appliance, but I do my best to make it work. If you have a support contract with Dell and require official support for OME, please contact Dell and request compatibility with XCP-ng.)

@florent No explicit assigned migration or backup network set. Backups generate errors somewhat randomly. As one VM won't get the "error" one time. But on the next backup run it will. And the third time it might not. This even happens on VMs that are turned off, and therefore shouldn't have any changed bytes between backups. Host is reachable from XO Network has NBD enabled All VMs use VHD IPv4 Backup type is Delta

@acebmxer we are far more explicit on the real usage of NBD, paing the way to mark a failed NBD connexion as an error in a few month also working on a diagnostic tool to be more efficient on the checks thank you all for your time

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

New version: https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases/tag/v20260903 This one includes native Ceph rbd SXM over SMAPIv3! GitHub updated with the latest commits and changes. As always, use with caution. I did run a lot of test scenario's but please do test yourself and let me know your findings!

@dkidd255 @jamesg I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control). In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

If you are running XOA, not XO from sources, then we could take a look remotely using the support tunnel. Otherwise, make sure that you have patched and rebooted each pool member.

This was sent to me yesterday!! See our Engineering Team lead's response: We evaluated XCP-ng 8.3 some time ago, but its dom0 kernel was too old for our driver to work properly. Running SupremeRAID inside a DomU should be feasible and is similar to the approach we previously proposed for VMware. However, with XCP-ng 8.3, we did not find a practical native path to export the SupremeRAID VD back to dom0 and use it as an XCP-ng SR. Using iSCSI for this purpose would add significant protocol and networking overhead, which is not ideal for high-performance NVMe storage. I revisited XCP-ng 9.0 and the current Xen/XAPI development. A more promising approach is to run SupremeRAID in a dedicated AlmaLinux VM with the GPU and NVMe drives passed through, then use Xen's native xen-blkback interface to export the SupremeRAID block device back to dom0. Dom0 would see the exported VD as a normal Xen block device, which could then potentially be used to create a standard XCP-ng LVM SR. This approach is much more attractive than iSCSI or NVMe/TCP because the data path uses Xen's blkif shared-memory interface rather than a network protocol. xen-blkback itself is an established Xen mechanism, and Xen supports using a separate domain as a block backend. However, XCP-ng does not currently provide complete first-class lifecycle management for this configuration, so we still need to validate the exact behavior on XCP-ng 9.0, particularly persistent attachment, storage VM startup ordering, and recovery after a host or storage VM reboot. If they can help confirm that the GPU and NVMe drives can be passed through to the storage VM, SupremeRAID can run normally there, and the resulting VD can be exported through xen-blkback to dom0 and used as an XCP-ng SR, I think this could be a very solid architecture for SupremeRAID on XCP-ng. As for write durability, SupremeRAID always operates in write-through mode. An I/O is acknowledged only after all associated data, including parity, has been committed to the drives. Therefore, acknowledged writes do not depend on data or parity remaining only in volatile GPU or host memory Can you do the initial testing with SupremeRAID PRO within your environment? I will begin the initial testing and builds with 4 nodes: 1 & 2 are HP DL380 Gen10 - 2x Xeon Gold 6151 36 Cores 384GB RAM 4 3.84Tb PCI4.0 NVMe per node. Twinstore Testing - Run the Build environment on these 3 is a HP Dl360 Gen10 2x Xeon Gold 6151 36 Cores 512Gb RAM 4x1.92TB NVMe Drives SuperServer SYS-122H-TN- X14 2x Xeon 6740 96 cores, 512GB RAM, GRAID Card - 4x Pci5.0 7.68TB drives SupremeRAID. All the nodes have 2x100Gbps ports, 2x25Gbps ports. Arista Backed network.