@acebmxer we are far more explicit on the real usage of NBD, paing the way to mark a failed NBD connexion as an error in a few month also working on a diagnostic tool to be more efficient on the checks thank you all for your time

@jcdick1 you should follow this procedure https://help.vates.tech/kb/en-us/34-support-tunnel-and-access/30-create-a-support-tunnel

@poddingue said: On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising tcp_mem buys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on an xo_server restart points the same way. It might be worth a mention to @Team-XO-Backend, since they would know whether xo-server keeps sockets per pool. This is why I'm not increasing the numbers quite yet. Since this would only mask a potential issue. If an underlying cause is responsible, then this would be of interest. Thanks for chiming in!

@poddingue Hey, it's just fun building stuff, even if it ends up in the bit-bucket (as unfotunately that's the destiny of all things anyway). Happy Day

@florent No explicit assigned migration or backup network set. Backups generate errors somewhat randomly. As one VM won't get the "error" one time. But on the next backup run it will. And the third time it might not. This even happens on VMs that are turned off, and therefore shouldn't have any changed bytes between backups. Host is reachable from XO Network has NBD enabled All VMs use VHD IPv4 Backup type is Delta

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

install_xen_orchestra v0.7.2 — twelve templates, and the builder can now use XO's own API Since the v0.5.0 post the template library has gone from one distro to twelve, and it picked up a second way of talking to the pool. Everything from v0.5.0 through v0.7.2, in one place. Templates: twelve buildable, three still to come v0.5.0 shipped with Debian 13 and I asked here which distros to do next. That list is now: Debian 12, 13 Ubuntu 22.04, 24.04, 26.04 LTS AlmaLinux 8, 9, 10 CentOS Stream 9, 10 Fedora 43, 44 Rocky Linux 8/9/10 are the only rows still marked Coming Soon... in the menu. Every one of these is a row in the catalogue table rather than new build code, which is what I said in the v0.5.0 post I was aiming for. Each image did get read individually rather than assumed to match its neighbour — default user, disk size, kernel type, partition table — because those differ inside a family. Ubuntu 22.04 expands to 2.2 GiB and 24.04 to 3.5 GiB, so copying one figure across the family would have inflated every VM cloned from the smaller one. Two things that had to be solved along the way and now apply to every image: qcow2 images import. Debian publishes raw; almost everyone else publishes qcow2, so the builder converts before importing. The shipped password actually works. The RHEL-family and Fedora images declare lock_passwd: True for their default user, and cloud-init re-locks that account on every boot — so the password the build set was already locked by the time a clone came up. The build now drops a file in /etc/cloud/cloud.cfg.d turning that off. Delete it once you've put your own SSH keys on. Key auth was never affected. Arrow keys in the template menu now skip the Coming Soon rows instead of stepping through them. The builder can now go through XO's API instead of SSH to dom0 Template building has always driven xe over SSH to the pool master, which means having the pool master's root password to hand. Everything it does that way, XO exposes over its own API — create the VM, import the disk, build the cloud-init drive — so v0.7.0 added that as a second path. Which one makes sense depends on where you run it from. If you installed with --deploy, this repo is sitting inside the XO VM and that's where you run --update from, so XO is on localhost and you already have an API token in xo-config.cfg for the pre-update task check. Same token, no root password. Run it from your workstation against a pool with no XO on it yet and none of that is true, so SSH stays. TEMPLATE_BUILD_METHOD picks: auto # default — API if it's reachable and authenticated, else SSH, and it says which and why api # API only, no fallback ssh # exactly what it did before The choice is made by a preflight check before anything gets created on the pool, so a fallback costs you nothing but a different password prompt. On the API path the image streams from its mirror straight into XO without being staged to a file, and XO builds the cloud-init drive itself — so no ISO writer needed on the machine you're running from. Two new config keys: XO_API_TOKEN and XO_URL. XO_API_TOKEN replaces XO_TASK_CHECK_TOKEN in name only — the old key is still read, so an existing config keeps working untouched. XO_URL can be left unset on the XO VM itself, and falls back to PUBLIC_URL if you've set that. --deploy fixes The appliance now gets the XCP-ng guest tools. A deployed VM had no guest agent at all, so the XO it was itself hosting couldn't report its IP, memory or disk usage, and couldn't shut it down cleanly. It ran fine while looking half-blank in the very interface it served. It now boots UEFI, not BIOS. Nothing was ever setting the firmware parameter, so a VM from --deploy and a VM cloned from a --build-templates template came out of the same script with different firmware. It's probed from the disk now, same as the builder does. The temporary deployment SSH key is actually removed. It wasn't being deleted, leaving a working key on the VM. Progress bars curl --progress-bar redraws the whole line every update and leaves the cursor sitting in it, which over SSH is a cursor visibly scribbling back and forth instead of a bar filling up. All four transfers now draw a bar that doesn't repaint, including the one that stages the image on the pool master — which is the longest step of a deploy and was the one people would actually be watching. Also Templates get sensible platform settings instead of inheriting them from the scaffolding: vga=std with 16 MiB video memory (the stock 4 MiB cirrus is what leaves the console at 640x480), viridian off (it's Hyper-V enlightenment, meant for Windows), and cores-per-socket matching the vCPU count. Checksum verification works for every origin now. AlmaLinux, CentOS Stream and Fedora would each have 404'd on the wrong filename and imported unverified with a warning. Debian 11 and Ubuntu 22.04 are deprecated with dates on them, and the menu says so rather than the row silently vanishing one day. The README's menu screenshots no longer need a horizontal scrollbar on GitHub. 353 unit tests, and CI smoke-tests the install on 10 distros. Repo: https://github.com/acebmxer/install_xen_orchestra Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md Templates doc: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

@dkidd255 @jamesg I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control). In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

v0.6.0 — log collection is in. This was the piece the first post said was next, and it works end to end now. A new Collect page runs a per-host collection as a background job: it pulls logs.tgz and the XAPI audit trail from Xen Orchestra, keeps the raw copies, and produces a redacted copy of each. On my own XCP-ng 8.3 pool that came to 434 MiB for the bundle and 769 MiB for the audit trail, 2.3 GiB stored across the five files, with 2,542,805 values masked — almost all of them session tokens. The download is streamed a megabyte at a time and never held in memory, and every chunk is a cancellation point, so Cancel actually stops a transfer rather than waiting it out. A failed or cancelled download deletes its partial file instead of leaving a half-written bundle sitting there looking like a collection. Both copies are kept and marked redacted — safe to send or raw — unmasked. Redaction is lossy, and the raw copy is the only thing that can answer "what did that used to say?" afterwards. Each collection writes the same redaction report the preview page produces, so you can see which rules fired — and which were switched off — before anything leaves the box. In the screenshot I had IPv4, MAC, UUIDs and hostnames turned off, and they show as off rather than as zero hits. That is deliberate: a partly-masked bundle should be obvious before you attach it to a ticket. There is also a retention policy. The newest n collections are kept whatever their age, and only what is left is judged on age — a long gap in collecting can't empty the store. It shows you what a cleanup would delete and how much space that returns before you press the button. Expect bugs. This is still very much a work in progress and it has only ever run against my own pool. If you try it in a different environment, assume things will break — please tell me what did. Known issues right now: The red "This connection uses a restricted account" warning on the Collect page shows for every connection, including an administrator one. It is a wrong condition in the template, not a real permission check — ignore it if your account is admin. The screenshot below has it, over a collection that succeeded. The Collect page says to expect "433 MB and 100 seconds per host". The 100 seconds is the download only; a full collection also fetches the audit trail and redacts both copies, which took 203 seconds on my pool. Budget three to four minutes. Two more things worth flagging if you try this. logs.tgz sends no Content-Length, so a reverse proxy that buffers will happily hand you a truncated archive with HTTP 200. Mine did. It now detects that and reports the bundle as cut short rather than as a protocol error, and repacks whatever did arrive instead of throwing the lot away. If you are behind Nginx Proxy Manager, proxy_buffering off; and proxy_max_temp_file_size 0; are what fixed the stall for me. The account requirement from the first post has not changed — the log download needs export:logs on host, and on the instance I measured that is only reachable via Administrator. Inventory still works fine on Read only. Next up: individual log categories pulled out of the cached bundle (~35 MB instead of 434 MiB), date ranges, and then findings. https://github.com/acebmxer/xcp_pulse [image: 1788812029125-screenshot_20260907_160937.png]

@irtaza9 Hello, could you share the content of your logs during the SR creation with us to investigate on our side ? You can produce a tar of logs with https://docs.xcp-ng.org/troubleshooting/log-files/#produce-a-status-report .

On one machine in my lab, /etc/rsyslog.d/xenserver.conf defines 17 outchannels, every one of them at 104857600 bytes, and /var/log/messages isn't one of them; it's written by the stock *.info;mail.none;authpriv.none;cron.none line in /etc/rsyslog.conf and rotated by /etc/logrotate.d/syslog, so nightly and with no size trigger. The docs at https://docs.xcp-ng.org/guides/logs#rsyslog describe the 100 MiB rule but don't list which files it actually covers, which is probably why this is hard to work out without reading the config the way you did. Whether messages can realistically outrun a nightly rotation I don't know, and I'd be guessing if I said either way; the one thing in your favour is that /var/log is its own filesystem, 3.9 GB on the box I looked at (small lab machine, I haven't searched in the code, so I don't know where that size is decided), so it can't take the rest of dom0 with it. Whether that omission is deliberate or just inherited (or even something fancier) is really a question for whoever owns that file, so it might be worth a mention to @Team-OS-Platform-Release.

New version: https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases/tag/v20260903 This one includes native Ceph rbd SXM over SMAPIv3! GitHub updated with the latest commits and changes. As always, use with caution. I did run a lot of test scenario's but please do test yourself and let me know your findings!

If you are running XOA, not XO from sources, then we could take a look remotely using the support tunnel. Otherwise, make sure that you have patched and rebooted each pool member.

While I think we need to do something about containers in VMs, I'm not convinced this is a good idea adding this to the regular guest agent for various reasons. Docker is one way of running containers, but there are numerous other ones like podman or other runc/containerd based ones. So we can't really make a one size fits all solution. And that doesn't fully answer the maintenance and long-term aspect. Well, actually what you are looking for is some form of control on guest docker runtime, but without using network, that's actually a solved problem, but plumbing is missing. And we don't need a custom docker-aware agent for this, we mostly need to use one of vsock/pv-channel/pvcalls to build a bridge between e.g guest docker socket and Xen Orchestra (which would talk to guest docker and control/get info from it).

Hello everyone ! I rebuild a new OME appliance with the latest version (4.7) What i did on the rebuild: Via live CD add xen-blkfront and xen-netfront driver, and rebuild the initrd change root password activate SSH. Via TUI Change admin password Set interface to DHCP Via SSH sed -i -e 's/sda/xvda/g' /opt/dell/omc/utilities/tui/bin/ome_disk_config.sh sed -i -e 's/sda/xvda/g' /opt/dell/mcsi/appliance/scripts/appliance_resources.sh bash /mnt/Linux/install.sh # To install XCP-ng tools As usual, please send a DM if you want the link to the VHD Also, if you want to build it yourself, the driver are here https://cecilemorange.fr/drivers/xen-netfront.ko https://cecilemorange.fr/drivers/xen-blkfront.ko Feedback is welcome ! (Disclamer: This build is something I create on my free time and to help people who have difficulty running Dell appliances on XCP-ng. I do not guarantee a fully functional appliance, but I do my best to make it work. If you have a support contract with Dell and require official support for OME, please contact Dell and request compatibility with XCP-ng.)

This was sent to me yesterday!! See our Engineering Team lead's response: We evaluated XCP-ng 8.3 some time ago, but its dom0 kernel was too old for our driver to work properly. Running SupremeRAID inside a DomU should be feasible and is similar to the approach we previously proposed for VMware. However, with XCP-ng 8.3, we did not find a practical native path to export the SupremeRAID VD back to dom0 and use it as an XCP-ng SR. Using iSCSI for this purpose would add significant protocol and networking overhead, which is not ideal for high-performance NVMe storage. I revisited XCP-ng 9.0 and the current Xen/XAPI development. A more promising approach is to run SupremeRAID in a dedicated AlmaLinux VM with the GPU and NVMe drives passed through, then use Xen's native xen-blkback interface to export the SupremeRAID block device back to dom0. Dom0 would see the exported VD as a normal Xen block device, which could then potentially be used to create a standard XCP-ng LVM SR. This approach is much more attractive than iSCSI or NVMe/TCP because the data path uses Xen's blkif shared-memory interface rather than a network protocol. xen-blkback itself is an established Xen mechanism, and Xen supports using a separate domain as a block backend. However, XCP-ng does not currently provide complete first-class lifecycle management for this configuration, so we still need to validate the exact behavior on XCP-ng 9.0, particularly persistent attachment, storage VM startup ordering, and recovery after a host or storage VM reboot. If they can help confirm that the GPU and NVMe drives can be passed through to the storage VM, SupremeRAID can run normally there, and the resulting VD can be exported through xen-blkback to dom0 and used as an XCP-ng SR, I think this could be a very solid architecture for SupremeRAID on XCP-ng. As for write durability, SupremeRAID always operates in write-through mode. An I/O is acknowledged only after all associated data, including parity, has been committed to the drives. Therefore, acknowledged writes do not depend on data or parity remaining only in volatile GPU or host memory Can you do the initial testing with SupremeRAID PRO within your environment? I will begin the initial testing and builds with 4 nodes: 1 & 2 are HP DL380 Gen10 - 2x Xeon Gold 6151 36 Cores 384GB RAM 4 3.84Tb PCI4.0 NVMe per node. Twinstore Testing - Run the Build environment on these 3 is a HP Dl360 Gen10 2x Xeon Gold 6151 36 Cores 512Gb RAM 4x1.92TB NVMe Drives SuperServer SYS-122H-TN- X14 2x Xeon 6740 96 cores, 512GB RAM, GRAID Card - 4x Pci5.0 7.68TB drives SupremeRAID. All the nodes have 2x100Gbps ports, 2x25Gbps ports. Arista Backed network.

@andriy.sultanov I finally got around to fixing my capture card and I gotta say, it's a real pain to be using both methods and doesn't feel "correct" especially when there are warnings about possibly losing dom0 configurations after updates and the knowledge base saying that the old hide PCI from dom0 method is now considered obsolete. Perhaps the best solution would just be to provide users with a CLI command that would add that particular device to the xl pci-assignable-list if it doesn't fall into classes 01XX, 02XX or 03XX. Because I had added my GPU to my VM using the webUI, but then I also wanted to add the capture card through the console, but it appears that in the end you basically need to undo what's been done in the webUI and do it all through the console, otherwise you'd add the capture card manually with command line and then that would remove the GPUs... So I think some sort of manual override is necessary, even if it's a CLI option and then once that's done we could go into the webUI to enable it once the device has been marked as safe to passthrough.

Hi, Thanks for your answers For the second problem, after deletion, it is impossible to have the connection to the xo-server despite the refresh of the XOCE page. In a new firefox tab = same problem I closed and then opened Firefox and xo-server connection = OK Best regards

@olivierlambert - Just updating. The issue is still present in the version: 6.7.1. I doubt any fix was made. May be not many people are deploying windows 11 with vTPM through templates. Otherwise this issue should have been reported by others.