@christopher-petzel Ok! Thanks!

@Cyrille Appreciate the help! Please keep posted.

@benapetr @olivierlambert I've made a new release which includes SMAPIv1 improvements and a proper SMAPIv3 volume + datapath plugin. https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases#release-v20260827 Let me know your thoughts and if there is anything you'll like to see added / changed / tested. Next step for me is CBT and SXM.

@dinhngtu Hello. I've updated to the latest commit. There is no need to give me credit. I just want to help so others and myself included can benefit from this. Anyway thank you for your work.

@dinhngtu Something called "Elpha Secure" ...none of our other antivirus shows it being bad but I wanted to ask around before I unflagged it.

@poddingue said: I haven't used that tab myself, and since it's built around VMs that do have backups, I don't know if it gives you the negative view you want. If it doesn't, https://feedback.vates.tech is where that gets counted. Maybe you can build some sort of filter but I don't think so. I'll use https://feedback.vates.tech for future suggestions. Tks

Thanks for the numbers. So the whole chain drags: 4 to 5 minutes to list the ~90 LUNs, another 1 to 3 to validate one, and the create on top of that. From the XO side I can't tell where that time goes, XO itself, the host's storage layer, or the array answering slowly, and the one measurement that would split it is the host-side probe I mentioned: time xe sr-probe type=lvmoiscsi device-config:target=<ip>, then the same call with device-config:targetIQN=<iqn> added. If the CLI takes just as long, XO is mostly off the hook. A 90-LUN multipath target doesn't sound exotic to me, so probes taking minutes feel worth a proper look, though I honestly don't know what's normal at that scale. @Team-Storage, the numbers above are concrete now, in case it's worth a look on your side.

@poddingue You're right. The RPU task shown in the list was a previous one. Even that one finished fine. So both RPU tasks didn't show at all today! My bad. The evacuation and remirroring tasks were all shown. Latest XOA 6.7.1

@poddingue Thanks for the pointers — the kernel module policy page and the VMD stretch of the 9.0 demonstrator thread were both exactly what I needed. Reporting back on the call as promised. Where Graid landed. They want as much detail as we can give them, and it goes to their R&D team in Taiwan. They flagged that they don't support kernel modules outside their qualified matrix, which is the crux of it. So I went digging into what that matrix actually contains, and there's considerably more overlap than the initial conversation suggested. Their Linux driver 2.0.1, released 21 August, lists AlmaLinux 10.0–10.2 on kernel 6.12 — the same base and kernel line as the 9.0 demonstrator. They also support Proxmox VE across five separate non-distro kernels (6.5, 6.8, 6.14, 6.17, 7.0), so building against a hypervisor's own kernel is routine for them. This isn't a new platform class; it's one more kernel. You also had VROC and VMD the right way round, and that turns out to matter more than it first looked. VROC is the easier half Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter. VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0. Roughly what that path needs: CONFIG_VMD in the dom0 kernel The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30) mdadm with IMSM in dom0 Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one. SupremeRAID is harder I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back. On packaging I've told them plainly that the .run installer is a non-starter, there's no DKMS in dom0, and every kernel update means a rebuild somebody has to own permanently. @dicode-nl's Ceph work turned out to be the single most useful thing I could send them — a public, current example of third-party kernel modules packaged to policy for dom0, spec file and smoketests included. Between that and the public build container, their team can reach a testable RPM without needing anything from Vates first. Nice timing, and thank you. Questions for @yann and @Team-Hypervisor-Kernel These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after: Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask. What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere. What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing. Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM? Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in? If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead. Storage side Worth putting in perspective against what @dicode-nl just delivered for Ceph. RBD needed backported kernel modules, a SMAPIv1 driver, a SMAPIv3 volume and datapath plugin, and a change to XAPI's revert logic, because RBD is a network storage protocol whose snapshot semantics fight XAPI's built-in behaviour. That's genuinely hard work. GRAID needs none of it. A virtual drive is a local block device. No protocol, no snapshot semantics to reconcile, no SMAPI driver to write or maintain — a stock LVM SR goes straight on top. The whole effort sits in getting the driver to load correctly in dom0. What caught my eye, though, is TwinStor rather than XOSTOR. TwinStor sits on a single local block device and reports it directly (backing: sda: write cache disabled (durable)). A GRAID VD is exactly that shape. The 2-node case is where GRAID actually earns its keep. With only two copies, one dead drive degrades a whole replica and leaves you one failure from an outage until resilver completes — and that resilver runs over the replication link, in an environment the TwinStor thread shows is already sensitive to network state. GRAID absorbs the drive failure locally on the GPU, no network traffic, both replicas UpToDate throughout. Write amplification is saner too: RAID6 plus 2-way beats RAID6 plus 3-way. @olivierlambert — one question before I take this further, since it decides whether the idea is viable at all. TwinStor checks that the backing device's volatile write cache is disabled. Does that check assume a plain SCSI/NVMe target, and would it correctly detect a volatile stage inside a compound device like a GRAID virtual drive? I've asked Graid to confirm whether their VDs honour FLUSH/FUA end to end and whether any parity is staged in GPU or host memory before acknowledgement. If there were a volatile stage TwinStor couldn't see, the durability check would report clean while the guarantee was false — which seems worse than an outright incompatibility given what the fencing design exists to prevent. Happy to be told I'm overthinking it. The obvious counterweight is dom0 pressure: TwinStor already runs DRBD, the LIO target and multipath in there, and the NVIDIA driver plus graid daemon on top is a real squeeze. I've asked Graid for measured memory footprint numbers. Hardware I have two bare nodes I can dedicate to 9.0 testing. Usefully, the first gate needs no drives at all — does the card enumerate in dom0, does the NVIDIA driver load in a PV domain, does the graid module load, does it survive with the IOMMU on and other GPUs passed through to guests. That answers the high-risk questions with no hardware commitment from Graid. Drives come at gate two. Will post again when Taiwan responds. Mike George Bitrefinery

It was too tempting not to test, so I went and tested the other half. On an 8.3 host I set the preferred resolution to 800x600 in the OVMF menu on a UEFI Debian VM, turned that VM into a template, and cloned it. The clone came up at 800x600. A control clone of the same original, with nothing set, came up at 1024x768. So your workaround holds, the resolution really does ride along into VMs built from the template. For anyone who wants to poke at it, the setting is a UEFI variable called PlatformConfig under GUID 7235c51c-0c80-4cab-87ac-3b084a6304b1. It only appears in NVRAM once you commit it in the menu, and it stores width and height as plain little-endian integers, which is why it travels with the VM record. One thing I didn't expect: the OVMF help text says the mode list is filtered against video RAM size, but the VM I used had the default 4 MB and still offered everything up to 1280x1024. So you may not need to raise videoram for the common ones. Fair warning though, I measured the console at the firmware stage rather than after the distro's own driver takes over, so a guest that sets its own mode later could still override it. At least, that's my understanding.

@mike.potapov Can you upgrade to the latest blktap-3.55.5-9.3.xcpng8.3 to check is the issue is still there?

Thanks a lot for this feedback, @pedro!

@mpiton Thanks for looking into this, I guess it wasn't apparent that I needed to click the Save Configuration button for that plugin. I did that and confirmed that the secret now survives an xo-server restart. Thanks again!

No problem about the delay, of course. I put on better glasses, and guess what? There is a Container management section in the XO docs after all, in docs/xo5/manage_infrastructure.md. It doesn't describe the old xscontainer route though. It says you can run Docker inside a VM, then links out to Docker's own docs and the Kubernetes recipe, so there's no procedure sitting there to walk anyone into a wall, and no sign of the feature either. On the features matrix I can't check it the way I checked the rest, because the page builds itself in the browser and fetching it gives me nothing, so treat that one as unchecked rather than confirmed. The Feeder entry still isn't there, I looked again today, so that part stands.

Thanks, keep us posted!

I measured this on a two-NIC 8.3 host. As far as I understand, there's no way to do it from XO: the call that edits a network doesn't take a bond mode, every bondMode in the codebase sits on a create path, and the REST API on a running XOA offers create_bonded_network plus get and delete on a network, with nothing that edits one. The CLI route looks cheap though. xe bond-set-mode on a live bond cost zero dropped packets in both directions I tried, pinging every 200ms across the change, and the command returned in about 1.4 seconds. Two caveats, and the second is about your case specifically: creating and destroying the bond did interrupt the host (roughly 8 and 16 seconds), and I only went active-backup to balance-slb and back, never touching lacp, because my switch port isn't configured as a LAG and I'd have dropped the box. So I can't tell you your lacp to active-backup move is free, only that mode changes in general didn't cost me anything. Worth putting on https://feedback.vates.tech either way so the votes have somewhere to gather.

I went digging and three threads turned up people who already have this hardware. @yeopil21 in https://xcp-ng.org/forum/topic/11799 had an LPe37102-D on a Dell R770 that never showed up at all, @jinju0504 in https://xcp-ng.org/forum/topic/11946 ended up on the -alt package on an R670, and @lukas in https://xcp-ng.org/forum/topic/8847 was running an LPe31002-M6-D with NVMe-oF back on 8.3 beta2. That last one is the older generation, so I'd guess it's the side where you most want to hear nothing regressed. Tagging them in case the cards are still on their benches.

v0.4.0 Release - https://github.com/acebmxer/install_xen_orchestra/releases 0.4.0 - 2026-08-23 Changed The cloud image is staged on the pool master by default instead of being streamed. Staging is the only path that can resume a broken download, retry a transient failure, and check the downloaded size before anything reaches the VDI; streaming can do none of those, because a pipe already feeding a fixed-Content-Length PUT cannot be rewound. On a link that drops the occasional TLS record — which any multi-gigabyte transfer eventually meets — streaming failed every attempt while staging rode it out. Streaming is now what it should always have been: the fallback for a host without the few gigabytes of scratch space staging needs. Fixed A stalled streaming import no longer has to be interrupted by hand. When the download end died, the upload sat waiting for a response XAPI would never send. --speed-time could not help — by that point curl is waiting, not transferring, so the speed meter has stopped ticking and only --max-time 3600 would eventually fire. The two transfers now run as separate processes with the download's exit status watched, and the upload is killed the moment it fails. A pool master short on scratch space no longer fails the whole deploy. The staged path signalled "no room" with a plain non-zero return, which under set -e took the script down before the fallback could be reached. A failed cloud-image download no longer looks like a successful import. The streaming import piped one curl into another, and the remote shell reported only the upload side's exit status. A download that died partway — a transient SSL_read ... bad record mac on a 3 GB transfer is the usual cause — therefore produced a truncated disk that the deploy reported as imported, and a VM that booted into a corrupt filesystem. The pipeline now runs under bash -o pipefail. A broken image download no longer hangs the deploy for an hour. When the download end died, the upload curl had already promised XAPI an exact Content-Length and sat waiting to send bytes that were never coming, with XAPI waiting alongside it until --max-time 3600 expired — the visible symptom being a XAPI task frozen at partial progress and a script that had to be interrupted. Both ends now abort after 60s below 1 KiB/s. The staged image download resumes instead of starting over. It now uses -C - with --retry 5 --retry-delay 3 --retry-all-errors, so the transient TLS failures that a multi-gigabyte single-connection download eventually hits are ridden out rather than failing the deploy. The streaming path deliberately does not retry: curl re-issues from byte 0, and piped into a fixed-Content-Length PUT those bytes would be appended to the ones already sent, corrupting the image while appearing to succeed. A short staged download is refused rather than imported. The file's size is now checked against the length the server advertised before anything is written into the VDI. The staged download reports progress. It was silent for several minutes on the longest step of the deploy, which reads as a hang worth killing. A failed deploy names the VM it left behind, with the xe vm-destroy command, instead of leaving a half-built VM to be rediscovered later in the pool's VM list. Nothing is destroyed automatically. --update/--reconfigure/--rebuild no longer abort on a root install. Reading User= from a systemd unit that has no such line (which is what a root install looks like) failed the pipeline under set -o pipefail and took the script down before the fallback could run. Deploy prompts validate values, not just their shape. 999.999.999.999 was accepted as an address and 70000 as a port; both were only rejected after the VM existed, by an unreachable guest or by the installer inside it. Prompted settings are no longer lost when the base config omits the key. The generated xo-config.cfg was patched with sed, which silently does nothing for a key that is not there — so the VM installed on the default while the summary showed the value you typed. Missing keys are now appended. An $EDITOR with arguments works. code --wait passed the availability check and then failed with "No such file", since the whole string was treated as one executable path. Values edited into the config are validated. An unusable port or branch was silently ignored, leaving the summary showing one thing and the VM installing another. Troubleshooting commands point at a key that still exists. The ssh -i lines printed on a failed install and on a non-200 health check named the temporary key, which the exit trap had already deleted. A second VM with the same hostname no longer overwrites the first one's SSH key, which was the only way into that machine. The disk-space check before staging an image on the pool master is derived from the image's actual size instead of a hard-coded 4 GiB, which rejected small images and let large ones fill /var/tmp mid-download. tests/probe-xapi-deploy.sh acquires its XAPI session from the pool master, the way --deploy does, so a firewall that blocks port 443 from your workstation no longer skips the HTTP transport probes that matter. It also validates --host, --user, --sr, --image and --payload-mb before they reach a shell, drops the eval in the workstation-side transport, and exits non-zero when any probe failed rather than whenever one transport worked. The menu example in the README and the layout comment above MENU_NAMES described the old fixed 5/4/centered grid; with ten items the menu draws five entries in each column. Security The cloud image is verified against its published checksum. The size check catches a download that was cut short; it cannot catch one that arrived complete from the wrong place, because a substituted image has a perfectly consistent Content-Length. The staged image is now checked against the SHA512SUMS its origin publishes beside it — which is what Debian ships — and a mismatch aborts before anything reaches the disk. An origin that publishes no sums warns and continues, so a custom XO_DEPLOY_IMAGE_URL keeps working. Set XO_DEPLOY_IMAGE_SHA512 to require a specific digest instead: that makes the check mandatory, aborting rather than continuing unverified, and refuses the streaming import outright because a pipe fed straight into the VDI leaves no file to hash. Fetching sums over the same connection as the image is not a detached signature — it defends against a bad mirror or a stale cache, not an attacker holding the TLS session for both requests. A pinned pool-master fingerprint is now enforced instead of advised. deploy_verify_host_key fingerprinted the host key and then returned success on every path that could not complete the check — so with XO_DEPLOY_POOL_FINGERPRINT set, a ssh-keyscan that timed out meant the host password was sent to whatever answered on that address, which is exactly what pinning exists to prevent and the easiest outcome for an on-path attacker to arrange. A pin that cannot be checked is now a hard failure. The verified host key is bound to the connection that carries the password. The scanned key was fingerprinted, shown, and then discarded, while dom0_exec connected with StrictHostKeyChecking=accept-new against the default known_hosts — verifying one transaction and trusting another. Nothing stopped a different key, or the host's RSA key when the ED25519 one had been displayed, being accepted at connect time. The whole scan is now pinned into a run-scoped known_hosts that dom0_exec enforces with StrictHostKeyChecking=yes, the same way deploy_wait_for_guest already treated the guest. A hostile pool master can no longer run commands on the workstation. The free-space probe in deploy_import_vdi_staged fed the host's reply straight into (( )), which expands an array subscript before evaluating it — so an answer of PATH[$(...)] executed locally rather than being rejected. It is now checked against ^[0-9]+$ first, matching the guards already applied to size and got in the same function. Note that set -euo pipefail does not cover this: set -u blocks only the unbound-variable form of the payload. This mattered more after staging became the default import path, because the probe went from rarely reached to running on every deploy. The pool master's root password is no longer visible in ps. Three calls predating dom0_exec still used sshpass -p "$HOST_PASSWORD", putting the password in the process list where any other user on the workstation could read it. They now use sshpass -e with $SSHPASS, as dom0_exec does. The admin password hash is kept out of XO_DEBUG=1 output. deploy_harden_guest_sudo and deploy_build_config_drive were missing the local - / set +x guard the rest of the script uses, so the hash was printed by xtrace. Previously masked on automated runs only because --non-interactive left the hash empty; requiring a password made it reachable on every deploy. Revoking the deployment key can no longer empty authorized_keys. A grep failure — no space for the temporary file, an unreadable source — was swallowed by || true and the empty result written back, taking the operator's own key with it. grep's "nothing matched" (a legitimate empty result) is now distinguished from a real error, which aborts and leaves the file untouched. The streaming import's FIFO is created inside a private directory. mktemp -u returns a name without creating anything, leaving a window in dom0's world-writable /tmp. The FIFO now lives in a mktemp -d directory. DSA public keys are rejected. ssh-dss was accepted by deploy_load_pubkey, but OpenSSH has refused DSA since 7.0 and removed it in 9.8, so it only installed a key that silently never worked. The cloud-init cache scrub covers cloud-config.txt. The rendered config holds hashed_passwd just as the raw user-data does; only the latter was being redacted. The deployment SSH key is destroyed at the end of a deploy. It used to be...