Small follow-up on the NBD connections idea, because I asked internally and it turns out that was not a false lead at all. The number of NBD connections is capped per host, so 20 per disk is well past what the host will actually hand out, and it is worth turning down whatever else is going on in this thread. I do not have the exact ceiling yet. I have asked the XAPI side for the real number and I will post it here once I have it, because guessing at it would not help anyone. @Andrew, that fits your side too as far as I can tell, since you are at 1 and have seen this once rather than constantly. probain, I will send you an upload link by direct message so you have somewhere to put the logs. I still do not know whether the fallback itself is new this week or simply newly visible, so please do not read anything into my message on that part.

@poddingue After fighting with AI a bit I think this is root cause..... The commit that broke it: 76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 — PR #10133, "fix(xo-server): ensure import task progress is correctly updated", merged Mon Jul 27 10:23:29 2026 +0200 by fbeauchamp. https://github.com/vatesfr/xen-orchestra/commit/76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 It was in my draft, but buried in a git log code block halfway down. Here's the reply rewritten with it as the opening line. This is a regression from a specific commit, not the certificate issue from #7496. Broken by: commit 76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 (PR #10133, "fix(xo-server): ensure import task progress is correctly updated", 27 Jul 2026). That commit added @xen-orchestra/xapi/_watchUploadProgress.mjs, which breaks all cloud-init VM creation. There is no follow-up commit to that file on master, so current master is still broken. Why it breaks: createCloudInitConfigDrive builds the config drive as an in-memory Buffer and passes it to importContent. The new code in importContent calls watchUploadProgress(stream, stream.length, …), which calls stream.on('data', …). A Buffer has .length, so it passes the existing if (stream.length === undefined) guard, but it has no .on() method — so it throws. Every other caller of importContent passes a real stream; the cloud-init config drive appears to be the only Buffer caller. The error (only visible in journalctl -u xo-server — it's swallowed by a .catch() and never shown in the UI): 2026-07-29T21:54:49.406Z xo:xapi WARN importVdiContent: { error: TypeError: stream.on is not a function at watchUploadProgress (file:///opt/xen-orchestra/@xen-orchestra/xapi/_watchUploadProgress.mjs:25:10) at Xapi.importContent (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:295:7) at Xapi.createCloudInitConfigDrive (file:///opt/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1545:5) at Xapi.createCloudInitConfig (file:///opt/xen-orchestra/@xen-orchestra/xapi/vm.mjs:435:14) at Xo.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:191:26) at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22) at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20) at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:475:18) Result: the failure is caught and logged as a WARN, so the deferred VDI cleanup never runs and execution continues to VBD_create. The VM gets a config drive attached that was never written — physical_utilisation: 3584 against a virtual_size of 10485760. cloud-init finds no user-data, hence no IP, no DHCP request, unusable console. VMs created during the failure can't be recovered by fixing XO; they must be recreated. Not a permissions issue. I switched the xo-server service account to root and rebuilt — identical error. It's a type error in the code, independent of the account. Affects: XO from sources on any build since 27 Jul. XOA lags master, which is why the same cloud-init config works there. Environment: xo-server v5.206.0, XO from sources, XCP-ng pool. Suggested fix: watchUploadProgress should return early when handed a non-stream, or createCloudInitConfigDrive should wrap the Buffer in a Readable.

I removed the start-delay parameter from one of the VMs (only this one has it) and now it works again. I don't get it. No idea how to achieve a startup delay for a specific VM though. There's also "auto_poweron_delay" and I don't understand what the difference from "start-delay" is.

Yes, you can force start and remove the backup job and that's it.

Hello @anthoineb, We captured another occurrence on the same VM, os-ott-data-3-2 (172.30.52.193), on the same hypervisor and QCOW2 VDI as the incident reported on 2026-07-25. This occurrence is useful because xen_blkfront.max_ring_page_order=3 was definitely active: xenstore showed ring-page-order=3 with eight ring references; the guest queue depth was 256; 255 write requests remained in flight; all 256 blk-mq scheduler tags were busy; wbt_lat_usec=0; I/O PSI full was approximately 98%; completed xvdb counters did not advance between samples. The backend state was nevertheless the same as in the previous capture: tapdisk was sleeping in scheduler_wait_for_events(); tap-ctl reported reqs_outstanding=0; tap request counters were 0/0; req_prod=0 and rsp_prod=0; no tap, image, VBD, map or xenbus errors were reported. One detail may be relevant: GDB still printed n_reqs=32 and n_reqs_free=32, although the frontend had negotiated an order-3 ring and the guest exposed 256 tags. Is this expected because n_reqs represents a separate fixed tapdisk request pool, or could it indicate that tapdisk is not observing the complete multi-page ring? The controller captured GDB and the guest/hypervisor state before reboot. After the ten-minute diagnostic window, the same signature was confirmed three more times and the VM was rebooted. It rejoined the cluster successfully. I have attached: the complete GDB/tap-ctl capture; the guest blk-mq, diskstats, PSI, D-state stack and kernel-journal snapshot; the corresponding tapdisk/xenstore, daemon.log and SMlog excerpt; a README and SHA-256 checksums. The important new result appears to be that increasing max_ring_page_order from 0 to 3 did not prevent the stall; the same mismatch recurred with 255/256 frontend requests/tags occupied while the backend ring and tapdisk remained empty. 20260730-005220-gdb-172.30.52.193.txt 20260730-pre-reboot-guest-172.30.52.193.txt 20260730-pre-reboot-hypervisor-172.30.50.191.txt README.txt SHA256SUMS.txt