Hello @anthoineb, We captured another occurrence on the same VM, os-ott-data-3-2 (172.30.52.193), on the same hypervisor and QCOW2 VDI as the incident reported on 2026-07-25. This occurrence is useful because xen_blkfront.max_ring_page_order=3 was definitely active: xenstore showed ring-page-order=3 with eight ring references; the guest queue depth was 256; 255 write requests remained in flight; all 256 blk-mq scheduler tags were busy; wbt_lat_usec=0; I/O PSI full was approximately 98%; completed xvdb counters did not advance between samples. The backend state was nevertheless the same as in the previous capture: tapdisk was sleeping in scheduler_wait_for_events(); tap-ctl reported reqs_outstanding=0; tap request counters were 0/0; req_prod=0 and rsp_prod=0; no tap, image, VBD, map or xenbus errors were reported. One detail may be relevant: GDB still printed n_reqs=32 and n_reqs_free=32, although the frontend had negotiated an order-3 ring and the guest exposed 256 tags. Is this expected because n_reqs represents a separate fixed tapdisk request pool, or could it indicate that tapdisk is not observing the complete multi-page ring? The controller captured GDB and the guest/hypervisor state before reboot. After the ten-minute diagnostic window, the same signature was confirmed three more times and the VM was rebooted. It rejoined the cluster successfully. I have attached: the complete GDB/tap-ctl capture; the guest blk-mq, diskstats, PSI, D-state stack and kernel-journal snapshot; the corresponding tapdisk/xenstore, daemon.log and SMlog excerpt; a README and SHA-256 checksums. The important new result appears to be that increasing max_ring_page_order from 0 to 3 did not prevent the stall; the same mismatch recurred with 255/256 frontend requests/tags occupied while the backend ring and tapdisk remained empty. 20260730-005220-gdb-172.30.52.193.txt 20260730-pre-reboot-guest-172.30.52.193.txt 20260730-pre-reboot-hypervisor-172.30.50.191.txt README.txt SHA256SUMS.txt

@poddingue After fighting with AI a bit I think this is root cause..... The commit that broke it: 76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 — PR #10133, "fix(xo-server): ensure import task progress is correctly updated", merged Mon Jul 27 10:23:29 2026 +0200 by fbeauchamp. https://github.com/vatesfr/xen-orchestra/commit/76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 It was in my draft, but buried in a git log code block halfway down. Here's the reply rewritten with it as the opening line. This is a regression from a specific commit, not the certificate issue from #7496. Broken by: commit 76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 (PR #10133, "fix(xo-server): ensure import task progress is correctly updated", 27 Jul 2026). That commit added @xen-orchestra/xapi/_watchUploadProgress.mjs, which breaks all cloud-init VM creation. There is no follow-up commit to that file on master, so current master is still broken. Why it breaks: createCloudInitConfigDrive builds the config drive as an in-memory Buffer and passes it to importContent. The new code in importContent calls watchUploadProgress(stream, stream.length, …), which calls stream.on('data', …). A Buffer has .length, so it passes the existing if (stream.length === undefined) guard, but it has no .on() method — so it throws. Every other caller of importContent passes a real stream; the cloud-init config drive appears to be the only Buffer caller. The error (only visible in journalctl -u xo-server — it's swallowed by a .catch() and never shown in the UI): 2026-07-29T21:54:49.406Z xo:xapi WARN importVdiContent: { error: TypeError: stream.on is not a function at watchUploadProgress (file:///opt/xen-orchestra/@xen-orchestra/xapi/_watchUploadProgress.mjs:25:10) at Xapi.importContent (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:295:7) at Xapi.createCloudInitConfigDrive (file:///opt/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1545:5) at Xapi.createCloudInitConfig (file:///opt/xen-orchestra/@xen-orchestra/xapi/vm.mjs:435:14) at Xo.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:191:26) at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22) at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20) at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:475:18) Result: the failure is caught and logged as a WARN, so the deferred VDI cleanup never runs and execution continues to VBD_create. The VM gets a config drive attached that was never written — physical_utilisation: 3584 against a virtual_size of 10485760. cloud-init finds no user-data, hence no IP, no DHCP request, unusable console. VMs created during the failure can't be recovered by fixing XO; they must be recreated. Not a permissions issue. I switched the xo-server service account to root and rebuilt — identical error. It's a type error in the code, independent of the account. Affects: XO from sources on any build since 27 Jul. XOA lags master, which is why the same cloud-init config works there. Environment: xo-server v5.206.0, XO from sources, XCP-ng pool. Suggested fix: watchUploadProgress should return early when handed a non-stream, or createCloudInitConfigDrive should wrap the Buffer in a Readable.

Small follow-up on the NBD connections idea, because I asked internally and it turns out that was not a false lead at all. The number of NBD connections is capped per host, so 20 per disk is well past what the host will actually hand out, and it is worth turning down whatever else is going on in this thread. I do not have the exact ceiling yet. I have asked the XAPI side for the real number and I will post it here once I have it, because guessing at it would not help anyone. @Andrew, that fits your side too as far as I can tell, since you are at 1 and have seen this once rather than constantly. probain, I will send you an upload link by direct message so you have somewhere to put the logs. I still do not know whether the fallback itself is new this week or simply newly visible, so please do not read anything into my message on that part.

I removed the start-delay parameter from one of the VMs (only this one has it) and now it works again. I don't get it. No idea how to achieve a startup delay for a specific VM though. There's also "auto_poweron_delay" and I don't understand what the difference from "start-delay" is.

Yes, you can force start and remove the backup job and that's it.

@acebmxer all good and thank you again for sharing your discovery on the forums

All hosts are restarted with 9.2.18 loaded now. Will see what happens next. [15:56 ovbh-pprod-xen05 ~]# drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36 DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION_CODE=0x090212 DRBD_KERNEL_VERSION=9.2.18 DRBDADM_VERSION_CODE=0x092100 DRBDADM_VERSION=9.33.0

Good news that it sorted itself out. Both changes went in at once and the next run came back clean on the ADDS VMs too, so I don't think there's anything left to pin down here, and I'd rather say that than invent an explanation after the fact. Your question about the snapshot modes was the useful thing to come out of this thread. I went looking, they genuinely aren't documented anywhere, and that's written up on our side now If the fall back to a full ever comes back on the ADDS VMs specifically, that would deserve its own thread with the SMlog output @tjkreidl asked for, since nobody got to look at that part.

@yannsionneau That's great to hear, I can't wait to see what it'll look like.

I have been in the Network Enterprise business for 28 years. I live in Virginia which is heavily populated with Data Centers and adding more all the time. However, we have quite a few customers especially municipalities which are saying NO to having anything hosted with cloud computing due to security and endless rising cost. We have been deploying xcp-ng in quite a few installations, on customer premise. I think VATES will find there is a whole other market in these areas they can reach. That being said, Thanks to VATES and all who have worked on solutions for this! It truly benefits everyone!

I got this running and when I set up a catalog and baseline the report tells me that it has nothing in the catalog. Is that because it doesn't understand XCP-ng or because my hardware is old enough that it has dropped out of the catalog? I have two PE R630 and one PE R730. I tried both the full default Dell catalog as well as the one that limits to BIOS and firmware updates.

@olivierlambert I have a full Realtek 8125/8126/8127 update to replace the existing r8125 driver, as required to support newer chips. The new 8125 driver drops the 8126 support but you just install the 8126 if needed (requires newer 8125 driver to avoid module conflicts).

i think im seeing the same with fake cpu spikes. I started a second pool yesterday to patch and left it on over night. typically i just shut it down once done so the servers dont get too far behind patching. Under the pool stats im seeing spikes but under hosts nothing. It doesnt really bother me, but figure i add to the thread that im also seeing it. [image: 1784817154694-c75fecec-d618-4f05-a931-edea6bc6ecce-image-resized.jpeg] [image: 1784817189696-1b9ea5aa-d349-4ff8-af89-30a970d36fbb-image-resized.jpeg] [image: 1784817219194-0f579999-cd96-4216-a0d4-d637ea5febf1-image-resized.jpeg]

Two PRs out of the lowest priority bug ever, that made my morning.

@john.c See this post - https://xcp-ng.org/forum/post/105564

Thanks for the feedback @norpan , worth it! Ping @florent

@poddingue Oh yes (dumb me). Very good, we'll see if they respond (as now there's two @ refferences). Happy Day

@pedro_udifar Hi, You can try again with minPartSize=100000000 indeed. But I would advise you to contact B2 directly. They should have a clearer error to show instead of a 500. Thanks for your patience and your help

Got this issue since 2026-07-12.