@poddingue
After fighting with AI a bit I think this is root cause.....
The commit that broke it:
76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 — PR #10133, "fix(xo-server): ensure import task progress is correctly updated", merged Mon Jul 27 10:23:29 2026 +0200 by fbeauchamp.
https://github.com/vatesfr/xen-orchestra/commit/76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2
It was in my draft, but buried in a git log code block halfway down. Here's the reply rewritten with it as the opening line.
This is a regression from a specific commit, not the certificate issue from #7496.
Broken by: commit 76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 (PR #10133, "fix(xo-server): ensure import task progress is correctly updated", 27 Jul 2026). That commit added @xen-orchestra/xapi/_watchUploadProgress.mjs, which breaks all cloud-init VM creation. There is no follow-up commit to that file on master, so current master is still broken.
Why it breaks: createCloudInitConfigDrive builds the config drive as an in-memory Buffer and passes it to importContent. The new code in importContent calls watchUploadProgress(stream, stream.length, …), which calls stream.on('data', …). A Buffer has .length, so it passes the existing if (stream.length === undefined) guard, but it has no .on() method — so it throws. Every other caller of importContent passes a real stream; the cloud-init config drive appears to be the only Buffer caller.
The error (only visible in journalctl -u xo-server — it's swallowed by a .catch() and never shown in the UI):
2026-07-29T21:54:49.406Z xo:xapi WARN importVdiContent: {
error: TypeError: stream.on is not a function
at watchUploadProgress (file:///opt/xen-orchestra/@xen-orchestra/xapi/_watchUploadProgress.mjs:25:10)
at Xapi.importContent (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:295:7)
at Xapi.createCloudInitConfigDrive (file:///opt/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1545:5)
at Xapi.createCloudInitConfig (file:///opt/xen-orchestra/@xen-orchestra/xapi/vm.mjs:435:14)
at Xo.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:191:26)
at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22)
at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20)
at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:475:18)
Result: the failure is caught and logged as a WARN, so the deferred VDI cleanup never runs and execution continues to VBD_create. The VM gets a config drive attached that was never written — physical_utilisation: 3584 against a virtual_size of 10485760. cloud-init finds no user-data, hence no IP, no DHCP request, unusable console. VMs created during the failure can't be recovered by fixing XO; they must be recreated.
Not a permissions issue. I switched the xo-server service account to root and rebuilt — identical error. It's a type error in the code, independent of the account.
Affects: XO from sources on any build since 27 Jul. XOA lags master, which is why the same cloud-init config works there.
Environment: xo-server v5.206.0, XO from sources, XCP-ng pool.
Suggested fix: watchUploadProgress should return early when handed a non-stream, or createCloudInitConfigDrive should wrap the Buffer in a Readable.
