Deploy VM via cloud-init config

[acebmxer]

I am currently having issues deploying a vm via cloud-init. This work previously as i was able to deploy a new XO from sources. Now when I try to deploy a vm via cloud-init I am not able to log into the new vm via the console from xo. nor does the vm show IP no do i see anything in my dhcp server.

If i deploy another test vm with XOA all works fine. Both XOA and XO using the same Cloud-init config.

Not sure if this issue is from recent commits to XO or recent patches to xcp-ng.

Anyone else having similar issues?

Edit also see this -

[poddingue]

I can't claim to know this part well, but the task name in your screenshot matches an older report, vatesfr/xen-orchestra#7496, where Importing content into VDI XO CloudConfigDrive also sat at 0% forever on XO from sources.
In that one the host's self-signed certificate was being refused, and the only visible sign was a DEPTH_ZERO_SELF_SIGNED_CERT line next to importVdiContent in the xo-server log.
Nothing showed in the UI at all.
Since XOA works for you with the same cloud-config, how each one trusts the host certificate feels like a plausible place to look first, though I could be fully wrong about that.
Could you check your xo-server output while one of those tasks is stuck and paste whatever importVdiContent line comes up, along with your commit number?
If it reproduces on another host or a fresh build, that would make a solid GitHub issue, and a mention to @Team-XO-Backend might help in the meantime.

[acebmxer]

@poddingue

With some trouble shooting with AI this is what I have to come back with.... Prior to posting this it struck me that i might have misconfigured this new xo deployment. I still had previous xo deployment and looked at its config for my installer and saw the xo service account was set to root. On the new deployment i forgot to change that and it used the default none root user. I then switch the user to root and still and issue. Below are the logs and issues from AI while XO service account was none root.

I hit what I believe is the same thing, and I have a stack trace that points at a specific regression rather than the certificate issue from #7496.

Setup: XO from sources, xo-server v5.206.0, XCP-ng pool, non-root service user.

Symptom: identical to the original post — "Importing content into VDI XO CloudConfigDrive" appears to do nothing, the VM boots with no IP, console is unusable, and the exact same cloud-init config works fine on XOA. Nothing is shown in the UI.

The reason nothing shows up in the UI is that the error is caught and downgraded to a WARN, so you only see it in journalctl -u xo-server:

2026-07-29T21:54:49.406Z xo:xapi WARN importVdiContent:  {
  error: TypeError: stream.on is not a function
      at watchUploadProgress (file:///opt/xen-orchestra/@xen-orchestra/xapi/_watchUploadProgress.mjs:25:10)
      at Xapi.importContent (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:295:7)
      at Xapi.createCloudInitConfigDrive (file:///opt/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1545:5)
      at Xapi.createCloudInitConfig (file:///opt/xen-orchestra/@xen-orchestra/xapi/vm.mjs:435:14)
      at Xo.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:191:26)
      at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22)
      at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20)
      at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:475:18)
(The warning is followed by a very large dump of the pool_master / SR / VDI objects, trimmed here.)

Probable cause: createCloudInitConfigDrive builds the config drive as an in-memory Buffer and passes it to importContent. importContent now calls watchUploadProgress(stream, stream.length, …), and that helper calls stream.on('data', …). A Buffer has .length — so it passes the existing if (stream.length === undefined) guard — but it has no .on() method, so it throws. Every other caller of importContent passes a real stream; the cloud-init config drive appears to be the only one passing a Buffer.

_watchUploadProgress.mjs is new — it has a single commit in its history:

$ git -C /opt/xen-orchestra log -1 --format='%h %ad %s' -- @xen-orchestra/xapi/_watchUploadProgress.mjs
76b45a9b3 Mon Jul 27 10:23:29 2026 +0200 fix(xo-server): ensure import task progress is correctly updated (#10133)
That would explain why XOA is unaffected (it lags master) while XO from sources breaks, and why this started recently.

Knock-on effect: because the failure is swallowed by a .catch(), the deferred VDI cleanup never runs and execution continues to VBD_create. The VM ends up with a config drive attached that was never written to — so cloud-init finds no user-data at all, which matches the "no IP, no DHCP request" symptom. Any VMs created during the failure need to be deleted; fixing XO won't repair them.

Local workaround — in @xen-orchestra/xapi/_watchUploadProgress.mjs, returning early when the input isn't a stream:

if (stream == null || typeof stream.on !== 'function') return
Progress reporting on a 10 MB in-memory buffer isn't meaningful anyway. This is just a local patch that a rebuild will overwrite — posting it as a datapoint, not a proper fix.

[acebmxer]

@poddingue

After fighting with AI a bit I think this is root cause.....

The commit that broke it:

76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 — PR #10133, "fix(xo-server): ensure import task progress is correctly updated", merged Mon Jul 27 10:23:29 2026 +0200 by fbeauchamp.
https://github.com/vatesfr/xen-orchestra/commit/76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2

It was in my draft, but buried in a git log code block halfway down. Here's the reply rewritten with it as the opening line.

This is a regression from a specific commit, not the certificate issue from #7496.

Broken by: commit 76b45a9b3b9ba636ba3d2148f8575ac4d9f57ff2 (PR #10133, "fix(xo-server): ensure import task progress is correctly updated", 27 Jul 2026). That commit added @xen-orchestra/xapi/_watchUploadProgress.mjs, which breaks all cloud-init VM creation. There is no follow-up commit to that file on master, so current master is still broken.

Why it breaks: createCloudInitConfigDrive builds the config drive as an in-memory Buffer and passes it to importContent. The new code in importContent calls watchUploadProgress(stream, stream.length, …), which calls stream.on('data', …). A Buffer has .length, so it passes the existing if (stream.length === undefined) guard, but it has no .on() method — so it throws. Every other caller of importContent passes a real stream; the cloud-init config drive appears to be the only Buffer caller.

The error (only visible in journalctl -u xo-server — it's swallowed by a .catch() and never shown in the UI):

2026-07-29T21:54:49.406Z xo:xapi WARN importVdiContent: {
error: TypeError: stream.on is not a function
at watchUploadProgress (file:///opt/xen-orchestra/@xen-orchestra/xapi/_watchUploadProgress.mjs:25:10)
at Xapi.importContent (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:295:7)
at Xapi.createCloudInitConfigDrive (file:///opt/xen-orchestra/packages/xo-server/src/xapi/index.mjs:1545:5)
at Xapi.createCloudInitConfig (file:///opt/xen-orchestra/@xen-orchestra/xapi/vm.mjs:435:14)
at Xo.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:191:26)
at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22)
at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20)
at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:475:18)
Result: the failure is caught and logged as a WARN, so the deferred VDI cleanup never runs and execution continues to VBD_create. The VM gets a config drive attached that was never written — physical_utilisation: 3584 against a virtual_size of 10485760. cloud-init finds no user-data, hence no IP, no DHCP request, unusable console. VMs created during the failure can't be recovered by fixing XO; they must be recreated.

Not a permissions issue. I switched the xo-server service account to root and rebuilt — identical error. It's a type error in the code, independent of the account.

Affects: XO from sources on any build since 27 Jul. XOA lags master, which is why the same cloud-init config works there.

Environment: xo-server v5.206.0, XO from sources, XCP-ng pool.

Suggested fix: watchUploadProgress should return early when handed a non-stream, or createCloudInitConfigDrive should wrap the Buffer in a Readable.