Ubuntu cloud images on XCP-ng 8.3 UEFI: ~15s per secondary vCPU at boot, caused by console=ttyS0

[dvinni]

Sharing a debugging result that may explain some of the “slow UEFI VM boot” reports (e.g. in the XO 6 feedback thread).

Symptom: Ubuntu 26.04 cloud image (cloud-init template, UEFI, 8 vCPU) takes ~1.5 min of black screen before any kernel output. OVMF phase is fast (7s per qemu-dm log, ExitBootServices OK), and systemd-analyze claims ~8s total — yet wall clock says otherwise.

Root cause found via sched_clock: early printk timestamps freeze during smp: Bringing up secondary CPUs / installing Xen timer for CPU N, and the correction shows up later:

[    1.671429] smp: Bringing up secondary CPUs ...
[    1.671429] installing Xen timer for CPU 1
[    1.671429] smpboot: x86: Booting SMP configuration:
[    1.671429] .... node  #0, CPUs:      #1
[    1.671429] installing Xen timer for CPU 2
[    1.671429]  #2
[    1.671429] installing Xen timer for CPU 3
[    1.671429]  #3
[    1.671429] installing Xen timer for CPU 4
[    1.671429]  #4
[    1.671429] installing Xen timer for CPU 5
[    1.671429]  #5
[    1.671429] installing Xen timer for CPU 6
[    1.671429]  #6
[    1.671429] installing Xen timer for CPU 7
[    1.671429]  #7
[    1.671429] cpu 1 spinlock event irq 81
[    1.671429] cpu 2 spinlock event irq 82
[    1.671429] cpu 3 spinlock event irq 83
[    1.671429] cpu 4 spinlock event irq 84
[    1.671429] cpu 5 spinlock event irq 85
[    1.697554] cpu 6 spinlock event irq 86
[    1.757519] cpu 7 spinlock event irq 87
[    1.758405] smp: Brought up 1 node, 8 CPUs
...
[    2.549234] sched_clock: Marking stable (1444006278, 1105086856)->(145865443398, -143316350264)

i.e. ~143 seconds of real time hidden at the SMP bringup stage (~15-20s per secondary vCPU). Scales linearly: with 2 vCPUs the correction is ~15s.

Culprit: console=ttyS0 in the cloud image’s default kernel cmdline (/etc/default/grub.d/50-cloudimg-settings.cfg). Early boot printk output is written synchronously to the emulated 16550 UART; every byte is an I/O port access = VM exit. The verbose early boot output serializes around AP bringup with frozen clocks. An ISO-installed 26.04 on an identical VM config (same platform flags, same kernel 7.0.0-29) doesn’t have ttyS0 in cmdline and boots ~7x faster through this phase.

Fix / proof:

sed -i 's/console=tty1 console=ttyS0/console=tty1/' /etc/default/grub /etc/default/grub.d/50-cloudimg-settings.cfg
update-grub

After reboot the same VM shows sched_clock ... -16146235182 — down from 143s to 16s. Tested on both amd64 and amd64v3 builds, identical results.

Remaining ~2s per vCPU seems to be the baseline UEFI/Xen-timer overhead others have reported — still there, but tolerable.

Environment: XCP-ng 8.3 (fully patched), Xen 4.17, pool of Xeon Gold 6342/6348, guests: Ubuntu 26.04 kernel 7.0.0-29-generic, device-model qemu-upstream-uefi.

For cloud-init templates the workaround is a runcmd in the cloud config applying the sed above.

[dvinni]

Update: possibly related to the TSC regression tracked in
this thread
(kernel 6.12.5+, upstream fix pending) — the symptom signature matches (slow “installing Xen timer” on UEFI only), though that thread’s repros are all AMD while my pool is Intel Ice Lake. The remaining ~2s/vCPU after the ttyS0 fix may be that underlying issue. Unlike the tsc_mode=2 workaround, removing ttyS0 keeps live migration intact.

[poddingue]

Welcome to the forum, and thanks for writing this up instead of just fixing it on your own box.

You already found the thread I was going to point you at. You're ahead of me there.
What happened in it after the part you read: Olivier bisected the problem down to Linux 6.12.5, and Teddy linked an upstream fix that landed in tip (f24df84cbe05), with no ETA for when distros pick it up. The tsc_mode=2 plus nomigrate workaround came out of that, and it carries the live migration cost you already spotted.

One thing I'd like to know, and I don't think anyone has tried it on your side. Does the leftover couple of seconds per vCPU go away if you set tsc_mode=2 on one of those VMs? If it does, that would tie your case to the same regression rather than to a second thing.

I'm not deep enough in the timer code to tell you whether the ttyS0 amplification and the TSC regression are the same mechanism or two things stacking. Probably worth a mention to @Team-Hypervisor-Kernel so someone who is can have a look.

[dvinni]

@poddingue Tested as requested: on a VM with the ttyS0 fix already applied (leftover was ~16s / ~2s per vCPU), setting tsc_mode=2 + nomigrate=true brings the sched_clock correction down to ~0.3s:

[    0.281587] sched_clock: Marking stable (281005908, 562301)->(625590665, -344022456)

vs ~-16s without it (reverted after the test). So it's the same regression, now also confirmed on Intel (Ice Lake, Xeon Gold 6342/6348) — with console=ttyS0 acting as a ~7x amplifier on top, presumably because every early printk to the emulated UART goes through the affected clock path.

[poddingue]

That settles it, thanks for running it. 0.3s against 16s on the same VM is a cleaner answer than I expected.

I read the commit Teddy linked, and it makes your "same thing or two things" question look like one thing: f24df84cbe05 registers the jiffies clocksource before it gets used, because until then its max delta sits at zero and reads clamp to zero, so time stops advancing. If that's the right read, console=ttyS0 isn't a second bug, it's just more wall clock spent inside the window where the clock is stuck.
Take it with a grain of salt, that's me reading a commit message rather than testing anything, so don't take it from me.

More useful, maybe: I went and checked kernel/time/jiffies.c across the stable branches, and the fix is in 6.12.97+, the current 6.18.y, 7.1.4+ and 7.2, but not in 7.0.y, which isn't on kernel.org's maintained list any more. So on your 7.0.0-29 it looks like it has to come from Canonical rather than from an upstream stable update.
Nothing in the patch is CPU-vendor specific either, for what that's worth.

[dvinni]

@poddingue Good news on that front — Canonical already picked it up: the fix ("time/jiffies: Register jiffies clocksource before usage") is in kernel 7.0.0-31, currently sitting in resolute-proposed as part of the 2026.08.03 SRU cycle. So it should reach -updates around early September without needing an upstream 7.0.y backport.

Your reading of the commit matches what I see too — one stalled clock, two consumers (AP bringup waits + printk/UART delays), not two separate bugs.

Removing console=ttyS0 from cloud images is still worth keeping in cloud-init templates though: verbose early printk to an emulated 16550 is pointless overhead on any kernel, and boot logs remain available via dmesg anyway.

[poddingue]

Update on this, and it's better news than what I was about to tell you.

I built a 26.04 cloud image VM on a lab host and walked it up the kernels. Stock image is 7.0.0-28 and gave me a 59s sched_clock correction on 6 vCPUs. apt upgrade got me 7.0.0-30 out of resolute-updates, which only landed this morning, and that's still 50 to 59s. Then 7.0.0-31 from resolute-proposed: 0.4s, twice. Wall clock from reboot to sshd went 78s to 31s.

The part that matters for you is that I left console=ttyS0 in the cmdline for all of those runs, and never set tsc_mode. So -31 looks like it fixes this without your grub edit and without losing live migration. Canonical did take the patch, it just hasn't come out of proposed yet, and -30 doesn't have it, so upgrading normally today won't get you there.

No idea when it promotes, and I wouldn't put a proposed kernel on a production pool on my say-so. Until it does, your ttyS0 change still looks like the cheaper of the two workarounds.