Ubuntu cloud images on XCP-ng 8.3 UEFI: ~15s per secondary vCPU at boot, caused by console=ttyS0

[dvinni]

Sharing a debugging result that may explain some of the “slow UEFI VM boot” reports (e.g. in the XO 6 feedback thread).

Symptom: Ubuntu 26.04 cloud image (cloud-init template, UEFI, 8 vCPU) takes ~1.5 min of black screen before any kernel output. OVMF phase is fast (7s per qemu-dm log, ExitBootServices OK), and systemd-analyze claims ~8s total — yet wall clock says otherwise.

Root cause found via sched_clock: early printk timestamps freeze during smp: Bringing up secondary CPUs / installing Xen timer for CPU N, and the correction shows up later:

[    1.671429] smp: Bringing up secondary CPUs ...
[    1.671429] installing Xen timer for CPU 1
[    1.671429] smpboot: x86: Booting SMP configuration:
[    1.671429] .... node  #0, CPUs:      #1
[    1.671429] installing Xen timer for CPU 2
[    1.671429]  #2
[    1.671429] installing Xen timer for CPU 3
[    1.671429]  #3
[    1.671429] installing Xen timer for CPU 4
[    1.671429]  #4
[    1.671429] installing Xen timer for CPU 5
[    1.671429]  #5
[    1.671429] installing Xen timer for CPU 6
[    1.671429]  #6
[    1.671429] installing Xen timer for CPU 7
[    1.671429]  #7
[    1.671429] cpu 1 spinlock event irq 81
[    1.671429] cpu 2 spinlock event irq 82
[    1.671429] cpu 3 spinlock event irq 83
[    1.671429] cpu 4 spinlock event irq 84
[    1.671429] cpu 5 spinlock event irq 85
[    1.697554] cpu 6 spinlock event irq 86
[    1.757519] cpu 7 spinlock event irq 87
[    1.758405] smp: Brought up 1 node, 8 CPUs
...
[    2.549234] sched_clock: Marking stable (1444006278, 1105086856)->(145865443398, -143316350264)

i.e. ~143 seconds of real time hidden at the SMP bringup stage (~15-20s per secondary vCPU). Scales linearly: with 2 vCPUs the correction is ~15s.

Culprit: console=ttyS0 in the cloud image’s default kernel cmdline (/etc/default/grub.d/50-cloudimg-settings.cfg). Early boot printk output is written synchronously to the emulated 16550 UART; every byte is an I/O port access = VM exit. The verbose early boot output serializes around AP bringup with frozen clocks. An ISO-installed 26.04 on an identical VM config (same platform flags, same kernel 7.0.0-29) doesn’t have ttyS0 in cmdline and boots ~7x faster through this phase.

Fix / proof:

sed -i 's/console=tty1 console=ttyS0/console=tty1/' /etc/default/grub /etc/default/grub.d/50-cloudimg-settings.cfg
update-grub

After reboot the same VM shows sched_clock ... -16146235182 — down from 143s to 16s. Tested on both amd64 and amd64v3 builds, identical results.

Remaining ~2s per vCPU seems to be the baseline UEFI/Xen-timer overhead others have reported — still there, but tolerable.

Environment: XCP-ng 8.3 (fully patched), Xen 4.17, pool of Xeon Gold 6342/6348, guests: Ubuntu 26.04 kernel 7.0.0-29-generic, device-model qemu-upstream-uefi.

For cloud-init templates the workaround is a runcmd in the cloud config applying the sed above.

[dvinni]

Update: possibly related to the TSC regression tracked in
this thread
(kernel 6.12.5+, upstream fix pending) — the symptom signature matches (slow “installing Xen timer” on UEFI only), though that thread’s repros are all AMD while my pool is Intel Ice Lake. The remaining ~2s/vCPU after the ttyS0 fix may be that underlying issue. Unlike the tsc_mode=2 workaround, removing ttyS0 keeps live migration intact.

[poddingue]

Welcome to the forum, and thanks for writing this up instead of just fixing it on your own box.

You already found the thread I was going to point you at. You're ahead of me there.
What happened in it after the part you read: Olivier bisected the problem down to Linux 6.12.5, and Teddy linked an upstream fix that landed in tip (f24df84cbe05), with no ETA for when distros pick it up. The tsc_mode=2 plus nomigrate workaround came out of that, and it carries the live migration cost you already spotted.

One thing I'd like to know, and I don't think anyone has tried it on your side. Does the leftover couple of seconds per vCPU go away if you set tsc_mode=2 on one of those VMs? If it does, that would tie your case to the same regression rather than to a second thing.

I'm not deep enough in the timer code to tell you whether the ttyS0 amplification and the TSC regression are the same mechanism or two things stacking. Probably worth a mention to @Team-Hypervisor-Kernel so someone who is can have a look.

[dvinni]

@poddingue Tested as requested: on a VM with the ttyS0 fix already applied (leftover was ~16s / ~2s per vCPU), setting tsc_mode=2 + nomigrate=true brings the sched_clock correction down to ~0.3s:

[    0.281587] sched_clock: Marking stable (281005908, 562301)->(625590665, -344022456)

vs ~-16s without it (reverted after the test). So it's the same regression, now also confirmed on Intel (Ice Lake, Xeon Gold 6342/6348) — with console=ttyS0 acting as a ~7x amplifier on top, presumably because every early printk to the emulated UART goes through the affected clock path.

[poddingue]

That settles it, thanks for running it. 0.3s against 16s on the same VM is a cleaner answer than I expected.

I read the commit Teddy linked, and it makes your "same thing or two things" question look like one thing: f24df84cbe05 registers the jiffies clocksource before it gets used, because until then its max delta sits at zero and reads clamp to zero, so time stops advancing. If that's the right read, console=ttyS0 isn't a second bug, it's just more wall clock spent inside the window where the clock is stuck.
Take it with a grain of salt, that's me reading a commit message rather than testing anything, so don't take it from me.

More useful, maybe: I went and checked kernel/time/jiffies.c across the stable branches, and the fix is in 6.12.97+, the current 6.18.y, 7.1.4+ and 7.2, but not in 7.0.y, which isn't on kernel.org's maintained list any more. So on your 7.0.0-29 it looks like it has to come from Canonical rather than from an upstream stable update.
Nothing in the patch is CPU-vendor specific either, for what that's worth.

[dvinni]

@poddingue Good news on that front — Canonical already picked it up: the fix ("time/jiffies: Register jiffies clocksource before usage") is in kernel 7.0.0-31, currently sitting in resolute-proposed as part of the 2026.08.03 SRU cycle. So it should reach -updates around early September without needing an upstream 7.0.y backport.

Your reading of the commit matches what I see too — one stalled clock, two consumers (AP bringup waits + printk/UART delays), not two separate bugs.

Removing console=ttyS0 from cloud images is still worth keeping in cloud-init templates though: verbose early printk to an emulated 16550 is pointless overhead on any kernel, and boot logs remain available via dmesg anyway.

[poddingue]

Update, and dvinni beat me to most of it: Canonical already has the fix in 7.0.0-31, and his SRU timing is better than anything I had.

Two things I can still add from a lab box. First, 7.0.0-30 does not carry the patch, and it only landed in resolute-updates this morning, so an apt upgrade today leaves you where you were.

Second, on keeping the ttyS0 removal in cloud-init templates, I was wrong to imply -31 makes that pointless. Same VM on 7.0.0-31, three boots each way, systemd-analyze:

console=tty1 console=ttyS0    10.95 / 11.14 / 11.23 s
console=tty1                   7.40 /  7.38 /  7.47 s

So the serial console is still worth about 3.7s of an 11s boot, spread across kernel, initrd and userspace rather than sitting in any one stage. That's a 6 vCPU VM on a little i5 lab box, so don't expect the same figure on your Xeons, but it's independent of the timekeeping bug, which is the part you had right and I talked past.

I also need to withdraw the wall-clock numbers from the first version of this post. I was timing reboot to sshd, and when I ran the same config twice it gave me 7s and 31s, so it wasn't measuring boot. Probably the network coming up rather than the boot itself, though I didn't chase it. The sched_clock and systemd-analyze figures held across every run.

Still wouldn't put a proposed kernel on a production pool on my say-so. Until it promotes, your ttyS0 change looks like the cheaper of the two workarounds.