Ubuntu cloud images on XCP-ng 8.3 UEFI: ~15s per secondary vCPU at boot, caused by console=ttyS0

[dvinni]

Sharing a debugging result that may explain some of the “slow UEFI VM boot” reports (e.g. in the XO 6 feedback thread).

Symptom: Ubuntu 26.04 cloud image (cloud-init template, UEFI, 8 vCPU) takes ~1.5 min of black screen before any kernel output. OVMF phase is fast (7s per qemu-dm log, ExitBootServices OK), and systemd-analyze claims ~8s total — yet wall clock says otherwise.

Root cause found via sched_clock: early printk timestamps freeze during smp: Bringing up secondary CPUs / installing Xen timer for CPU N, and the correction shows up later:

[    1.671429] smp: Bringing up secondary CPUs ...
[    1.671429] installing Xen timer for CPU 1
[    1.671429] smpboot: x86: Booting SMP configuration:
[    1.671429] .... node  #0, CPUs:      #1
[    1.671429] installing Xen timer for CPU 2
[    1.671429]  #2
[    1.671429] installing Xen timer for CPU 3
[    1.671429]  #3
[    1.671429] installing Xen timer for CPU 4
[    1.671429]  #4
[    1.671429] installing Xen timer for CPU 5
[    1.671429]  #5
[    1.671429] installing Xen timer for CPU 6
[    1.671429]  #6
[    1.671429] installing Xen timer for CPU 7
[    1.671429]  #7
[    1.671429] cpu 1 spinlock event irq 81
[    1.671429] cpu 2 spinlock event irq 82
[    1.671429] cpu 3 spinlock event irq 83
[    1.671429] cpu 4 spinlock event irq 84
[    1.671429] cpu 5 spinlock event irq 85
[    1.697554] cpu 6 spinlock event irq 86
[    1.757519] cpu 7 spinlock event irq 87
[    1.758405] smp: Brought up 1 node, 8 CPUs
...
[    2.549234] sched_clock: Marking stable (1444006278, 1105086856)->(145865443398, -143316350264)

i.e. ~143 seconds of real time hidden at the SMP bringup stage (~15-20s per secondary vCPU). Scales linearly: with 2 vCPUs the correction is ~15s.

Culprit: console=ttyS0 in the cloud image’s default kernel cmdline (/etc/default/grub.d/50-cloudimg-settings.cfg). Early boot printk output is written synchronously to the emulated 16550 UART; every byte is an I/O port access = VM exit. The verbose early boot output serializes around AP bringup with frozen clocks. An ISO-installed 26.04 on an identical VM config (same platform flags, same kernel 7.0.0-29) doesn’t have ttyS0 in cmdline and boots ~7x faster through this phase.

Fix / proof:

sed -i 's/console=tty1 console=ttyS0/console=tty1/' /etc/default/grub /etc/default/grub.d/50-cloudimg-settings.cfg
update-grub

After reboot the same VM shows sched_clock ... -16146235182 — down from 143s to 16s. Tested on both amd64 and amd64v3 builds, identical results.

Remaining ~2s per vCPU seems to be the baseline UEFI/Xen-timer overhead others have reported — still there, but tolerable.

Environment: XCP-ng 8.3 (fully patched), Xen 4.17, pool of Xeon Gold 6342/6348, guests: Ubuntu 26.04 kernel 7.0.0-29-generic, device-model qemu-upstream-uefi.

For cloud-init templates the workaround is a runcmd in the cloud config applying the sed above.

[dvinni]

Update: possibly related to the TSC regression tracked in
this thread
(kernel 6.12.5+, upstream fix pending) — the symptom signature matches (slow “installing Xen timer” on UEFI only), though that thread’s repros are all AMD while my pool is Intel Ice Lake. The remaining ~2s/vCPU after the ttyS0 fix may be that underlying issue. Unlike the tsc_mode=2 workaround, removing ttyS0 keeps live migration intact.

[poddingue]

Welcome to the forum, and thanks for writing this up instead of just fixing it on your own box.

You already found the thread I was going to point you at. You're ahead of me there.
What happened in it after the part you read: Olivier bisected the problem down to Linux 6.12.5, and Teddy linked an upstream fix that landed in tip (f24df84cbe05), with no ETA for when distros pick it up. The tsc_mode=2 plus nomigrate workaround came out of that, and it carries the live migration cost you already spotted.

One thing I'd like to know, and I don't think anyone has tried it on your side. Does the leftover couple of seconds per vCPU go away if you set tsc_mode=2 on one of those VMs? If it does, that would tie your case to the same regression rather than to a second thing.

I'm not deep enough in the timer code to tell you whether the ttyS0 amplification and the TSC regression are the same mechanism or two things stacking. Probably worth a mention to @Team-Hypervisor-Kernel so someone who is can have a look.