VM trying to add serial console to examine boot process.

[kevdog]

So long story short is that I was working with an arch-linux vm within xcp-ng and trying to debug why ZFSBootMenu wont work to boot my VM with my / partition on zfs. If interested the details or documented here: https://www.reddit.com/r/zfs/comments/1p1veqf/damn_struggling_to_get_zfsbootmenu_to_work/

Anyway one of the proposed methods of trying to debug the problem of why the ZFSBootMenu EFI wont boot the kernel was to try to add a serial port to the VM to gather output. Adding a serial port to a VM was something new to me and I actually tried two methods:

1. SSH'd into the DOM image, I tried running the commands:

# xl vm-list  ---- This got me the VM id #
# xl console -t serial <ID #>

This didn't seem to produce any output so I went another method was to add a serial port that dumped it's output to a tcp port:

# xe vm-param-add uuid=<uuid> param-name=platform hvm_serial=tcp::7001,server,nodelay

I then from a remote host tried to see to listed on tcp port 7001:

#nc -v <ip_address> 7001

and yes that didn't work either in terms of producing any output.

I am going about this the correct way in terms of trying to attach a serial port? For the record if anyone has worked with ZFSBootMenu my parameters I'm adjusting here are the following:

# zfs set org.zfsbootmenu:commandline="spl.spl_hostid=$(hostid) loglevel=7 rw console=ttyS0" tank/sys/arch/ROOT/default

[olivierlambert]

Question for @Team-Hypervisor-Kernel maybe or anyone with guest console knowledge

[dinhngtu]

@kevdog Try adding console=ttyS0 earlyprintk=serial,ttyS0. You shouldn't need hvm_serial, xl console should be enough. Make sure that the boot parameters show up correctly in grub.

[kevdog]

@dinhngtu Thanks for suggestions.

I'm really delving into areas I haven't delved into before so thanks for help.
I'm going to include a few screen shots here as I think this is the best way to show what's going on. I'm not sure if you've ever used ZFSBootMenu but there isn't any grub involved, but I understand what you're saying about the kernel command line parameters.

So I after some testing - I want to confirm the serial method does work.

To get serial console, ssh into DOM0:

xl vm-list ---> To get <ID number>
xl console -t serial <ID number>

***I'm not certain if this has anything to do with xcp-ng at all. Learning more about the issue and how zfsbootmenu works -- it's more a wrapper for kexec. ZFSBootMenu does allow for a chroot and from there you can issue a command line. Anyway I booted into the recovery console and typed the following:

kexec -l /boot/vmlinuz-linux-lts --initrd=/boot/initramfs-linux-lts.img --command-line="zfs=tank/sys/arch/ROOT/default loglevel=7 rw console=ttyS0 earlyprintk=serial,ttys0"
kexec -e

Unfortunately probably beyond this forum, I ended up getting a kernel panic:

Invalid physical address chosen!




Physical KASLR disabled: no suitable memory region!

[    0.000000] Linux version 6.12.58-1-lts (linux-lts@archlinux) (gcc (GCC) 15.2.1 20251112, GNU ld (GNU Binutils) 2.45.1) #1 SMP PREEMPT_DYNAMIC Fri, 14 Nov 2025 05:38:58 +0000
[    0.000000] Command line: zfs=tank/sys/arch/ROOT/default noresume rw loglevel=7 console=ttyS0 earlyprintk=serial,ttyS0
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000ee8c0fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000ee8c1000-0x00000000ee8c1fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ee8c2000-0x00000000ee910fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000ee911000-0x00000000ee91afff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000ee91b000-0x00000000ef99afff] usable
[    0.000000] BIOS-e820: [mem 0x00000000ef99b000-0x00000000ef9f2fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ef9f3000-0x00000000ef9fafff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000ef9fb000-0x00000000ef9fefff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000ef9ff000-0x00000000effdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000effdf000-0x00000000efffefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000effff000-0x00000000efffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000fc00b000-0x00000000fedfffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fef00000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000020f7fffff] usable
[    0.000000] random: crng init done
[    0.000000] printk: legacy bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] extended physical RAM map:
[    0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] reserve setup_data: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] reserve setup_data: [mem 0x0000000000100000-0x00000000ee8c0fff] usable
[    0.000000] reserve setup_data: [mem 0x00000000ee8c1000-0x00000000ee8c1fff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000ee8c2000-0x00000000ee910fff] usable
[    0.000000] reserve setup_data: [mem 0x00000000ee911000-0x00000000ee91afff] ACPI data
[    0.000000] reserve setup_data: [mem 0x00000000ee91b000-0x00000000ef99afff] usable
[    0.000000] reserve setup_data: [mem 0x00000000ef99b000-0x00000000ef9f2fff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000ef9f3000-0x00000000ef9fafff] ACPI data
[    0.000000] reserve setup_data: [mem 0x00000000ef9fb000-0x00000000ef9fefff] ACPI NVS
[    0.000000] reserve setup_data: [mem 0x00000000ef9ff000-0x00000000effdefff] usable
[    0.000000] reserve setup_data: [mem 0x00000000effdf000-0x00000000efffefff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000effff000-0x00000000efffffff] usable
[    0.000000] reserve setup_data: [mem 0x00000000fc00b000-0x00000000fedfffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fef00000-0x00000000ffffffff] reserved
[    0.000000] reserve setup_data: [mem 0x0000000100000000-0x000000020f7fa19f] usable
[    0.000000] reserve setup_data: [mem 0x000000020f7fa1a0-0x000000020f7fa21f] usable
[    0.000000] reserve setup_data: [mem 0x000000020f7fa220-0x000000020f7fffff] usable
[    0.000000] efi: EFI v2.7 by EDK II
[    0.000000] efi: ACPI=0xef9fa000 ACPI 2.0=0xef9fa014 SMBIOS=0xef9cb000 MEMATTR=0xee210798 RNG=0xef9f4f18 INITRD=0xee0f0998
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: Xen HVM domU, BIOS 4.17 10/21/2025
[    0.000000] DMI: Memory slots populated: 1/1
[    0.000000] Hypervisor detected: Xen HVM
[    0.000000] Xen version 4.17.
[    0.000000] platform_pci_unplug: Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[    0.000000] platform_pci_unplug: Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[    0.000000] You might have to change the root device
[    0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
[    0.000000] in your root= kernel command line option
[    0.030291] tsc: Fast TSC calibration using PIT
[    0.036454] tsc: Detected 2712.051 MHz processor
[    0.043153] tsc: Detected 2712.000 MHz TSC
[    0.048895] last_pfn = 0x20f800 max_arch_pfn = 0x400000000
[    0.061261] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs
[    0.074487] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Memory KASLR using RDRAND RDTSC...
[    0.086526] x2apic: enabled by BIOS, switching to x2apic ops
[    0.092575] last_pfn = 0xf0000 max_arch_pfn = 0x400000000
[    0.111076] Using GB pages for direct mapping
[    0.118548] Secure boot disabled
[    0.123286] RAMDISK: [mem 0x20a9cd000-0x20b7fffff]
[    0.131788] ACPI: Early table checksum verification disabled
[    0.141163] ACPI: RSDP 0x00000000EF9FA014 000024 (v02 Xen   )
[    0.148182] ACPI: XSDT 0x00000000EF9F90E8 000044 (v01 Xen    HVM      00000000      01000013)
[    0.161363] ACPI: FACP 0x00000000EF9F8000 0000F4 (v04 Xen    HVM      00000000 HVML 00000000)
[    0.174042] ACPI: DSDT 0x00000000EE911000 0092A3 (v02 Xen    HVM      00000000 INTL 20160527)
[    0.186753] ACPI: FACS 0x00000000EF9FE000 000040
[    0.194096] ACPI: APIC 0x00000000EF9F7000 000068 (v02 Xen    HVM      00000000 HVML 00000000)
[    0.207668] ACPI: HPET 0x00000000EF9F6000 000038 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.219189] ACPI: WAET 0x00000000EF9F5000 000028 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.231948] ACPI: Reserving FACP table memory at [mem 0xef9f8000-0xef9f80f3]
[    0.242019] ACPI: Reserving DSDT table memory at [mem 0xee911000-0xee91a2a2]
[    0.251672] ACPI: Reserving FACS table memory at [mem 0xef9fe000-0xef9fe03f]
[    0.261611] ACPI: Reserving APIC table memory at [mem 0xef9f7000-0xef9f7067]
[    0.275065] ACPI: Reserving HPET table memory at [mem 0xef9f6000-0xef9f6037]
[    0.286614] ACPI: Reserving WAET table memory at [mem 0xef9f5000-0xef9f5027]
[    0.297485] APIC: Switched APIC routing to: cluster x2apic
[    0.306415] No NUMA configuration found
[    0.311594] Faking a node at [mem 0x0000000000000000-0x000000020f7fffff]
[    0.321366] NODE_DATA(0) allocated [mem 0x20f7cf340-0x20f7fa07f]
[    0.330997] Zone ranges:
[    0.334797]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.344273]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.353819]   Normal   [mem 0x0000000100000000-0x000000020f7fffff]
[    0.362229]   Device   empty
[    0.367131] Movable zone start for each node
[    0.373999] Early memory node ranges
[    0.378773]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.390474]   node   0: [mem 0x0000000000100000-0x00000000ee8c0fff]
[    0.401444]   node   0: [mem 0x00000000ee8c2000-0x00000000ee910fff]
[    0.409440]   node   0: [mem 0x00000000ee91b000-0x00000000ef99afff]
[    0.417934]   node   0: [mem 0x00000000ef9ff000-0x00000000effdefff]
[    0.427208]   node   0: [mem 0x00000000effff000-0x00000000efffffff]
[    0.435979]   node   0: [mem 0x0000000100000000-0x000000020f7fffff]
[    0.444922] Initmem setup node 0 [mem 0x0000000000001000-0x000000020f7fffff]
[    0.464839] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.664383] On node 0, zone DMA: 96 pages in unavailable ranges
[   11.987775] On node 0, zone DMA32: 1 pages in unavailable ranges
[   11.992954] On node 0, zone DMA32: 10 pages in unavailable ranges
[   12.000827] On node 0, zone DMA32: 100 pages in unavailable ranges
[   12.006762] On node 0, zone DMA32: 32 pages in unavailable ranges
[   12.024634] On node 0, zone Normal: 2048 pages in unavailable ranges
[   12.034858] ACPI: PM-Timer IO Port: 0xb008
[   12.040633] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[   12.050352] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[   12.060030] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[   12.070986] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[   12.080237] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[   12.089886] ACPI: Using ACPI (MADT) for SMP configuration information
[   12.099440] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[   12.107122] TSC deadline timer available
[   12.111961] CPU topo: Max. logical packages:   1
[   12.118536] CPU topo: Max. logical dies:       1
[   12.124758] CPU topo: Max. dies per package:   1
[   12.131268] CPU topo: Max. threads per core:   1
[   12.137904] CPU topo: Num. cores per package:     1
[   12.143768] CPU topo: Num. threads per package:   1
[   12.151213] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs
[   12.159593] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[   12.170508] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[   12.181623] PM: hibernation: Registered nosave memory: [mem 0xee8c1000-0xee8c1fff]
[   12.193092] PM: hibernation: Registered nosave memory: [mem 0xee911000-0xee91afff]
[   12.202372] PM: hibernation: Registered nosave memory: [mem 0xef99b000-0xef9fefff]
[   12.211854] PM: hibernation: Registered nosave memory: [mem 0xeffdf000-0xefffefff]
[   12.223405] PM: hibernation: Registered nosave memory: [mem 0xf0000000-0xffffffff]
[   12.233969] [mem 0xf0000000-0xfc00afff] available for PCI devices
[   12.243240] Booting paravirtualized kernel on Xen HVM
[   12.251069] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6370452778343963 ns
[   12.273490] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[   12.285349] percpu: Embedded 67 pages/cpu s237568 r8192 d28672 u2097152
[   12.295893] Kernel command line: zfs=tank/sys/arch/ROOT/default noresume rw loglevel=7 console=ttyS0 earlyprintk=serial,ttyS0
[   12.314016] Unknown kernel command line parameters "zfs=tank/sys/arch/ROOT/default", will be passed to user space.
[   12.330535] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[   15.191489] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[   15.203342] Fallback order for Node 0: 0
[   15.203347] Built 1 zonelists, mobility grouping on.  Total pages: 2094864
[   15.220969] Policy zone: Normal
[   15.225746] mem auto-init: stack:all(zero), heap alloc:on, heap free:off
[   15.235420] software IO TLB: area num 1.
[   15.267432] BUG: unable to handle page fault for address: 0000000000002fd8
[   15.279495] #PF: supervisor write access in kernel mode
[   15.287760] #PF: error_code(0x0002) - not-present page
[   15.295463] PGD 0 P4D 0
[   15.298645] Oops: Oops: 0002 [#1] PREEMPT SMP PTI
[   15.305787] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.12.58-1-lts #1 4a3b6be8628e50ea24aa79ad254248308dcf5cb0
[   15.321935] Hardware name: Xen HVM domU, BIOS 4.17 10/21/2025
[   15.330517] RIP: 0010:__free_pages_core+0x77/0x220
[   15.337143] Code: 48 83 c0 40 48 39 d0 75 eb 48 8b 45 00 48 89 c2 48 c1 e8 33 48 c1 ea 36 83 e0 07 48 69 c0 c0 06 00 00 48 8b 14 d5 60 a5 6d 94 <f0> 48 01 b4 02 98 00 00 00 41 bc 00 10 00 00 89 d9 48 89 ef 48 2b
[   15.368251] RSP: 0000:ffffffff93a03e20 EFLAGS: 00010002
[   15.376726] RAX: 0000000000002f40 RBX: 0000000000000008 RCX: 0000000000000008
[   15.390585] RDX: 0000000000000000 RSI: 0000000000000100 RDI: fffff6dd40004000
[   15.400332] RBP: fffff6dd40004000 R08: 0000000000100000 R09: ffffffff93a03e58
[   15.410724] R10: 00000000ee8c1000 R11: 0000000000000039 R12: ffffffff93a03e60
[   15.421491] R13: 0000000000000080 R14: 0000000000000200 R15: 0000000000000100
[   15.432930] FS:  0000000000000000(0000) GS:ffff88870f400000(0000) knlGS:0000000000000000
[   15.444655] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   15.453221] CR2: 0000000000002fd8 CR3: 000000020da22001 CR4: 00000000000200b0
[   15.464104] Call Trace:
[   15.467862]  <TASK>
[   15.470559]  memblock_free_all+0x1b8/0x230
[   15.476299]  mem_init+0x1a/0x1d0
[   15.481736]  mm_core_init+0xfb/0x140
[   15.486818]  start_kernel+0x767/0x9e0
[   15.492324]  x86_64_start_reservations+0x24/0x30
[   15.496579]  x86_64_start_kernel+0x98/0xa0
[   15.501183]  common_startup_64+0x13e/0x141
[   15.507019]  </TASK>
[   15.509901] Modules linked in:
[   15.511979] CR2: 0000000000002fd8
[   15.517229] ---[ end trace 0000000000000000 ]---
[   15.522802] RIP: 0010:__free_pages_core+0x77/0x220
[   15.528278] Code: 48 83 c0 40 48 39 d0 75 eb 48 8b 45 00 48 89 c2 48 c1 e8 33 48 c1 ea 36 83 e0 07 48 69 c0 c0 06 00 00 48 8b 14 d5 60 a5 6d 94 <f0> 48 01 b4 02 98 00 00 00 41 bc 00 10 00 00 89 d9 48 89 ef 48 2b
[   15.553390] RSP: 0000:ffffffff93a03e20 EFLAGS: 00010002
[   15.561510] RAX: 0000000000002f40 RBX: 0000000000000008 RCX: 0000000000000008
[   15.571508] RDX: 0000000000000000 RSI: 0000000000000100 RDI: fffff6dd40004000
[   15.581418] RBP: fffff6dd40004000 R08: 0000000000100000 R09: ffffffff93a03e58
[   15.591445] R10: 00000000ee8c1000 R11: 0000000000000039 R12: ffffffff93a03e60
[   15.601914] R13: 0000000000000080 R14: 0000000000000200 R15: 0000000000000100
[   15.612522] FS:  0000000000000000(0000) GS:ffff88870f400000(0000) knlGS:0000000000000000
[   15.624495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   15.633191] CR2: 0000000000002fd8 CR3: 000000020da22001 CR4: 00000000000200b0
[   15.643140] Kernel panic - not syncing: Attempted to kill the idle task!

I guess it's my probably to fix at this point.

What's wierd is if I delete the efibootmgr entry for the zfsbootmenu and then attempt booting using systemd-boot, the system will boot.

[kevdog]

So I actually posted these findings to the ZFSBootMenu Discussion github: https://github.com/zbm-dev/zfsbootmenu/discussions/787

Looks like they have tested ZBM against KVM but not Xen.

Can Xen not handle a kernel being kexec'd?