fdrcrtl

fdrcrtl

@fdrcrtl @andSmv @olivierlambert
Bump, ty

fdrcrtl

Good morning, any update on this?
Meanwhile 60+ days stable with max_cstate=5 cpufreq=xen:powersave

fdrcrtl

Right @andSmv , what i've found so far

Due wrong voltage reporting in kernel < 5.12, offset voltage had to be higher
Implementing ZenStates may can help https://forum.level1techs.com/t/overclock-your-ryzen-cpu-from-linux/126025
Some success from AMD forum: https://community.amd.com/t5/processors/ryzen-5900x-system-constantly-crashing-restarting-whea-logger-id/td-p/423321/page/84
Some kernel patches neede for ryzen 5000 series: https://unix.stackexchange.com/questions/628222/what-changes-had-to-be-made-to-linux-kernel-in-order-to-support-ryzen-5000-serie

Dont' know if can help but I've added max_cstate=5 and cpufreq=xen:powersave to limit CPU usage and reduce power requirement. Those settings will be system-wide or only to xen?

fdrcrtl

Thanks for the clarification @olivierlambert, just seen in the docs: Citrix Hypervisor 8.2, Base version of CentOS in dom0: 7.5, Xen 4.13.1 + patches, Kernel 4.19 + patches

Just want to give more info to the support team! Anyway from hetzner perspective is a negative point. Just for info, amd microcode is installed by default? Now the server is under testing, home they find something hw related

Update
Unfortunately test completed without any errors

Your server finished the hardware check test without any hardware related issues. We boot the server back to the installed System. As we recommended try to use kernel version at least 5.1.

Summary of the test:

-----------------%<-----------------
DMESG: Ok
CPUFREQ-CHECK: Ok
STRESSTEST-CPU-TEMP: Ok
FANCHECK: Ok
STRESSTEST: Ok
MCE-CHECK: Ok

HDDTEST S64HNE0T******: Ok
HDDTEST S64HNE0T******: Ok

-----------------%<-----------------

fdrcrtl

Update before starting hw test by hetzner, they said "Please note that this server is 5000 series ryzen and it needs at least Linux kernel version 5.1 to run smoothly as it gets proper support from kernel version 5.12 and above. We have seen many problem from customers running kernel version below 5.1"

Deep into the rabbit hole: https://bugzilla.kernel.org/show_bug.cgi?id=212087 - As xcpng running on 4.19 ..

fdrcrtl

@andSmv three crash in a row just now!
grub.cfg: multiboot2 /boot/xen.gz dom0_mem=4096M,max:4096M watchdog ucode=scan dom0_max_vcpus=4 dom0_vcpus_pin ler=true cpufreq=xen:performance max_cstate=1 iommu=0 crashkernel=256M,below=4G console=vga vga=mode-0x0311

xen.log:(XEN) [ 711.242947] Panic on CPU 14:
xen.log:(XEN) [ 711.242948] FATAL TRAP: vector = 6 (invalid opcode)

xen.log:(XEN) [ 854.061272] Panic on CPU 8:
xen.log:(XEN) [ 854.061273] FATAL TRAP: vector = 2 (nmi)

xen.log:(XEN) [ 556.104951] Panic on CPU 14:
xen.log:(XEN) [ 556.104951] FATAL TRAP: vector = 6 (invalid opcode)

Dumped crash folder, kdump and .map files (where I could find them), what do you need/where to send? I'll powering off the host now for an extended memtest by hetzner

fdrcrtl

@andSmv I'm blown away by your professionalism, thank you!
Today another crash, i'll reverted grub to basic dom0_max_vcpus=4 dom0_vcpus_pin max_cstate=0 and ler=true (hope for another crash within 1-2 days)

I'll schedule a deep check/memtest with Hetzner this weekend to see if they can address this issue, I'll keep you updated!

PS. Are cpufreq=xen:performance max_cstate=1 iommu=0 a good combination for better performance/stability (no hw passthr)

fdrcrtl

As I seen, random as I work on it..

For debug, ask me anything
alt text

fdrcrtl

Thank you @olivierlambert, it's the same host in different time spans. When this happened, I was setting up and playing with an Oracle Linux 8 via console (the other vm is an opnsense)

Grepping some text i can see:

./20220619-112257-UTC/xen.log:(XEN) [73345.480262] FATAL TRAP: vector = 2 (nmi)
./20220619-112257-UTC/xen.log:(XEN) [73345.480262] Panic on CPU 13:

./20220621-145540-UTC/xen.log:(XEN) [  459.021625] FATAL TRAP: vector = 6 (invalid opcode)
./20220621-145540-UTC/xen.log:(XEN) [  459.021625] Panic on CPU 15:

./20220617-133059-UTC/xen.log:(XEN) [ 6286.633327] FATAL TRAP: vector = 2 (nmi)
./20220617-133059-UTC/xen.log:(XEN) [ 6286.633327] Panic on CPU 8:

./20220621-144636-UTC/xen.log:(XEN) [66874.346274] FATAL TRAP: vector = 6 (invalid opcode)
./20220621-144636-UTC/xen.log:(XEN) [66874.346273] Panic on CPU 15:

Dunno if matters but wondering about scaling governor, for now I've edited grub removing cpufreq=xen:performance and setting max_cstate=0 and iommu=0

fdrcrtl

Hi forum, I'm experiencing crash on my AX101 hosted at hetzner (5950X, ecc RAM -tested, no probs) with no load (2VM x 2core 4gb). I was able to find this on /var/crash dumps, how can I investigate more? TY

grub cfg

multiboot2 /boot/xen.gz dom0_mem=4096M,max:4096M watchdog ucode=scan dom0_max_vcpus=4 dom0_vcpus_pin cpufreq=xen:performance max_cstate=1 iommu=0 crashkernel=256M,below=4G console=vga vga=mode-0x0311
	module2 /boot/vmlinuz-4.19-xen root=LABEL=root-ebdyoh ro nolvm hpet=disable rd.auto console=hvc0 console=tty0 vga=785 plymouth.ignore-serial-consoles

(XEN) [73341.747357] ----[ Xen-4.13.4-9.22.2  x86_64  debug=n   Not tainted ]----
(XEN) [73341.747358] CPU:    12
(XEN) [73341.747358] RIP:    e008:[<ffff82d08022cb69>] common/sched_credit.c#csched_unit_wake+0xf9/0x6e0
(XEN) [73341.747362] RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) [73341.747363] rax: ffff831fd31961e0   rbx: ffff82d080598088   rcx: ffff831fd31961e0
(XEN) [73341.747363] rdx: ffff831fd31961e0   rsi: 00000000d318ef30   rdi: ffff831fd31961e0
(XEN) [73341.747364] rbp: ffff83202bcad5f8   rsp: ffff831fd39ffc30   r8:  000042b403efae21
(XEN) [73341.747364] r9:  ffff83202bcac000   r10: 000042b434cc56d9   r11: 0000000000000001
(XEN) [73341.747365] r12: ffff82d0805bc300   r13: ffff82d08059704c   r14: ffff831fd318ef30
(XEN) [73341.747365] r15: ffff831fd318ee60   cr0: 000000008005003b   cr4: 00000000003506e0
(XEN) [73341.747366] cr3: 00000000cce7b000   cr2: ffff82d092cd836c
(XEN) [73341.747366] fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) [73341.747366] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) [73345.479207] Watchdog timer detects that CPU13 is stuck!
(XEN) [73345.479208] ----[ Xen-4.13.4-9.22.2  x86_64  debug=n   Not tainted ]----
(XEN) [73345.479208] CPU:    13
(XEN) [73345.479209] RIP:    e008:[<ffff82d08023faf5>] _spin_lock_recursive+0x35/0x70
(XEN) [73345.479210] RFLAGS: 0000000000000002   CONTEXT: hypervisor
(XEN) [73345.479211] rax: 00000000000044aa   rbx: 0000000000000096   rcx: 000000000000000d
(XEN) [73345.479211] rdx: 00000000000044ab   rsi: ffff82d0803c44aa   rdi: ffff82d08047eb18
(XEN) [73345.479212] rbp: 0000000000000000   rsp: ffff831fd39efb70   r8:  0000000000000000
(XEN) [73345.479212] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [73345.479213] r12: 0000000000000000   r13: 0000000000000000   r14: ffff831fd39efc28
(XEN) [73345.479213] r15: ffff82d092cd836c   cr0: 000000008005003b   cr4: 00000000003506e0
(XEN) [73345.479214] cr3: 00000000cce7b000   cr2: ffff82d092cd836c
(XEN) [73345.479214] fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) [73345.479214] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) [73345.479216] Xen code around <ffff82d08023faf5> (_spin_lock_recursive+0x35/0x70):
(XEN) [73345.479216]  74 0b 90 f3 90 66 8b 07 <66> 39 c2 75 f6 48 89 e2 48 8d 05 48 75 35 00 48
(XEN) [73345.479218] Xen stack trace from rsp=ffff831fd39efb70:
(XEN) [73345.479218]    ffff82d08024f140 ffff831fd39efc28 ffff82d0802ae75e 0000000000000000
(XEN) [73345.479219]    0000000000000000 0000000000000000 ffff82d0802af837 ffff82d000000000
(XEN) [73345.479220]    00000000d3193210 0000000000000000 0000000000000087 0000000000000000
(XEN) [73345.479221]    00000000cce77063 0000000000000000 0000000000000096 000000000000000d
(XEN) [73345.479221]    ffff831fd39f3000 0000000000000000 0000000000000000 0000000000000000
(XEN) [73345.479222]    ffff831fd39effff 0000000000000000 ffff82d080370a17 ffff831206417da0
(XEN) [73345.479223]    ffff831206417e90 ffff82d08059704c ffff82d0805bc300 ffff831fd37298f8
(XEN) [73345.479223]    ffff82d080598088 ffff82d080597e28 000042b43831e5c9 ffff83121d04c000
(XEN) [73345.479224]    000042b403efae15 ffff831fd31931f0 ffff831fd31931f0 ffff831fd31931f0
(XEN) [73345.479225]    0000000006417e90 ffff831fd31931f0 0000000e00000000 ffff82d08022cb69
(XEN) [73345.479225]    000000000000e008 0000000000010046 ffff831fd39efcd0 0000000000000000
(XEN) [73345.479226]    0000000000000000 ffff831206417da0 0000000000000010 ffff82d080598088
(XEN) [73345.479227]    0000000001c9c380 000042b432d43ad5 ffff831fd31931f0 ffff83131bd83db8
(XEN) [73345.479228]    0000000100000000 ffff82d080270a80 ffff82d080288d1c ffff831206402068
(XEN) [73345.479228]    ffff831fd37299a0 ffff831206402000 0000000000000000 ffff831fd37298e0
(XEN) [73345.479229]    ffff83121d04c000 0000000000000000 0000000080000000 0000000000000000
(XEN) [73345.479230]    0000000000000000 ffff831fd39f3000 00000000003506e0 0000000000000046
(XEN) [73345.479230]    0000000000000000 ffff82d08027f216 000000000000000d ffff831206402000
(XEN) [73345.479231]    ffff831fd37298f8 ffff83121d04cb84 ffff82d08059704c ffff82d080598088
(XEN) [73345.479232]    ffff831206417da0 ffff82d08023af68 0000000000000202 ffff83121d04c000
(XEN) [73345.479232] Xen call trace:
(XEN) [73345.479233]    [<ffff82d08023faf5>] R _spin_lock_recursive+0x35/0x70
(XEN) [73345.479234]    [<ffff82d08024f140>] S console_lock_recursive_irqsave+0x10/0x20
(XEN) [73345.479236]    [<ffff82d0802ae75e>] S show_execution_state+0xe/0x40
(XEN) [73345.479237]    [<ffff82d0802af837>] S do_page_fault+0x657/0x760
(XEN) [73345.479239]    [<ffff82d080370a17>] S x86_64/entry.S#handle_exception_saved+0x68/0x94
(XEN) [73345.479240]    [<ffff82d08022cb69>] S common/sched_credit.c#csched_unit_wake+0xf9/0x6e0
(XEN) [73345.479242]    [<ffff82d080270a80>] S apic_timer_interrupt+0/0x30
(XEN) [73345.479243]    [<ffff82d080288d1c>] S do_IRQ+0x29c/0x720
(XEN) [73345.479244]    [<ffff82d08027f216>] S arch/x86/i387.c#_vcpu_save_fpu+0x86/0x180
(XEN) [73345.479245]    [<ffff82d08023af68>] S vcpu_wake+0x258/0x6c0
(XEN) [73345.479246]    [<ffff82d08020ae96>] S send_guest_vcpu_virq+0xd6/0xe0
(XEN) [73345.479247]    [<ffff82d08027c9f5>] S vcpu_kick+0x15/0x60
(XEN) [73345.479248]    [<ffff82d080241c4d>] S common/tasklet.c#do_tasklet_work+0x6d/0xb0
(XEN) [73345.479249]    [<ffff82d080241ccc>] S common/tasklet.c#tasklet_softirq_action+0x3c/0x70
(XEN) [73345.479250]    [<ffff82d08023ef82>] S common/softirq.c#__do_softirq+0x62/0x90
(XEN) [73345.479251]    [<ffff82d080278a35>] S arch/x86/domain.c#idle_loop+0x65/0xf0
(XEN) [73345.479251]    [<ffff82d0802789d0>] S arch/x86/domain.c#idle_loop+0/0xf0
(XEN) [73345.479251] 
(XEN) [73345.479255] CPU0 @ e008:ffff82d08023f905 (_spin_lock_irqsave+0x25/0x50)
(XEN) [73345.479256] CPU1 @ e008:ffff82d08023f8b5 (_spin_lock_irq+0x25/0x50)
(XEN) [73345.479258] CPU2 @ e008:ffff82d0802a71f0 (flush_area_mask+0xf0/0x140)
(XEN) [73345.479260] CPU3 @ e008:ffff82d08023f865 (_spin_lock+0x25/0x50)
(XEN) [73345.479261] CPU12 @ e008:ffff82d0802e7ca2 (mcheck_cmn_handler+0x302/0x470)
(XEN) [73345.479773] CPU11 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479775] CPU10 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479781] CPU26 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479785] CPU27 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479788] CPU9 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479791] CPU8 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479795] CPU24 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479798] CPU25 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479801] CPU7 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479803] CPU6 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479807] CPU23 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479810] CPU22 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479814] CPU5 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479816] CPU4 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479819] CPU21 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479823] CPU20 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479826] CPU15 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479828] CPU14 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479831] CPU19 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479834] CPU18 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479838] CPU17 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479840] CPU16 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479844] CPU31 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479846] CPU30 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479850] CPU29 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.479852] CPU28 @ e008:ffff82d0802d9c68 (arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x98/0xc0)
(XEN) [73345.480261] 
(XEN) [73345.480261] ****************************************
(XEN) [73345.480262] Panic on CPU 13:
(XEN) [73345.480262] FATAL TRAP: vector = 2 (nmi)
(XEN) [73345.480263] [error_code=0000] , IN INTERRUPT CONTEXT
(XEN) [73345.480263] ****************************************
(XEN) [73345.480263] 
(XEN) [73345.480263] Reboot in five seconds...
(XEN) [73345.480264] Executing kexec image on cpu13
(XEN) [73345.481278] Shot down all CPUs

(XEN) [  459.021596] r15: ffff82d0805bc300   cr0: 000000008005003b   cr4: 00000000003506e0
(XEN) [  459.021596] cr3: 00000012064cd000   cr2: 0000000800418c60
(XEN) [  459.021596] fsb: 0000000801006120   gsb: ffffffff82610000   gss: 0000000000000000
(XEN) [  459.021597] ds: 0000   es: 0000   fs: 0013   gs: 001b   ss: 0000   cs: e008
(XEN) [  459.021598] Xen code around <ffff82d08027bdd2> (context_switch+0x862/0xdd0):
(XEN) [  459.021599]  d2 e9 1d fa ff ff f3 48 <0f> ae d8 eb ce 0f 01 f8 f3 48 0f ae d8 0f 01 f8
(XEN) [  459.021601] Xen stack trace from rsp=ffff831fd39c7da8:
(XEN) [  459.021601]    0000000000000000 ffff83120640d060 ffff82d080597160 ffff831fd39d2000
(XEN) [  459.021602]    ffff83120640d000 ffff831fd39e7ec0 0000006adfd1840c ffff83120640d000
(XEN) [  459.021602]    ffff82d0805bc300 ffff82d08023d243 ffff8312064e7f00 000000000000000f
(XEN) [  459.021603]    000000000000000f ffff8312064e7f00 ffff831fd39e7ec0 ffff82d08023da94
(XEN) [  459.021604]    ffff831f00000001 ffff831fd39e7e00 0000000000000200 ffff831fd39e7e18
(XEN) [  459.021604]    ffff831206409000 ffff82d0803087c5 0000000000000078 ffff82d080308873
(XEN) [  459.021605]    ffff83120640d000 ffff83120640d000 ffff83122de4a000 ffff82d080300440
(XEN) [  459.021606]    ffff831206409000 00000000ffffffff ffffffffffffffff ffff831fd39c7fff
(XEN) [  459.021607]    ffff82d08059db00 0000000000000000 0000000000000000 ffff82d08023ef82
(XEN) [  459.021607]    ffff83120640d000 ffff83120640d000 0000000000000000 0000000000000000
(XEN) [  459.021608]    0000000000000000 ffff82d0803144fa 0000000000000002 0000013f46a2a9c6
(XEN) [  459.021608]    0000000000000001 fffff8000389ac28 fffffe0007388dd0 fffff8000389ac00
(XEN) [  459.021609]    0000000000000001 00000000ffffffff fffff80003970258 0000006ac6495968
(XEN) [  459.021610]    0000013f46a2a9c6 0000000000000000 0000013f00000000 0000000000000000
(XEN) [  459.021610]    0000000000000000 0000000000000000 ffffffff8110d476 0000000000000000
(XEN) [  459.021611]    0000000000000246 fffffe0007388dd0 0000000000000000 0000000000000000
(XEN) [  459.021612]    0000000000000000 0000000000000000 0000000000000000 0000e0100000000f
(XEN) [  459.021612]    ffff83120640d000 0000004f53436000 00000000003506e0 0000000000000000
(XEN) [  459.021613]    0000000000000000 0000000000000000 0000000000000000
(XEN) [  459.021613] Xen call trace:
(XEN) [  459.021614]    [<ffff82d08027bdd2>] R context_switch+0x862/0xdd0
(XEN) [  459.021616]    [<ffff82d08023d243>] S common/schedule.c#sched_context_switch+0x53/0x170
(XEN) [  459.021617]    [<ffff82d08023da94>] S common/schedule.c#schedule+0x1e4/0x240
(XEN) [  459.021618]    [<ffff82d0803087c5>] S vlapic_accept_pic_intr+0x45/0x80
(XEN) [  459.021619]    [<ffff82d080308873>] S vlapic_has_pending_irq+0x53/0x120
(XEN) [  459.021621]    [<ffff82d080300440>] S hvm_vcpu_has_pending_irq+0x60/0xb0
(XEN) [  459.021622]    [<ffff82d08023ef82>] S common/softirq.c#__do_softirq+0x62/0x90
(XEN) [  459.021623]    [<ffff82d0803144fa>] S svm_stgi_label+0x13/0x18
(XEN) [  459.021623] 
(XEN) [  459.021624] 
(XEN) [  459.021625] ****************************************
(XEN) [  459.021625] Panic on CPU 15:
(XEN) [  459.021625] FATAL TRAP: vector = 6 (invalid opcode)
(XEN) [  459.021626] ****************************************
(XEN) [  459.021626] 
(XEN) [  459.021626] Reboot in five seconds...
(XEN) [  459.021627] Executing kexec image on cpu15
(XEN) [  459.022640] Shot down all CPUs

fdrcrtl

@fdrcrtl

Latest posts made by fdrcrtl