I upgraded a pool which was affected from 8.1 to 8.2 this weekend and installed the driver on one of the Hosts. Its a little early, but as you can see, there seems to be a difference in the memory usage:
Stock Driver
Stromis Driver:
One can allready see a constanty, slowly growing mem-usage in "small steps" on the Server with the stock driver, wheras the server with stormis driver seems to be stable.
Today another customer called:
He had a host (pool master) with 16GB Dom0 mem and uptime of 119 days.
Currently all my affected Systems were using megaraid_sas and iscsi and 10g intel nics.
megaraid_sas is found in @MrMike and @inaki-martinez mods too.
This is the customers lsmod:
Module Size Used by
tun 49152 0
ebtable_filter 16384 0
ebtables 36864 1 ebtable_filter
nls_utf8 16384 0
cifs 929792 0
ccm 20480 0
fscache 380928 1 cifs
iscsi_tcp 20480 16
libiscsi_tcp 28672 1 iscsi_tcp
libiscsi 61440 2 libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi 110592 3 iscsi_tcp,libiscsi
bonding 176128 0
bridge 196608 1 bonding
8021q 40960 0
garp 16384 1 8021q
mrp 20480 1 8021q
stp 16384 2 bridge,garp
llc 16384 3 bridge,stp,garp
ipt_REJECT 16384 3
nf_reject_ipv4 16384 1 ipt_REJECT
xt_tcpudp 16384 8
xt_multiport 16384 1
xt_conntrack 16384 5
nf_conntrack 163840 1 xt_conntrack
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 1 nf_conntrack
iptable_filter 16384 1
dm_multipath 32768 0
sunrpc 413696 1
sb_edac 24576 0
intel_powerclamp 16384 0
crct10dif_pclmul 16384 0
crc32_pclmul 16384 0
ghash_clmulni_intel 16384 0
pcbc 16384 0
aesni_intel 200704 0
aes_x86_64 20480 1 aesni_intel
crypto_simd 16384 1 aesni_intel
cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel
glue_helper 16384 1 aesni_intel
dm_mod 151552 285 dm_multipath
ipmi_si 65536 0
ipmi_devintf 20480 0
intel_rapl_perf 16384 0
ipmi_msghandler 61440 2 ipmi_devintf,ipmi_si
i2c_i801 28672 0
sg 40960 0
lpc_ich 28672 0
acpi_power_meter 20480 0
ip_tables 28672 2 iptable_filter
x_tables 45056 7 ebtables,xt_conntrack,iptable_filter,xt_multiport,xt_tcpudp,ipt_REJECT,ip_tables
hid_generic 16384 0
usbhid 57344 0
hid 122880 2 usbhid,hid_generic
sd_mod 53248 9
isci 163840 0
ahci 40960 0
libsas 86016 1 isci
libahci 40960 1 ahci
scsi_transport_sas 45056 2 isci,libsas
xhci_pci 16384 0
ehci_pci 16384 0
igb 233472 0
libata 274432 3 libahci,ahci,libsas
ehci_hcd 90112 1 ehci_pci
xhci_hcd 258048 1 xhci_pci
e1000e 286720 0
megaraid_sas 167936 12
scsi_dh_rdac 16384 0
scsi_dh_hp_sw 16384 0
scsi_dh_emc 16384 0
scsi_dh_alua 20480 1
scsi_mod 253952 15 isci,scsi_dh_emc,scsi_transport_sas,sd_mod,dm_multipath,scsi_transport_iscsi,scsi_dh_alua,iscsi_tcp,libsas,libiscsi,megaraid_sas,libat a,sg,scsi_dh_rdac,scsi_dh_hp_sw
ipv6 548864 545 bridge
crc_ccitt 16384 1 ipv6
Don`t restart openvswitch, if you have active iSCSI storage attached.
@petr-bena Thanks.
I can confirm: Until now everything is stable for us, too. ( with nmi=dom0 )
@olivierlambert Since i have only production-servers with the affected hardware ATM, i cant test the 8.1 beta right now. But after relase i will try 8.1 final. Do you think there is a real chance that this error wont appear in 8.1 stock? Or should I do the same change?
Hi!
@petr-bena did you have crashes since your change nmi=dom0 ?
We have a similar problem.
There are 4 servers in different locations, two standalone, two of them in pools, all with the same hardware:
Supermicro X11SRA-RF Version: 1.02
and
Intel(R) Xeon(R) W-2145 CPU
We tried all BIOS Versions and a lot off different settings.
Two of them are runnig XCP 7.6 and have uptimes of 143 and 160 days. No Problems at all.
Two of them are running XCP 8.0 and crash regulary between 2 or 30 days, everytime with the same error.
NMI - PCI system error (SERR)
The crash is more likely to happen, if we produce high IO and/or network load on those hosts.
We suspected a hardware error, so we took one of those crashing servers to our workshop and testet it for almost two weeks with Prime95 and Memtest86 and other things that came in mind.
We were not able to produce any crash. Neither were we able to detect any errors.
We put this particular server back in production and it crashed within the first hours while we were migrating some VMs back to him. (with Storage Migration)
So i think, it has something to do with XCP-ng 8.0.
I will try the change nmi=dom0 next.
(XEN) [395218.940883]
(XEN) [395218.940886]
(XEN) [395218.940886] NMI - PCI system error (SERR)
(XEN) [395218.940889] ----[ Xen-4.11.1-7.8.xcpng8.0 x86_64 debug=n Not tainted ]----
(XEN) [395218.940889] CPU: 0
(XEN) [395218.940890] RIP: e008:[<ffff82d0802c6d38>] mwait_idle_with_hints+0xf8/0x160
(XEN) [395218.940894] RFLAGS: 0000000000000046 CONTEXT: hypervisor
(XEN) [395218.940896] rax: 0000000000000001 rbx: 000167730fc96b09 rcx: 0000000000000001
(XEN) [395218.940897] rdx: 0000000000000000 rsi: ffff83006f667ef8 rdi: ffff83006f667fff
(XEN) [395218.940898] rbp: 0000000000000000 rsp: ffff83006f667e00 r8: 0000000000000048
(XEN) [395218.940899] r9: 000530dec0dbea3e r10: 0000000000000008 r11: ffff83207cac1a68
(XEN) [395218.940900] r12: 0000000000000000 r13: 0000000000000001 r14: 0000000000000001
(XEN) [395218.940902] r15: ffff82d080573d00 cr0: 0000000080050033 cr4: 0000000000362660
(XEN) [395218.940903] cr3: 00000012c899a000 cr2: ffffe783981eb000
(XEN) [395218.940904] fsb: 0000000000000000 gsb: ffff88827bf40000 gss: 0000000000000000
(XEN) [395218.940906] ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) [395218.940908] Xen code around <ffff82d0802c6d38> (mwait_idle_with_hints+0xf8/0x160):
(XEN) [395218.940908] 89 f0 44 89 e9 0f 01 c9 <0f> b6 47 f5 80 a6 fd 00 00 00 fe 44 89 c1 0f 30
(XEN) [395218.940912] Xen stack trace from rsp=ffff83006f667e00:
(XEN) [395218.940913] ffff83207cac4f08 0000000000000000 ffff83207cac4e90 ffff82d080573d00
(XEN) [395218.940915] ffff82d0805baa50 ffff82d080592b20 ffff83207cac4f08 ffff82d0802ccd07
(XEN) [395218.940916] 000167730faf9015 0000000100000002 00000108000004c9 0000000000000000
(XEN) [395218.940918] 0000000000000000 ffff82d08035b43e ffff83006f7fc000 ffffffffffffffff
(XEN) [395218.940919] ffff82d08035b400 ffff82d080573d00 ffff82d0805baa50 0000000000000000
(XEN) [395218.940921] 0000000000000000 ffff82d080592b20 ffff83006f667fff ffff82d08026e505
(XEN) [395218.940922] ffff83006f7fc000 ffff83006f7fc000 ffff83006f7bf000 ffff83207cb69000
(XEN) [395218.940924] 00000000ffffffff ffff8320246cc000 ffff82d080592b20 ffff88827ae3d700
(XEN) [395218.940926] ffff88827ae3d700 0000000000000000 0000000000000000 0000000000000005
(XEN) [395218.940927] ffff88827ae3d700 0000000000000246 ffffc9004106b930 0000000000000000
(XEN) [395218.940928] 000000000001ca00 0000000000000000 ffffffff810013aa ffffffff8203c190
(XEN) [395218.940930] 0000000000000000 0000000000000001 0000010000000000 ffffffff810013aa
(XEN) [395218.940931] 000000000000e033 0000000000000246 ffffc90040113eb0 000000000000e02b
(XEN) [395218.940933] 6f5b7c2b6f667fe0 6f5b7cae00097f76 6f5b7da200000000 6f5b79516f667fe0
(XEN) [395218.940934] 0000e01000000000 ffff83006f7fc000 0000000000000000 0000000000362660
(XEN) [395218.940936] 0000000000000000 800000207caef002 0000070100000000 6f5b883e00097f00
(XEN) [395218.940938] Xen call trace:
(XEN) [395218.940939] [<ffff82d0802c6d38>] mwait_idle_with_hints+0xf8/0x160
(XEN) [395218.940942] [<ffff82d0802ccd07>] mwait-idle.c#mwait_idle+0x337/0x3d0
(XEN) [395218.940945] [<ffff82d08035b43e>] lstar_enter+0xae/0x120
(XEN) [395218.940946] [<ffff82d08035b400>] lstar_enter+0x70/0x120
(XEN) [395218.940950] [<ffff82d08026e505>] domain.c#idle_loop+0x85/0xb0
(XEN) [395218.940951]
(XEN) [395218.940952]
(XEN) [395218.940953] ****************************************
(XEN) [395218.940953] Panic on CPU 0:
(XEN) [395218.940954] FATAL TRAP: vector = 2 (nmi)
(XEN) [395218.940955] [error_code=0000] , IN INTERRUPT CONTEXT
(XEN) [395218.940955] ****************************************
(XEN) [395218.940956]
(XEN) [395218.940956] Reboot in five seconds...
(XEN) [395218.940958] Executing kexec image on cpu0
(XEN) [395218.941963] Shot down all CPUs