Hello, no, since I changed this, server is rock solid:

20:59:01 up 136 days, 22:40, 1 user, load average: 0.45, 0.31, 0.36

@fbifido yes, I have 3 CEPH nodes running in separate VM's that have direct passthrough to underlying physical disks. CEPH volume is connected as RBD that forms shared block device on XCP-ng servers. On that shared block device I use LVM.

It's all described here: https://github.com/xcp-ng/xcp/wiki/Ceph-on-XCP-ng-7.5-or-later#lvm-on-rbd

there must be some low level way to do that from dom0

@olivierlambert yes

how does API get these metrics?

@olivierlambert but does that really export CPU utilization? From what I saw it only shows memory utilization. Also how does orchestra get these metrics? There must be some low level way to accomplish this, or is xsconsole also using the API for that?

Hello,

xsconsole and xen orchestra can display host CPU usage, is it possible to get these metrics somehow in shell into a variable? I would like to create some exporter for prometheus so that I can collect utilization metrics of host, not just dom0.

Or perhaps someone already created prometheus exporter for Xen?

Hello, no, since I changed this, server is rock solid:

20:59:01 up 136 days, 22:40, 1 user, load average: 0.45, 0.31, 0.36

@fbifido yes, I have 3 CEPH nodes running in separate VM's that have direct passthrough to underlying physical disks. CEPH volume is connected as RBD that forms shared block device on XCP-ng servers. On that shared block device I use LVM.

It's all described here: https://github.com/xcp-ng/xcp/wiki/Ceph-on-XCP-ng-7.5-or-later#lvm-on-rbd

@maxcuttins current kernel supports RBD, which feature are you missing? I am using kernel RBD and it works OK

It is running one of production CEPH nodes, but it if crashes, CEPH will transparently failover. VMs running there are just for backup and non-prod stuff, if I knew which HW is causing it, I would replace it, but this message isn't very clear on what is really going on.

Other than that everything is running OK, so far no crash...

I don't know, I added nmi=dom0 to xen boot parameter line and so far it didn't crash, I've seen some weird things in dmesg on dom0 which I don't know if are related, but it didn't crash. So far it's working. I would rather think that this is somehow connected to C-states or something similar, this is old motherboard from around 2015.

[Thu Oct  3 16:10:14 2019] block tdd: sector-size: 512/512 capacity: 67108864
[Thu Oct  3 16:11:05 2019] swapper/0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[Thu Oct  3 16:11:05 2019] swapper/0 cpuset=/ mems_allowed=0
[Thu Oct  3 16:11:05 2019] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      4.19.0+1 #1
[Thu Oct  3 16:11:05 2019] Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.2 05/14/2018
[Thu Oct  3 16:11:05 2019] Call Trace:
[Thu Oct  3 16:11:05 2019]  <IRQ>
[Thu Oct  3 16:11:05 2019]  dump_stack+0x5a/0x73
[Thu Oct  3 16:11:05 2019]  warn_alloc+0xee/0x180
[Thu Oct  3 16:11:05 2019]  __alloc_pages_slowpath+0x84d/0xa09
[Thu Oct  3 16:11:05 2019]  ? get_page_from_freelist+0x14c/0xf00
[Thu Oct  3 16:11:05 2019]  __alloc_pages_nodemask+0x271/0x2b0
[Thu Oct  3 16:11:05 2019]  page_frag_alloc+0x103/0x120
[Thu Oct  3 16:11:05 2019]  __napi_alloc_skb+0x82/0xd0
[Thu Oct  3 16:11:05 2019]  rtl8169_poll+0x249/0x640 [r8169]
[Thu Oct  3 16:11:05 2019]  net_rx_action+0x2a5/0x3e0
[Thu Oct  3 16:11:05 2019]  __do_softirq+0xd1/0x28c
[Thu Oct  3 16:11:05 2019]  irq_exit+0xa8/0xc0
[Thu Oct  3 16:11:05 2019]  xen_evtchn_do_upcall+0x2c/0x50
[Thu Oct  3 16:11:05 2019]  xen_do_hypervisor_callback+0x29/0x40
[Thu Oct  3 16:11:05 2019]  </IRQ>
[Thu Oct  3 16:11:05 2019] RIP: e030:xen_hypercall_sched_op+0xa/0x20
[Thu Oct  3 16:11:05 2019] Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[Thu Oct  3 16:11:05 2019] RSP: e02b:ffffffff82003e58 EFLAGS: 00000246
[Thu Oct  3 16:11:05 2019] RAX: 0000000000000000 RBX: ffffffff82011740 RCX: ffffffff810013aa
[Thu Oct  3 16:11:05 2019] RDX: ffffffff8203c190 RSI: 0000000000000000 RDI: 0000000000000001
[Thu Oct  3 16:11:05 2019] RBP: 0000000000000000 R08: 000000000001ca00 R09: 0000000000000000
[Thu Oct  3 16:11:05 2019] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000
[Thu Oct  3 16:11:05 2019] R13: 0000000000000000 R14: ffffffff82011740 R15: ffffffff82011740
[Thu Oct  3 16:11:05 2019]  ? xen_hypercall_sched_op+0xa/0x20
[Thu Oct  3 16:11:05 2019]  ? xen_safe_halt+0xc/0x20
[Thu Oct  3 16:11:05 2019]  ? default_idle+0x1a/0x140
[Thu Oct  3 16:11:05 2019]  ? do_idle+0x1ea/0x260
[Thu Oct  3 16:11:05 2019]  ? cpu_startup_entry+0x6f/0x80
[Thu Oct  3 16:11:05 2019]  ? start_kernel+0x558/0x578
[Thu Oct  3 16:11:05 2019]  ? set_init_arg+0x55/0x55
[Thu Oct  3 16:11:05 2019]  ? xen_start_kernel+0x583/0x58d
[Thu Oct  3 16:11:05 2019] Mem-Info:
[Thu Oct  3 16:11:05 2019] active_anon:25841 inactive_anon:33848 isolated_anon:0
 active_file:60244 inactive_file:478866 isolated_file:0
 unevictable:4117 dirty:10002 writeback:28515 unstable:0
 slab_reclaimable:7812 slab_unreclaimable:8819
 mapped:29136 shmem:3449 pagetables:4220 bounce:0
 free:4766 free_pcp:868 free_cma:0
[Thu Oct  3 16:11:05 2019] Node 0 active_anon:103364kB inactive_anon:135392kB active_file:240976kB inactive_file:1915464kB unevictable:16468kB isolated(anon):0kB isolated(file):0kB mapped:116544kB dirty:40008kB writeback:114060kB shmem:13796kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Thu Oct  3 16:11:05 2019] DMA free:10032kB min:36kB low:48kB high:60kB active_anon:4kB inactive_anon:0kB active_file:136kB inactive_file:3392kB unevictable:0kB writepending:128kB present:15868kB managed:15784kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Thu Oct  3 16:11:05 2019] lowmem_reserve[]: 0 2505 2505 2505 2505
[Thu Oct  3 16:11:05 2019] DMA32 free:9032kB min:6372kB low:8936kB high:11500kB active_anon:103360kB inactive_anon:135392kB active_file:240840kB inactive_file:1911944kB unevictable:16468kB writepending:153940kB present:2720256kB managed:2565380kB mlocked:16468kB kernel_stack:8656kB pagetables:16880kB bounce:0kB free_pcp:3472kB local_pcp:244kB free_cma:0kB
[Thu Oct  3 16:11:05 2019] lowmem_reserve[]: 0 0 0 0 0
[Thu Oct  3 16:11:05 2019] Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:131072kB managed:0kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Thu Oct  3 16:11:05 2019] lowmem_reserve[]: 0 0 0 0 0
[Thu Oct  3 16:11:05 2019] DMA: 2*4kB (M) 1*8kB (M) 36*16kB (ME) 39*32kB (UME) 20*64kB (ME) 12*128kB (UME) 7*256kB (ME) 1*512kB (E) 3*1024kB (UM) 0*2048kB 0*4096kB = 10032kB
[Thu Oct  3 16:11:05 2019] DMA32: 68*4kB (MEH) 64*8kB (MEH) 77*16kB (EH) 61*32kB (H) 31*64kB (H) 4*128kB (H) 4*256kB (H) 2*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 9536kB
[Thu Oct  3 16:11:05 2019] Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[Thu Oct  3 16:11:05 2019] 544236 total pagecache pages
[Thu Oct  3 16:11:05 2019] 0 pages in swap cache
[Thu Oct  3 16:11:05 2019] Swap cache stats: add 0, delete 0, find 0/0
[Thu Oct  3 16:11:05 2019] Free swap  = 1048572kB
[Thu Oct  3 16:11:05 2019] Total swap = 1048572kB
[Thu Oct  3 16:11:05 2019] 716799 pages RAM
[Thu Oct  3 16:11:05 2019] 0 pages HighMem/MovableOnly
[Thu Oct  3 16:11:05 2019] 71508 pages reserved
[Thu Oct  3 16:11:05 2019] 0 pages cma reserved
[Thu Oct  3 16:11:05 2019] 0 pages hwpoisoned
[Thu Oct  3 16:14:23 2019] block tdd: sector-size: 512/512 capacity: 67108864