XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XCP-NG vm's extremly slow

    Scheduled Pinned Locked Moved Compute
    45 Posts 6 Posters 13.3k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H Offline
      hoerup @Andi79
      last edited by hoerup

      @Andi79
      Is the problem occurring both when the vm's are on HDD and on SSD storage or only on HDD ?

      A 1 Reply Last reply Reply Quote 0
      • A Offline
        Andi79 @hoerup
        last edited by

        @hoerup on both. I allready thought this could happen on some idle thing on the hdd's, but same problem on the ssd's, so that could not be the cause

        ForzaF 1 Reply Last reply Reply Quote 0
        • ForzaF Offline
          Forza @Andi79
          last edited by

          Can you try ext/thin instead of LVM? Not that this should matter that much.

          ForzaF A 2 Replies Last reply Reply Quote 0
          • ForzaF Offline
            Forza @Forza
            last edited by

            Looking at the video it seems the writes get queued up (Dirty in /proc/meminfo). I wonder why writes are so slow.

            What filesystems do you use on the guest?

            A 1 Reply Last reply Reply Quote 0
            • A Offline
              Andi79 @Forza
              last edited by

              @Forza

              this was the standard from xcp-ng during installation. The problem is that there allready are VMs on this machine and i can't reformat it without any problems (no other hosts in this data center where i could use the same IPs).

              guest system uses ext3

              A 1 Reply Last reply Reply Quote 0
              • A Offline
                Andi79 @Andi79
                last edited by olivierlambert

                Some more Infos:

                #cat /proc/mdstat  
Personalities : [raid1] 
md126 : active raid1 sdb[1] sda[0]
      3906886464 blocks super 1.2 [2/2] [UU]
      bitmap: 3/30 pages [12KB], 65536KB chunk

md127 : active raid1 sdd[1] sdc[0]
      937692352 blocks super 1.0 [2/2] [UU]
      bitmap: 1/7 pages [4KB], 65536KB chunk

                
#mdadm --detail /dev/md126
/dev/md126:
           Version : 1.2
     Creation Time : Sat Jun  4 12:08:56 2022
        Raid Level : raid1
        Array Size : 3906886464 (3725.90 GiB 4000.65 GB)
     Used Dev Size : 3906886464 (3725.90 GiB 4000.65 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Jun 14 19:55:36 2022
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : server2-neu:md126  (local to host server2)
              UUID : 784c25d6:18f3a0c2:ca8fe399:d16ec0e2
            Events : 35383

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb

                #mdadm --detail /dev/md127
/dev/md127:
           Version : 1.0
     Creation Time : Sat Jun  4 09:56:54 2022
        Raid Level : raid1
        Array Size : 937692352 (894.25 GiB 960.20 GB)
     Used Dev Size : 937692352 (894.25 GiB 960.20 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Jun 14 19:56:15 2022
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : localhost:127
              UUID : b5ab10b2:b89109af:9f4a274a:d7af50b3
            Events : 4450

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       1       8       48        1      active sync   /dev/sdd

                #vgdisplay
  Device read short 82432 bytes remaining
  Device read short 65536 bytes remaining
  --- Volume group ---
  VG Name               VG_XenStorage-cd8f9061-df06-757c-efb6-4ada0927a984
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  93
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                5
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <3,64 TiB
  PE Size               4,00 MiB
  Total PE              953826
  Alloc PE / Size       507879 / <1,94 TiB
  Free  PE / Size       445947 / 1,70 TiB
  VG UUID               yqsyV9-h2Gl-5lMf-486M-BI7f-r3Ar-Todeh9
   
  --- Volume group ---
  VG Name               VG_XenStorage-c29b2189-edf2-8349-d964-381431c48be1
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  25
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <852,74 GiB
  PE Size               4,00 MiB
  Total PE              218301
  Alloc PE / Size       1 / 4,00 MiB
  Free  PE / Size       218300 / 852,73 GiB
  VG UUID               4MFkwD-1JW1-zVE3-QFKf-XmOX-QsSf-60oCKZ
                A 1 Reply Last reply Reply Quote 0
                • A Offline
                  Andi79 @Andi79
                  last edited by

                  io.png

                  ok.... it really seems to be an io problem. any ideas what could cause this?

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    @fohdeesha does it ring any bell?

                    fohdeeshaF 1 Reply Last reply Reply Quote 0
                    • A Offline
                      Andi79 @Forza
                      last edited by

                      @Forza

                      now i noticed that tje jdb2 and kworker processes have gone... but system is still extremly slow

                      https://www.pzka.de/xen2.mp4

                      on this video i try to install munin for future data to analyse the problem. As you can see absolutly nothing happens (this can take many minutes now). There is a rsync "running" at a very slow speed, but as you can see cpu usage is ultra low and also top says no system load.

                      I think it must be a problem with xcp-ng, but I have no idea what it could be.

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Can you do a dmesg and also a smartctl -a /dev/sdb and smartctl /dev/sda?

                        A 1 Reply Last reply Reply Quote 0
                        • A Offline
                          Andi79 @olivierlambert
                          last edited by Andi79

                          @olivierlambert

                          output of dmsg (on the vm):

                          [132514.270681] systemd[1]: Starting Journal Service...
[132604.475796] systemd[1]: systemd-journald.service: start operation timed out. Terminating.
[132694.707996] systemd[1]: systemd-journald.service: State 'stop-sigterm' timed out. Killing.
[132694.708039] systemd[1]: systemd-journald.service: Killing process 60936 (systemd-journal) with signal SIGKILL.
[132784.940258] systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring.
[132797.289939] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL
[132797.289947] systemd[1]: systemd-journald.service: Failed with result 'timeout'.
[132797.290322] systemd[1]: Failed to start Journal Service.
[132797.291608] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 18.
[132797.291833] systemd[1]: Stopped Journal Service.
[132797.324750] systemd[1]: Starting Journal Service...
[132808.130140] systemd-journald[61002]: File /var/log/journal/eb029b8cf0534f998db52d5afecd252b/system.journal corrupted or uncleanly shut down, renaming and replacing.
[132817.090264] systemd[1]: Started Journal Service.
[158702.774840] VFS: busy inodes on changed media sr0

                          i haven't installed smarttools on the VMs, but i can do.... from the video to now the apt for installing munin is on 4% now. it tooks very very long

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Not in the VM, in the Dom0 please 🙂

                            A 1 Reply Last reply Reply Quote 0
                            • A Offline
                              Andi79 @olivierlambert
                              last edited by

                              @olivierlambert sorry :). that seems to be interesting.

                              [775961.059191] vif vif-5-1 vif5.1: Guest Rx stalled
[775971.060899] vif vif-5-1 vif5.1: Guest Rx ready
[775978.737860] vif vif-5-1 vif5.1: Guest Rx stalled
[775981.172611] vif vif-1-1 vif1.1: Guest Rx stalled
[775981.172612] vif vif-13-1 vif13.1: Guest Rx stalled
[775988.745427] vif vif-5-1 vif5.1: Guest Rx ready
[775988.803015] vif vif-5-1 vif5.1: Guest Rx stalled
[775991.305032] vif vif-13-1 vif13.1: Guest Rx ready
[775991.308940] vif vif-1-1 vif1.1: Guest Rx ready
[775992.359938] vif vif-13-1 vif13.1: Guest Rx stalled
[775997.663683] vif vif-10-1 vif10.1: Guest Rx stalled
[775998.983413] vif vif-5-1 vif5.1: Guest Rx ready
[775998.983418] vif vif-5-1 vif5.1: Guest Rx stalled
[776002.566713] vif vif-13-1 vif13.1: Guest Rx ready
[776002.713980] vif vif-13-1 vif13.1: Guest Rx stalled
[776007.653843] vif vif-10-1 vif10.1: Guest Rx ready
[776010.001242] vif vif-5-1 vif5.1: Guest Rx ready
[776012.516412] vif vif-5-1 vif5.1: Guest Rx stalled
[776012.804789] vif vif-13-1 vif13.1: Guest Rx ready
[776020.986445] vif vif-1-1 vif1.1: Guest Rx stalled
[776020.986456] vif vif-13-1 vif13.1: Guest Rx stalled
[776022.534917] vif vif-5-1 vif5.1: Guest Rx ready

                              smartctl -a /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.19.0+1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     INTEL SSDSC2KG960G8
Serial Number:    BTYG201203PZ960CGN
LU WWN Device Id: 5 5cd2e4 154e9cb05
Firmware Version: XCV10132
User Capacity:    960.197.124.096 bytes [960 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jun 16 09:47:04 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(    0) seconds.
Offline data collection
capabilities: 			 (0x79) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 (   2) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       607
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       15
170 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       8
175 Program_Fail_Count_Chip 0x0033   100   100   010    Pre-fail  Always       -       66811922817
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   065   061   000    Old_age   Always       -       35 (Min/Max 26/40)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       8
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       35
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
225 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       29336
226 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       20
227 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       81
228 Power-off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       36300
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       0
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       66811922817
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       29336
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       132918
243 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       90104

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

                              # smartctl -a /dev/sdb
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.19.0+1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     INTEL SSDSC2KG960G8
Serial Number:    BTYG201306VP960CGN
LU WWN Device Id: 5 5cd2e4 154eade4f
Firmware Version: XCV10132
User Capacity:    960.197.124.096 bytes [960 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jun 16 09:47:55 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(    0) seconds.
Offline data collection
capabilities: 			 (0x79) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 (   2) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       607
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       15
170 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       8
175 Program_Fail_Count_Chip 0x0033   100   100   010    Pre-fail  Always       -       66811922837
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   065   062   000    Old_age   Always       -       35 (Min/Max 25/39)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       8
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       35
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
225 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       57980
226 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       30
227 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       64
228 Power-off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       36302
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       0
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       66811922837
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       57980
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       97397
243 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       144726

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


                               smartctl -a /dev/sdc
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.19.0+1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 2.5 5400
Device Model:     ST4000LM024-2AN17V
Serial Number:    WCK8GFTY
LU WWN Device Id: 5 000c50 0e04fd944
Firmware Version: 0001
User Capacity:    4.000.787.030.016 bytes [4,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5526 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jun 16 09:48:22 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(    0) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 659) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x30a5)	SCT Status supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   081   064   006    Pre-fail  Always       -       118330775
  3 Spin_Up_Time            0x0003   100   099   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       20
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   081   060   045    Pre-fail  Always       -       134329070
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       606 (32 213 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       19
183 SATA_Downshift_Count    0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   063   062   040    Old_age   Always       -       37 (Min/Max 29/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   097   097   000    Old_age   Always       -       6920
194 Temperature_Celsius     0x0022   037   040   000    Old_age   Always       -       37 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   081   064   000    Old_age   Always       -       118330775
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       562 (116 178 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       16861005126
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       11745641547
254 Free_Fall_Sensor        0x0032   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

                              that is the first host (i have only 1 raid on the ssd and 1 single hdd on this host)

                              it takes about 3-4sec for smartctl to display the data there

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                All disks are taking 3/4 sec to display smartctl ?

                                A 1 Reply Last reply Reply Quote 0
                                • A Offline
                                  Andi79 @olivierlambert
                                  last edited by

                                  @olivierlambert
                                  only the 4TB hdd. i also have testet it on the other dom0, there it's quicker. Maybe it's related because there are some running rsyncs on a virtual machine to this disk.

                                  1 Reply Last reply Reply Quote -1
                                  • fohdeeshaF Offline
                                    fohdeesha Vates 🪐 Pro Support Team @olivierlambert
                                    last edited by

                                    @olivierlambert definitely doesn't ring any bells but definitely smells like a storage issue, I would test disk performance directly in dom0 on the RAID SR using something like FIO, although we don't provide this in our repos so that would be difficult to get in dom0. perhaps a simple dd speed test under the SR mount directly, you should be able to find it by running df -h. Just be sure you don't overwrite any VM disk files 🙂

                                    1 Reply Last reply Reply Quote 0
                                    • A Offline
                                      Andi79
                                      last edited by

                                      @fohdeesha

                                      i would agree that it sounds like an storage problem, but it's unlikley that this happends on 2 brand new machines at the same time and the dom0 should be affected with the same problems when it's a hardware case.

                                      What does this "vif vif-5-1 vif5.1: Guest Rx stalled" mean? Rx stand for some information recieve?

                                      fohdeeshaF 2 Replies Last reply Reply Quote 0
                                      • fohdeeshaF Offline
                                        fohdeesha Vates 🪐 Pro Support Team @Andi79
                                        last edited by

                                        @Andi79 it's impossible to say without more testing, it could be a firmware issue on these boards causing IO stalls or similar, the only way to be sure now is to run some perf tests inside dom0 to confirm it's not VM related, once you get that confirmation, install (or live boot) plain CentOS on one of the machine and test there as well. Are both the SSD and the HDD underneath an mdadm software raid? I suppose it could be a software raid configuration issue as well, but it's hard to say without further testing

                                        A 1 Reply Last reply Reply Quote 0
                                        • fohdeeshaF Offline
                                          fohdeesha Vates 🪐 Pro Support Team @Andi79
                                          last edited by

                                          @Andi79 the guest RX messages are normal, some OSs like debian etc put unused interfaces in some type of "sleep" mode if I remember correctly and this causes these harmless messages

                                          F A 3 Replies Last reply Reply Quote 0
                                          • F Offline
                                            fred974 @fohdeesha
                                            last edited by

                                            I am no expert here but I had vm crashing on me before when transfering data and it was because the VM started its life with a very large amount of RAM. Then it was reduce and the setting in Memory limits (min/max) was messed up.
                                            What do you have for Memory limits (min/max) and what is the current VM memory?

                                            A 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post