XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. sapcode
    S
    Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 2
    • Posts 15
    • Groups 0

    sapcode

    @sapcode

    1
    Reputation
    10
    Profile views
    15
    Posts
    0
    Followers
    0
    Following
    Joined
    Last Online

    sapcode Unfollow Follow

    Latest posts made by sapcode

    • RE: PCI Passthrough of QAT adapter IQA89601G1P5

      @olivierlambert thank you for the quick answers, just to put it into context the plan was to use accelerated Wireguard VPN in pfSense+ which comes with own FreeBSD optimized kernel modules:

      Some more info on CPU extensions / QAT / IPSsec-MB / AES-NI related to that:

      1. https://github.com/intel/intel-ipsec-mb
      2. https://docs.netgate.com/pfsense/en/latest/hardware/cryptographic-accelerators.html
      3. https://networkbuilders.intel.com/docs/networkbuilders/intel-avx-512-and-intel-qat-accelerate-wireguard-processing-with-intel-xeon-d-2700-processor-technology-guide-1647024663.pdf
      4. https://2023.asiabsdcon.org/program/_p05b/paper.pdf
      5. https://forum.opnsense.org/index.php?topic=37808.0
      6. https://forum.netgate.com/topic/173606/solved-intel-qat-driver-with-wireguard-support/9
      7. https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/dev/qat/include/common/adf_accel_devices.h#L38

      Example CPU's having the relevant features incl. AVX-512:

      1. https://ark.intel.com/content/www/de/de/ark/products/226102/intel-xeon-d-1713nt-processor-10m-cache-up-to-3-50-ghz.html
      2. https://ark.intel.com/content/www/de/de/ark/products/226113/intel-xeon-d-1722ne-processor-10m-cache-up-to-2-70-ghz.html

      Some older Atoms have Intel SSE4.2 but miss the AVX-512 which i think bring the most performance:

      1. https://ark.intel.com/content/www/de/de/ark/products/97937/intel-atom-processor-c3558-8m-cache-up-to-2-20-ghz.html
      2. https://ark.intel.com/content/www/de/de/ark/products/134803/intel-atom-processor-c5115-9m-cache-2-8ghz.html

      Best regards

      posted in Compute
      S
      sapcode
    • RE: PCI Passthrough of QAT adapter IQA89601G1P5

      @impovich I have the same issue with Intel Atom Processor c3758, in XCP 8.2.1 the co-processor is visible:

      lspci -nn
      01:00.0 Co-processor [0b40]: Intel Corporation Atom Processor C3000 Series QuickAssist Technology [8086:19e2] (rev 11)
       
      lspci -vd 8086:19e2
      01:00.0 Co-processor: Intel Corporation Atom Processor C3000 Series QuickAssist Technology (rev 11)
              Subsystem: Intel Corporation Device 0000
              Flags: fast devsel, IRQ 255
              Memory at df440000 (64-bit, non-prefetchable) [disabled] [size=256K]
              Memory at df400000 (64-bit, non-prefetchable) [disabled] [size=256K]
              Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
              Capabilities: [60] MSI-X: Enable- Count=17 Masked-
              Capabilities: [6c] Power Management version 3
              Capabilities: [74] Express Endpoint, MSI 00
              Capabilities: [100] Advanced Error Reporting
              Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
              Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
              Capabilities: [1b0] Access Control Services
      

      But i am missing those entrie:

       Kernel driver in use: c3xxx  <- Not present
       Kernel modules: qat_c3xxx  <- Not present
      

      Also "xl pci-assignable-list" show's ntothing.

      In the OpNsense VM i get this, without any QAT co-processor:

      lspci -nn
      00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
      00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
      00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010]
      00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01)
      00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 01)
      00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]
      00:03.0 SCSI storage controller [0100]: XenSource, Inc. Xen Platform Device [5853:0001] (rev 01)
      

      Is there anything which can be done to pass it through ?

      @olivierlambert I remeber there where some threads about GPU pass through with some crazy hacks 😉 maybe we could try appling some stuff here ?

      Best regards

      posted in Compute
      S
      sapcode
    • RE: openssl 1.1.1 update or manual build in XCP 8.2.1 possible ?

      @olivierlambert this error line from your call shows that the updates are not fixing the issue

      140304244799376:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769:
      

      A correct output which you can reproduce with any version openssl higher or equal to 1.1.1 looks like this:

      openssl s_client -connect ssl-tools.net:443
      CONNECTED(00000003)
      depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
      verify return:1
      depth=1 C = US, O = Let's Encrypt, CN = E6
      verify return:1
      depth=0 CN = ssl-tools.net
      verify return:1
      ---
      Certificate chain
       0 s:CN = ssl-tools.net
         i:C = US, O = Let's Encrypt, CN = E6
       1 s:C = US, O = Let's Encrypt, CN = E6
         i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
      ---
      Server certificate
      -----BEGIN CERTIFICATE-----
      MIIDfDCCAwKgAwIBAgISAwe686407pktUx6L9xPVL1ZsMAoGCCqGSM49BAMDMDIx
      CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF
      NjAeFw0yNDA5MTYxNjMwMTZaFw0yNDEyMTUxNjMwMTVaMBgxFjAUBgNVBAMTDXNz
      bC10b29scy5uZXQwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASX0K6Lk50W7c5e
      oInqD7aIThmZDpyP+fK8QSP6/bg77XrN/hxovn+ruTYWpfMiBAp4QI0JZbwYvUWs
      0YNrpColo4ICEDCCAgwwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUF
      BwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQIz+2YvUOQXQMM
      lvk9SGwHxLrMZzAfBgNVHSMEGDAWgBSTJ0aYA6lRaI6Y1sRCSNsjv1iU0jBVBggr
      BgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9lNi5vLmxlbmNyLm9yZzAi
      BggrBgEFBQcwAoYWaHR0cDovL2U2LmkubGVuY3Iub3JnLzAYBgNVHREEETAPgg1z
      c2wtdG9vbHMubmV0MBMGA1UdIAQMMAowCAYGZ4EMAQIBMIIBBQYKKwYBBAHWeQIE
      AgSB9gSB8wDxAHYAPxdLT9ciR1iUHWUchL4NEu2QN38fhWrrwb8ohez4ZG4AAAGR
      ++E2+wAABAMARzBFAiAndRz0BOt7p6ZPW0LM70zHu0hIsO0FzjSrL566uNLIuQIh
      AL0VFL/qfax8P3qWTaXqlrETL6Ro4bDF1BMtZbLhF83nAHcA3+FW66oFr7WcD4Zx
      jajAMk6uVtlup/WlagHRwTu+UlwAAAGR++E3xQAABAMASDBGAiEA9kbdYv9Jf+Lt
      hE8I2IsYaW0cyTO1KP6Of8IFBn2y/W0CIQCaNM1+tvQ5eYKE3GN0Ln0IYZniWaH5
      v7OhnxzdTgm+0jAKBggqhkjOPQQDAwNoADBlAjEAtffvHcb5z4g2MxxdD6aAK7M5
      Nfdy0SeeGAgyzsxolMyyFwkMtmLi0svYVvBYznR1AjB1STPF0cvcy8S5CfEbt/sR
      0L59P09hLMTvqV2xqumaXo5Upd6JbaF0oLaD8wXpGl0=
      -----END CERTIFICATE-----
      subject=CN = ssl-tools.net
      issuer=C = US, O = Let's Encrypt, CN = E6
      ---
      No client certificate CA names sent
      Peer signing digest: SHA256
      Peer signature type: ECDSA
      Server Temp Key: X25519, 253 bits
      ---
      SSL handshake has read 2371 bytes and written 379 bytes
      Verification: OK
      ---
      New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
      Server public key is 256 bit
      Secure Renegotiation IS NOT supported
      Compression: NONE
      Expansion: NONE
      No ALPN negotiated
      Early data was not sent
      Verify return code: 0 (ok)
      ---
      ---
      Post-Handshake New Session Ticket arrived:
      SSL-Session:
          Protocol  : TLSv1.3
          Cipher    : TLS_AES_128_GCM_SHA256
          Session-ID: BC6467F92591807810680DA798C335BDE23409C87BB0A5BC3F2A07F8AD2557B8
          Session-ID-ctx:
          Resumption PSK: 285BC2D246C8D2026C71B0ADBAC8F93AC7287A44580AECE5B06881AAC701C037
          PSK identity: None
          PSK identity hint: None
          SRP username: None
          TLS session ticket lifetime hint: 604800 (seconds)
          TLS session ticket:
          0000 - 4b ef 37 0e 69 7d 6e f9-2f 15 51 e1 2b 1a 22 78   K.7.i}n./.Q.+."x
          0010 - e1 ba 22 dd 99 ed 6a eb-c5 b1 8e 06 3c 63 65 fb   .."...j.....<ce.
          0020 - a3 a8 14 63 0f bf 43 f2-7a fb c4 cd 3f 3d 7b a0   ...c..C.z...?={.
          0030 - 73 cc b0 d1 1d a6 35 30-b9 51 48 1a e8 d5 45 67   s.....50.QH...Eg
          0040 - 9a c1 cf bf f3 1b 77 62-e0 91 0e a0 68 3d 5f 7d   ......wb....h=_}
          0050 - ae 3f d5 1e 58 3c 24 b2-bd 44 4d 22 41 c7 37 b5   .?..X<$..DM"A.7.
          0060 - 6b 64 9a dd 72 c2 85 d1-f4                        kd..r....
      
          Start Time: 1730967208
          Timeout   : 7200 (sec)
          Verify return code: 0 (ok)
          Extended master secret: no
          Max Early Data: 0
      ---
      read R BLOCK
      

      I guess that the error has to do with https://wiki.openssl.org/index.php/TLS1.3 support was introduced in openssl 1.1.1 and servers which don't allow to downgrade to TLS 1.2 / 1.1 will simply fail in opensll 1.0.2 with the above error line.

      As per TLS 1.3: The new ciphersuites are defined differently and do not specify the certificate type (e.g. RSA, DSA, ECDSA) or the key exchange mechanism (e.g. DHE or ECDHE). This has implications for ciphersuite configuration, that looks like the root problem if servers enforce this then openssl 1.0.2 will fail to establish a connection at all.

      The next option would be to manually build openssl 1.1.1 in XCP 8.2.1 itself, @olivierlambert could you please test if this steps will work and not break an existing XCP installation ?

      # uninstall - this may uninstall xcp-ng-deps as well, don't know if this has breaking impact to XCP
      yum remove opensll
      
      # Install required packages
      yum install -y make gcc perl-core pcre-devel wget zlib-devel
      
      # Download the latest version of OpenSSL source code
      wget https://ftp.openssl.org/source/openssl-1.1.1k.tar.gz
      
      # Configure, build and install OpenSSL Uncompress the source file
      tar -xzvf openssl-1.1.1k.tar.gz
      
      # Change to the OpenSSL directory
      cd openssl-1.1.1k
      
      # Configure the package for compilation
      ./config --prefix=/usr --openssldir=/etc/ssl --libdir=lib no-shared zlib-dynamic
      
      # Compile package
      make
      
      # Test compiled package
      make test
      
      # Install compiled package
      make install
      
      # Export library path Create environment variable file
      vim /etc/profile.d/openssl.sh
      
      # Add the following content
      export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64
      
      # Load the environment variable
      source /etc/profile.d/openssl.sh
      
      # Verify the OpenSSL version
      openssl version
      

      This two blogs shows how to build openssl 1.1.1 in CentOS 7 manually, i havn't tried them yet in an existing XCP installation:

      Blog 1: https://gist.github.com/Bill-tran/5e2ab062a9028bf693c934146249e68c
      Blog 2: https://computingforgeeks.com/how-to-install-openssl-1-1-on-centos-rhel-7/?utm_content=cmp-true

      Hope it will work
      Best regards
      User

      posted in Development
      S
      sapcode
    • RE: openssl 1.1.1 update or manual build in XCP 8.2.1 possible ?

      @olivierlambert so you get the same error even after applying the pending updates ?

      posted in Development
      S
      sapcode
    • RE: openssl 1.1.1 update or manual build in XCP 8.2.1 possible ?

      @Danp can you please do a quick test on you machine if this will work, after applying the pending updates:

      openssl s_client -connect ssl-tools.net:443
      
      posted in Development
      S
      sapcode
    • RE: openssl 1.1.1 update or manual build in XCP 8.2.1 possible ?

      Hi Oliver,

      the issue is when using curl and wget which both uses openssl (OpenSSL 1.0.2k-fips 26 Jan 2017), in bash scripts some servers are not allowing communication with TLS lower/equal than tls1_2 and the following error is thrown:

      openssl s_client -connect ssl-tools.net:443
      CONNECTED(00000003)
      139843806320528:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769
      Secure Renegotiation IS NOT supported
      

      We tested the same call from another linux machine using "OpenSSL 1.1.1t-freebsd 7 Feb 2023" and it worked fine:

      openssl s_client -connect ssl-tools.net:443
      
      Peer signing digest: SHA256
      Peer signature type: ECDSA
      Server Temp Key: X25519, 253 bits
      ---
      SSL handshake has read 2371 bytes and written 379 bytes
      Verification: OK
      ---
      New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
      Server public key is 256 bit
      Secure Renegotiation IS NOT supported
      

      Best regards
      User

      posted in Development
      S
      sapcode
    • openssl 1.1.1 update or manual build in XCP 8.2.1 possible ?

      Dear XCP team,

      is it possible to update or manual build openssl 1.1.1 in 8.2.1 ?

      1. Would it be safe to run yum remove openssl as the first step or would this break the XCP installation:
      Dependencies Resolved
      ===========================================================================================================================================================================================
       Package                                         Arch                              Version                                           Repository                                       Size
      ===========================================================================================================================================================================================
      Removing:
       openssl                                         x86_64                            1:1.0.2k-26.2.xcpng8.2                            @xcp-ng-testing                                 814 k
      Removing for dependencies:
       awscli                                          noarch                            1.23.2-1.el7_9.1                                  @updates                                        9.0 M
       openssl-perl                                    x86_64                            1:1.0.2k-26.2.xcpng8.2                            @xcp-ng-testing                                  28 k
       python2-cryptography                            x86_64                            1.7.2-2.el7                                       @base                                           2.6 M
       xcp-ng-deps                                     noarch                            8.2.0-12                                          @install/$releasever                            0.0
      
      Transaction Summary
      ===========================================================================================================================================================================================
      Remove  1 Package (+4 Dependent packages)
      Installed size: 12 M
      

      We found the following manual steps how to build openssl 1.1.1 in CentOS 7 here: https://gist.github.com/Bill-tran/5e2ab062a9028bf693c934146249e68c but can't test it on our server:

      # Requirements Upgrade the system
      yum -y update
      
      # Install required packages
      yum install -y make gcc perl-core pcre-devel wget zlib-devel
      
      # Download the latest version of OpenSSL source code
      wget https://ftp.openssl.org/source/openssl-1.1.1k.tar.gz
      
      # Configure, build and install OpenSSL Uncompress the source file
      tar -xzvf openssl-1.1.1k.tar.gz
      
      # Change to the OpenSSL directory
      cd openssl-1.1.1k
      
      # Configure the package for compilation
      ./config --prefix=/usr --openssldir=/etc/ssl --libdir=lib no-shared zlib-dynamic
      
      # Compile package
      make
      
      # Test compiled package
      make test
      
      # Install compiled package
      make install
      
      # Export library path Create environment variable file
      vim /etc/profile.d/openssl.sh
      
      # Add the following content
      export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64
      
      # Load the environment variable
      source /etc/profile.d/openssl.sh
      
      # Verify the OpenSSL version
      openssl version
      
      1. Can anyone here perform the test or are there any other possibilities to get openssl 1.1.1 in XCP 8.2.1 ?

      Best regards
      User

      posted in Development
      S
      sapcode
    • RE: XCP 8.2 VCPUs-max settings

      @jeff Hi jeff, we have the same issue with another Xeon CPU's.

      This thread should be linked with:
      https://xcp-ng.org/forum/topic/4604/xcp-8-2-vcpus-max-80-but-vm-shows-64-cpu-only-numa-nodes-and-threads-per-core-not-matching/12

      Can you fire the following command during all your VM's are started: xl vcpu-list

      In our case we see all vCPUS's greater than 64 in the state "--p" which is bad:

      Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
      Domain-0                             0     0   79   r--     257.2  all / all
      0__Confluence & Jira & Gitlab        1     0   40   -b-     142.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    64    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    65    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    66    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    67    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    68    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    69    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    70    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    71    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    72    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    73    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    74    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    75    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    76    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    77    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    78    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    79    -   --p       0.0  all / all
      

      It looks like that XEN passes only 64 vCPU's and the rest of CPU's has state "--p" in the pause state, god only knows why and where this limit has been implemented....

      Source: https://linux.die.net/man/1/xm

      STATES
      The State field lists 6 states for a Xen Domain, and which ones the current Domain is in.

      r - running The domain is currently running on a CPU

      b - blocked The domain is blocked, and not running or runnable. This can be caused because the domain is waiting on IO (a traditional wait state) or has gone to sleep because there was nothing else for it to do.

      p - paused The domain has been paused, usually occurring through the administrator running xm pause. When in a paused state the domain will still consume allocated resources like memory, but will not be eligible for scheduling by the Xen hypervisor.

      s - shutdown The guest has requested to be shutdown, rebooted or suspended, and the domain is in the process of being destroyed in response.

      c - crashed The domain has crashed, which is always a violent ending. Usually this state can only occur if the domain has been configured not to restart on crash. See xmdomain.cfg for more info.

      d - dying The domain is in process of dying, but hasn't completely shutdown or crashed.

      posted in Compute
      S
      sapcode
    • RE: XCP 8.2 VCPUs-max=80 but VM shows 64 CPU only, NUMA nodes and Threads per core not matching

      @sapcode
      Found a description here: https://linux.die.net/man/1/xm
      But no idea why it starts some CPUs in the pause state....
      Will do some testing with C states and govenor performance....

      STATES

      The State field lists 6 states for a Xen Domain, and which ones the current Domain is in.
      r - running      The domain is currently running on a CPU
      b - blocked     The domain is blocked, and not running or runnable. This can be caused because the domain is waiting on IO (a traditional wait state) or has gone to sleep because there was nothing else for it to do.
      p - paused       The domain has been paused, usually occurring through the administrator running xm pause. When in a paused state the domain will still consume allocated resources like memory, but will not be eligible for scheduling by the Xen hypervisor.
      s - shutdown   The guest has requested to be shutdown, rebooted or suspended, and the domain is in the process of being destroyed in response.
      c - crashed      The domain has crashed, which is always a violent ending. Usually this state can only occur if the domain has been configured not to restart on crash. See xmdomain.cfg for more info.
      d - dying         The domain is in process of dying, but hasn't completely shutdown or crashed.

      posted in Compute
      S
      sapcode
    • RE: XCP 8.2 VCPUs-max=80 but VM shows 64 CPU only, NUMA nodes and Threads per core not matching

      @olivierlambert The strange thing is follwing, all vCPU's and NUMA nodes are shown correctly in xl info:

      xl info -n
      host                   : xcpng82
      release                : 4.19.0+1
      version                : #1 SMP Tue Mar 30 22:34:15 CEST 2021
      machine                : x86_64
      nr_cpus                : 80
      max_cpu_id             : 79
      nr_nodes               : 2
      cores_per_socket       : 20
      threads_per_core       : 2
      cpu_mhz                : 2300.006
      hw_caps                : bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
      virt_caps              : pv hvm hvm_directio pv_directio hap shadow iommu_hap_pt_share
      total_memory           : 786321
      free_memory            : 14403
      sharing_freed_memory   : 0
      sharing_used_memory    : 0
      outstanding_claims     : 0
      free_cpus              : 0
      cpu_topology           :
      cpu:    core    socket     node
        0:       0        0        0
        1:       0        0        0
        2:       1        0        0
        3:       1        0        0
        4:       2        0        0
        5:       2        0        0
        6:       3        0        0
        7:       3        0        0
        8:       4        0        0
        9:       4        0        0
       10:       8        0        0
       11:       8        0        0
       12:       9        0        0
       13:       9        0        0
       14:      10        0        0
       15:      10        0        0
       16:      11        0        0
       17:      11        0        0
       18:      12        0        0
       19:      12        0        0
       20:      16        0        0
       21:      16        0        0
       22:      17        0        0
       23:      17        0        0
       24:      18        0        0
       25:      18        0        0
       26:      19        0        0
       27:      19        0        0
       28:      20        0        0
       29:      20        0        0
       30:      24        0        0
       31:      24        0        0
       32:      25        0        0
       33:      25        0        0
       34:      26        0        0
       35:      26        0        0
       36:      27        0        0
       37:      27        0        0
       38:      28        0        0
       39:      28        0        0
       40:       0        1        1
       41:       0        1        1
       42:       1        1        1
       43:       1        1        1
       44:       2        1        1
       45:       2        1        1
       46:       3        1        1
       47:       3        1        1
       48:       4        1        1
       49:       4        1        1
       50:       8        1        1
       51:       8        1        1
       52:       9        1        1
       53:       9        1        1
       54:      10        1        1
       55:      10        1        1
       56:      11        1        1
       57:      11        1        1
       58:      12        1        1
       59:      12        1        1
       60:      16        1        1
       61:      16        1        1
       62:      17        1        1
       63:      17        1        1
       64:      18        1        1
       65:      18        1        1
       66:      19        1        1
       67:      19        1        1
       68:      20        1        1
       69:      20        1        1
       70:      24        1        1
       71:      24        1        1
       72:      25        1        1
       73:      25        1        1
       74:      26        1        1
       75:      26        1        1
       76:      27        1        1
       77:      27        1        1
       78:      28        1        1
       79:      28        1        1
      device topology        :
      device           node
      0000:03:00.0      0
      0000:83:04.0      1
      0000:00:1f.2      0
      0000:00:1c.0      0
      0000:09:00.0      0
      0000:84:00.0      1
      0000:00:04.4      0
      0000:80:05.2      1
      0000:80:02.0      1
      0000:00:1f.0      0
      0000:83:0c.0      1
      0000:02:00.0      0
      0000:00:04.2      0
      0000:00:01.0      0
      0000:80:05.0      1
      0000:80:04.6      1
      0000:08:00.0      0
      0000:83:00.0      1
      0000:00:04.0      0
      0000:80:04.4      1
      0000:00:16.0      0
      0000:01:00.0      0
      0000:82:00.0      1
      0000:80:04.2      1
      0000:80:01.0      1
      0000:00:05.1      0
      0000:04:08.0      0
      0000:00:04.7      0
      0000:00:1f.3      0
      0000:00:00.0      0
      0000:80:04.0      1
      0000:04:10.0      0
      0000:00:04.5      0
      0000:06:00.0      0
      0000:81:00.0      1
      0000:00:03.0      0
      0000:87:00.0      1
      0000:02:00.1      0
      0000:00:04.3      0
      0000:80:05.1      1
      0000:00:1a.0      0
      0000:00:11.4      0
      0000:80:04.7      1
      0000:05:00.0      0
      0000:00:04.1      0
      0000:00:1d.0      0
      0000:80:04.5      1
      0000:86:00.0      1
      0000:00:05.4      0
      0000:00:16.1      0
      0000:01:00.1      0
      0000:83:08.0      1
      0000:00:11.0      0
      0000:80:04.3      1
      0000:00:05.2      0
      0000:00:02.0      0
      0000:85:00.0      1
      0000:00:14.0      0
      0000:80:04.1      1
      0000:00:1c.2      0
      0000:00:05.0      0
      0000:00:04.6      0
      0000:80:05.4      1
      numa_info              :
      node:    memsize    memfree    distances
         0:    395264       9023      10,21
         1:    393216       5379      21,10
      xen_major              : 4
      xen_minor              : 13
      xen_extra              : .1-9.9.1
      xen_version            : 4.13.1-9.9.1
      xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
      xen_scheduler          : credit
      xen_pagesize           : 4096
      platform_params        : virt_start=0xffff800000000000
      xen_changeset          : 6278553325a9, pq 70d4b5941e4f
      xen_commandline        : dom0_mem=8192M,max:8192M watchdog ucode=scan crashkernel=256M,below=4G console=vga vga=mode-0x0311 cpuid=no-ibrsb,no-ibpb,no-stibp bti=thunk=jmp,rsb_native=no,rsb_vmexit=no xpti=no cpufreq=xen:performance max_cstate=0 nospec_store_bypass_disable noibrs noibpb nopti l1tf=off mitigations=off spectre_v2_user=off spectre_v2=off nospectre_v2 nospectre_v1 no_stf_barrier mds=off spec-ctrl=no dom0_max_vcpus=1-1
      cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
      cc_compile_by          : mockbuild
      cc_compile_domain      : [unknown]
      cc_compile_date        : Thu Feb  4 18:23:36 CET 2021
      build_id               : a76c6ee84d87600fa0d520cd8ecb8113b1105af4
      xend_config_format     : 4
      

      But in xl vcpu-list for the specific VM "0__SAP S4H S4HANA 2.0 1709 SP4" for the vCPU 64-79 the state looks strange:

      # xl vcpu-list
      [10:35 xcpng82 ~]# xl vcpu-list
      Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
      Domain-0                             0     0   79   r--     257.2  all / all
      0__Confluence & Jira & Gitlab        1     0   40   -b-     142.2  all / all
      0__Confluence & Jira & Gitlab        1     1   19   -b-     142.8  all / all
      0__Confluence & Jira & Gitlab        1     2   65   -b-     141.5  all / all
      0__Confluence & Jira & Gitlab        1     3   70   -b-     136.8  all / all
      0__Confluence & Jira & Gitlab        1     4   34   -b-     138.2  all / all
      0__Confluence & Jira & Gitlab        1     5    3   -b-     141.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     0   76   -b-      47.3  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     1   64   -b-      25.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     2   23   -b-      25.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     3   28   -b-      22.8  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     4   32   -b-      34.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     5   53   -b-      53.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     6   11   -b-      36.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     7    7   -b-      35.4  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     8   12   -b-      39.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2     9   55   -b-      26.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    10   49   -b-      26.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    11   20   -b-      28.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    12   60   -b-      41.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    13   43   -b-      22.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    14   69   -b-      29.8  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    15   36   -b-      24.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    16   62   -b-      28.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    17   37   -b-      28.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    18   66   -b-      26.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    19   10   -b-      22.3  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    20   68   -b-      30.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    21   13   -b-      28.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    22   38   -b-      21.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    23   27   -b-      24.3  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    24   46   -b-      28.3  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    25   74   -b-      24.8  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    26   52   -b-      23.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    27   56   -b-      23.8  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    28   35   -b-      29.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    29   77   -b-      25.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    30   57   -b-      19.4  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    31   47   -b-      26.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    32   51   -b-      27.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    33   54   -b-      43.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    34   71   -b-      23.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    35   67   -b-      22.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    36   22   -b-      25.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    37   15   -b-      21.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    38   75   -b-      24.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    39    2   -b-      23.4  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    40   58   -b-      28.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    41   73   -b-      23.3  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    42   16   -b-      24.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    43   17   -b-      21.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    44   26   -b-      31.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    45   78   -b-      21.3  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    46    9   -b-      20.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    47   18   -b-      21.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    48   42   -b-      28.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    49   50   -b-      22.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    50   45   -b-      23.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    51   61   -b-      23.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    52   44   -b-      28.2  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    53   30   -b-      24.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    54    5   -b-      24.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    55   24   -b-      27.6  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    56   29   -b-      30.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    57   14   -b-      26.9  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    58   33   -b-      21.4  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    59    4   -b-      24.4  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    60   45   -b-      29.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    61   72   -b-      28.1  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    62    0   -b-      24.5  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    63   41   -b-      22.7  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    64    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    65    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    66    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    67    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    68    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    69    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    70    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    71    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    72    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    73    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    74    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    75    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    76    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    77    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    78    -   --p       0.0  all / all
      0__SAP S4H S4HANA 2.0 1709 SP4       2    79    -   --p       0.0  all / all
      
      

      Can you see the "State" flag "-b-" for CPU 1-63 seems okbut for CPU 64-79 it says "--p" what does that mean ?

      Best regards

      posted in Compute
      S
      sapcode