Access historical CPU and memory metrics - Xen Orchestra v5

puneet336

Hi All,
I am in need to access the historical raw data of the XenOrchestra v5 to create some custom visualizations representing overall state of the systems.

I.e.
Dashboard>Overview>VMs> (select VM) > (select stats Tab)

Here we see the CPU and Memory usage of the servers, and i believe that this historical data is being pulled from a backend database.

Have few queries:

May i know which database is being used here to store CPU and Memory USage information?
Where are the raw data files located and how can i pull the raw data ? i plan to convert it into a json format for the custom visualization.

olivierlambert

Hi,

Data is pulled from https://xapi-project.github.io/xen-api/metrics.html in live in XO (they are not stored in XO)

We have plans to build an Open Metrics converter at some point (maybe as an XO plugin)

puneet336

@olivierlambert
Thank you for the link , i think this should help me:

wget http://xoahostname/vm_rrd?session_id=<token>&uuid=x&start=<secondsinceepoch>

seems it needs session id token. So in order to generate one , i installed XenAPI-24.15.0 , and used following snippet to generate the token -

import XenAPI
username = "user1"
password = "user1's ldap password"
url = "http://xoahostname.example.com"
session = XenAPI.Session(url)
session.xenapi.login_with_password(username, password, "1.0", "session_getter")
print(session._session)

i got following error -

File "./test.py", line 6, in <module>
  session.xenapi.login_with_password(username, password, "1.0", "session_getter")
File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 260, in __call__
  return self.__send(self.__name, args)
File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 145, in xenapi_request
  self._login(methodname, params)
File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 170, in _login
  getattr(self, 'session.%s' % method)(*params))
File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1112, in __call__
  return self.__send(self.__name, args)
File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1452, in __request
  verbose=self.__verbose
File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1154, in request
  return self.single_request(host, handler, request_body, verbose)
File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1187, in single_request
  dict(resp.getheaders())
xmlrpc.client.ProtocolError: <ProtocolError for xoahostname.example.com/RPC2: 302 Found>

I retried with HTTPS

url = "https://xoahostname"

and i get following error -

Traceback (most recent call last):
  File "./test.py", line 6, in <module>
    session.xenapi.login_with_password(username, password, "1.0", "session_getter")
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 260, in __call__
    return self.__send(self.__name, args)
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 145, in xenapi_request
    self._login(methodname, params)
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 182, in _login
    raise e
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 170, in _login
    getattr(self, 'session.%s' % method)(*params))
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1166, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1279, in send_request
    self.send_content(connection, request_body)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1309, in send_content
    connection.endheaders(request_body)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

Any hints on know what additional flags are required in the XENApi's python api to make this work ?

We are using xo-server 5.111.1 , XCP-ng 8.2 OS. This version supports the pythonapi which i am trying to use ?
If i am able to generate the session id token, how long will this token valid? can i use this same token for 3 hours to download data from across multiple VMs ?
the user1 account is a non admin account but with privileges to create/terminate VMs and view host utilization metrics. Do i strictly need to use the root user account's session id to ken to use the API?. user1 has permission to access the following sections of the XOA UI
Instead of the python API , the tokens generated using the following web page are valid to be used with wget as sessionid?

puneet336

UPDATE:
wget needs session id, uuid and epoch time so regarding the point number 4 in my previous query. i tried to obtain that information from the WebUI as follows

I generated a token from the following page -
got the uuid from the web UI :

took epoch time for start argument -

used the token , uuid and start time generated from the previous 3 steps. Seems i am being redirected to the login page.

wget --no-check-certificate -k "https://xoaserver1.example.com/vm_rrd\?session_id=OpaqueRef:GnpwaCjEoMA-n6a8bhfAG-ktGXXXXXQDea-BXXXXXXU\&uuid=37fdd603-e31f-7de4-b7e4-9XXXXXXXX2e9\&start=1716212040"

Seems i am being redirected to the login web page with following content :

<!DOCTYPE html><html><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>Xen Orchestra</title><meta name="author" content="Vates SAS"><link rel="stylesheet" href="https://xoaserver1.example.com/index.css"></head><body style="display: flex; height: 100vh;"><div style="margin: auto; width: 20em;"><div class="mb-2" style="display: flex;"><img src="https://xoaserver.example.com/assets/logo.png" style="margin: auto;"></div><h2 class="text-xs-center mb-2">Xen Orchestra</h2><div class="mb-2"></div><form action="https://xoaserver1.example.com/signin/local" method="post"><fieldset><div class="input-group mb-1"><span class="input-group-addon"><i class="xo-icon-user fa-fw"></i></span><input class="form-control" autofocus name="username" placeholder="Username" required type="text"></div><div class="input-group mb-1"><span class="input-group-addon"><i class="fa fa-key fa-fw"></i></span><input class="form-control" name="password" type="password" placeholder="Password" required></div><div class="checkbox"><label><input name="remember-me" type="checkbox">&nbsp;
Remember me</label></div><div><button class="btn btn-block btn-info"><i class="fa fa-sign-in"></i> Sign in</button></div></fieldset></form></div></body></html>

olivierlambert

Don't mix an XO token with a XAPI token, they are 2 different things.

puneet336

@olivierlambert Thank you for the prompt response and i Got it, so i will focus on getting token via XenAPI python interface.

With following code -

#!/usr/bin/env python3
import XenAPI
username = "user1"
password = "user1's ldap password"
url = "https://xoaserver1.example.com"
session = XenAPI.Session(url)
session.xenapi.login_with_password(username, password, "1.0", "session_getter")
print(session._session)

i am getting SSL error

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

Using the information from the following post i modified the code to

session = XenAPI.Session(url,ignore_ssl=True)

and i see following error

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    session.xenapi.login_with_password(username, password, "1.0", "session_getter")
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 260, in __call__
    return self.__send(self.__name, args)
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 145, in xenapi_request
    self._login(methodname, params)
  File "/work/user1/MyWork/XAPI/venv01/lib/python3.6/site-packages/XenAPI/XenAPI.py", line 170, in _login
    getattr(self, 'session.%s' % method)(*params))
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/tools/FOSS/python3/3.6.7/x86_64/lib/python3.6/xmlrpc/client.py", line 1187, in single_request
    dict(resp.getheaders())
xmlrpc.client.ProtocolError: <ProtocolError for xoaserver1.example.com/RPC2: 302 Found>

I am stuck here, any help/hint would be helpful.

I was using the following python package - XenAPI-24.15.0
OS xo-server 5.111.1 , XCP-ng 8.2 OS

olivierlambert

IDK if this lib is still working, maybe @yann can take a quick look

puneet336

@olivierlambert Thank you ,
Hi @yann
Please do let me know if you think is version incompatibility may be an issue here.
I was using python 3.6 + . XenAPI-24.15.0. and i am using a remote host

i.e. test.py was executed from server1 ---connects to--> xoaserver1.example.com

I plan to try out 3.9 or higher versions today
OS xo-server 5.111.1 , XCP-ng 8.2 OS

yann

@puneet336 @olivierlambert the lib is still working, but I'm usually use it with a local connection, running scripts inside dom0, with this code:

def get_xapi_session():
    session = XenAPI.xapi_local()
    try:
        session.xenapi.login_with_password('root', '', '', 'xcp-ng-tests session')
    except Exception as e:
        raise Exception('Cannot get XAPI session: {{}}'.format(e))
    return session

Will check what happens with a remote session.

yann

@puneet336 you need to make your connection to your pool master (not your XOA guest), that's where the XAPI endpoint is.

If you did not deploy TLS certificates on your pool, XenAPI.Session(URL) will complain that the SSL certificate is self-signed (which you can workaround by adding ignore_ssl=True as you did above, but don't keep this in production).

You will also find a few guidelines to write a robust XenAPI client at https://xapi-project.github.io/xen-api/usage.html

puneet336

@yann i tried running the following script from the xoa server.

#!/usr/bin/env python3
import XenAPI
username = "username"
password = "ldap password"
#url = "https://xoaserver1.example.com"
session = XenAPI.xapi_local()

#session.xenapi.login_with_password('root', '', "", "xcp-ng-tests session")
#session.xenapi.login_with_password(username, password, "1.0", "session_getter")
session.xenapi.login_with_password(username, password, "", "xcp-ng-tests session")


print(session._session)

i see following error :

Traceback (most recent call last):
  File "/home/user1/MC3/test.py", line 10, in <module>
    session.xenapi.login_with_password(username, password, "", "xcp-ng-tests session")
  File "/home/user1/MC3/envs/user1env/lib/python3.12/site-packages/XenAPI/XenAPI.py", line 260, in __call__
    return self.__send(self.__name, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/MC3/envs/user1env/lib/python3.12/site-packages/XenAPI/XenAPI.py", line 145, in xenapi_request
    self._login(methodname, params)
  File "/home/user1/MC3/envs/user1env/lib/python3.12/site-packages/XenAPI/XenAPI.py", line 182, in _login
    raise e
  File "/home/user1/MC3/envs/user1env/lib/python3.12/site-packages/XenAPI/XenAPI.py", line 170, in _login
    getattr(self, 'session.%s' % method)(*params))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/MC3/envs/user1env/lib/python3.12/xmlrpc/client.py", line 1122, in __call__
    return self.__send(self.__name, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/MC3/envs/user1env/lib/python3.12/xmlrpc/client.py", line 1461, in __request
    response = self.__transport.request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/MC3/envs/user1env/lib/python3.12/xmlrpc/client.py", line 1166, in request
    return self.single_request(host, handler, request_body, verbose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/MC3/envs/user1env/lib/python3.12/xmlrpc/client.py", line 1178, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/MC3/envs/user1env/lib/python3.12/xmlrpc/client.py", line 1291, in send_request
    self.send_content(connection, request_body)
  File "/home/user1/MC3/envs/user1env/lib/python3.12/xmlrpc/client.py", line 1321, in send_content
    connection.endheaders(request_body)
  File "/home/user1/MC3/envs/user1env/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/user1/MC3/envs/user1env/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/home/user1/MC3/envs/user1env/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/home/user1/MC3/envs/user1env/lib/python3.12/site-packages/XenAPI/XenAPI.py", line 99, in connect
    self.sock.connect(path)
FileNotFoundError: [Errno 2] No such file or directory

Did i miss anything while running the script?

yann

@puneet336 xapi_local() will only work in the dom0 (it looks for a local socket to the XAPI daemon, which is why you get a FileNotFoundError). If you run it from the XOA appliance (or from anywhere else than the dom0, you'll need XenAPI.Session(URL), but again with a URL to the dom0, not your XOA appliance.

Remember XAPI is a service running in the host's dom0, XOA is just a client connecting to that service.

puneet336

Thank you for the response @yann . I was not aware about the dom0 and came across https://stackoverflow.com/questions/19147398/is-xen-dom0-a-guest-or-a-host ,

I believe i ran the python script from within the dom0 VM i.e. the server which hosts the Xen XOA Web UI. When i log in to the server ( which i beleive to be dom0 as of now) i see the following banner :

Is there a way to list out the IP of the dom0 vm ? or
Any method confirm if i am running the scripts from the dom0 VM (presence of any special file or any command which can list out the same)?

puneet336

Hi @yann ,
is there way to get the ip of dm0 vm via cli/ Web GUI?

AtaxyaNetwork

@puneet336 Hi,

The IP of the dom0 is the IP of the "host" you added in Xen Orchestra. You can find it the settings -> server tab

puneet336

@AtaxyaNetwork Thank you for the response,

I was unable to see settings Option from the Web UI,
but using Home>Pools, i saw following :

then i clicked on the 4x pools to get information on the hosts for this pool

I was able to get the name of the master server for this pool (tagged as master), say pool1.master.example.com

when i use hostname=pool1.master.example.com

#!/usr/bin/env python3
import XenAPI
username = "root"
password = "root account password"
url = "https://pool1.master.example.com"
session = XenAPI.Session(url)
session.xenapi.login_with_password(username, password, "1.0", "session_getter")
print(session._session)

i am able to see session tokens

identified uuid of one of the VM being managed by pool master

and i am able to download the data using :

wget -c  "http://pool1.master.example.com/host_rrd?session_id=pool1.master.opaqreftoken&uuid=VM1uuid" --output-document test.xml

puneet336

Hi @olivierlambert , @yann
I was able to get rrd data using instructions i mentioned in previous comment, But i am not sure if it contains historical data or this is just current metrics snapshot.

wget -c  "http://pool1.master.example.com/host_rrd?session_id=pool1.master.opaqreftoken&uuid=VM1uuid" --output-document test.xml

but the file which i downloaded was just 20MB file and this host has been up since last 1 year.
Not sure if the xml file which i got will have entire year's data as there is just 1 lastupdate tag in the document , i.e. <lastupdate>1718163483</lastupdate>

I am looking to parse the xml for raw data and dump the same in following form for each hosts's xml file
timestamp1, cpu metrics1, memory usage metrics1
timestamp2, cpu metrics2, memory usage metrics2
timestamp3, cpu metrics3, memry usage metrics3
timestamp4, cpu metrics4, memry usage metrics4

Could you please share some information or link which can help me understand the format of xml or,
if you could share information about the tags which i need to look upon ?

Though i am going through the xml document , Any help hint which could help me understand the format or obtain memory and cpu metrics would help.

olivierlambert

Have you read https://xapi-project.github.io/xen-api/metrics.html ? It explains the diff between getting all data and only the diff (delta). It's a must read to understand the way RRDs can be fetch.