XOA json-rpc call basic exemple

mco-system

Hello,

This is my first post here, I apologize for its length.

I'm trying to understand how to use xo-api call to perform some actions programmatically.

I'm aware of how to use a ReST API but I don't know much about JSON-RPC.

So I looked into xen-orchestra code base (expecially xo-cli, xo-lib, xo-server) and also the xen-orchestra terraform provider which I already use with success.

Unfortunately I'm not enough well versed in JS nor Go to really understand how to correctly request the JSON-RPC API, not even even to sucesfully connect to the XOA (from source) instance via JSON-RPC.

In fact, I'm not even sure of the exact exposed URL of the JSON-RPC API !

I tried with my favorite language, Python3, using "jsonrpcclient" to connect passing module without success.

# Very basic json-rpc auth attempt

from jsonrpcclient import request

# url = 'wss://xoa.local.example.com'
# url = 'wss://xoa.local.example.com/api'
# url = 'https://xoa.local.example.com'
url = 'https://xoa.local.example.com/api'
credentials = {'email': 'test@example.com', 'password': 'testme'}

response = request(url, "signin", credentials=json.dumps(credentials)

The above code fails on a misinterpretation of the XO server response. I guess the response is not JSON formated but more something like HTTP 302 response.

I probably miss some very basic knowledge but can't find any direction.
I need to know:

What precise URL to call
What form and content should have the given parameters

Does someone could be kind enough to post a basic code snippet in another language than JS or Go that use the XOA JSON-RPC, even with plain cURL ?

A sample like the following one would help me a lot to understand (the following code snippet is not working):

curl --include \
     --no-buffer \
     -H "Host: xoa.local.example.com:443" \
     -H "Origin: https://xoa.local.example.com:443" \
     -H "Upgrade: websocket" \
     -H "Connection: Upgrade" \
     -H "content-type: application/json" \
     -H "Accept: application/json" \
     -X POST https://xoa.local.example.com/api \
     -d '{"method":"signin","credentials":{'email': 'test@example.com', 'password': 'testme'}}'

olivierlambert

Pinging @julien-f

julien-f

@mco-system I don't know how to do a WebSocket call with cURL, it's probably not a simple HTTP request.

julien-f

Maybe try something like https://github.com/websockets/wscat, though I have no experience with it.

mco-system

@julien-f Thank you for your reply.

I tried wscat and I can see I'm successfully connected at TCP level as I would with a basic telnet client but then I have no response from the server at all. until timeout occurs.

Though, the xo terraform provider is working without problem, so I guess I'm doing something wrong or miss a very basic step at protocol level (i'm very new at using json-rpc).

I also don't get how does the authentication mechanism work.
Where and when send credentials to the API ?

/me is currently reading json-rpc specification

mco-system

I successfully sniffed traffic between the XOA Terraform provider and I have now a better understanding of what a JSON-RPC requests looks like.

However, I'm still unable to connect to the XO host with wscat which systematically timeouts .
I can't get the Connected (press CTRL+C to quit) prompt from wscat as I would if the connection was successful.

However, on XO host, I can see traffic using tcpdump, the problem does not seem network, nor SSL related.

xoa:/opt/xo/xo-server# tcpdump -nni eth0 host 192.168.2.1 and not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes


00:28:35.456615 IP 192.168.2.1.49074 > 192.168.2.251.443: Flags [S], seq 3784699052, win 64240, options [mss 1460,sackOK,TS val 3147372683 ecr 0,nop,wscale 7], length 0
00:28:35.456660 IP 192.168.2.251.443 > 192.168.2.1.49074: Flags [S.], seq 962165778, ack 3784699053, win 65160, options [mss 1460,sackOK,TS val 4217115831 ecr 3147372683,nop,wscale 7], length 0
00:28:35.456904 IP 192.168.2.1.49074 > 192.168.2.251.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3147372684 ecr 4217115831], length 0
00:28:35.457104 IP 192.168.2.1.49074 > 192.168.2.251.443: Flags [P.], seq 1:372, ack 1, win 502, options [nop,nop,TS val 3147372684 ecr 4217115831], length 371
00:28:35.457120 IP 192.168.2.251.443 > 192.168.2.1.49074: Flags [.], ack 372, win 507, options [nop,nop,TS val 4217115831 ecr 3147372684], length 0
00:28:35.460902 IP 192.168.2.251.443 > 192.168.2.1.49074: Flags [P.], seq 1:1672, ack 372, win 507, options [nop,nop,TS val 4217115835 ecr 3147372684], length 1671
00:28:35.461216 IP 192.168.2.1.49074 > 192.168.2.251.443: Flags [.], ack 1672, win 495, options [nop,nop,TS val 3147372688 ecr 4217115835], length 0
00:28:35.462613 IP 192.168.2.1.49074 > 192.168.2.251.443: Flags [P.], seq 372:700, ack 1672, win 501, options [nop,nop,TS val 3147372690 ecr 4217115835], length 328
00:28:35.463727 IP 192.168.2.251.443 > 192.168.2.1.49074: Flags [P.], seq 1672:2214, ack 700, win 505, options [nop,nop,TS val 4217115838 ecr 3147372690], length 542
00:28:35.464040 IP 192.168.2.1.49074 > 192.168.2.251.443: Flags [.], ack 2214, win 501, options [nop,nop,TS val 3147372691 ecr 4217115838], length 0

In xo-server config.toml, I turned verboseApiLogsOnErrors to true and restarted the xo-server service but the log does not show any connection attempt.

It's just like the connection is lost by the xo-server... I'm very confused...

mco-system

Using opensll s_client I can connect and upgrade to WebSocket without any difficulties:

openssl s_client -crlf -connect xoa.local.example.com:443 -servername xoa.local.exampe.com
CONNECTED(00000003)
depth=1 C = NC, ST = Somewhere, L = Shell-city, O = Example Co, OU = Lab, CN = Root CA example.com
verify return:1
depth=0 C = NC, ST = Somewhere, O = Example Co, CN = xoa.example.com
verify return:1
[...]

GET /api/ HTTP/1.1
Host: xoa.local.example.com:443
User-Agent: Go-http-client/1.1
Connection: Upgrade
Sec-WebSocket-Key: 7lcEMScTa7sTRzwW8jebLQ==
Sec-WebSocket-Version: 13
Upgrade: websocket

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: 5pKEtEmZTJbNiFx/gZ9VIMVy2B8=

And the corresponding line in syslog

Apr 30 02:22:04 xoa xo-server[31752]: 2021-04-30T02:22:04.785Z xo:main INFO + WebSocket connection (192.168.2.1)

So everything should work but I'm still missing something that I'm pretty sure is very simple. Some basic stuff that I can not see by myself....
Very frustrating...

olivierlambert

Don't worry @mco-system @julien-f will answer when he's around

julien-f

@mco-system I don't know wcat, I just mentioned it because it looked interesting.

I cannot explain how the WebSocket protocol works, all I can tell you is that XO API is JSON-RPC inside a WebSocket connection, and that before being able to use methods, you need to sign in via the session.signIn method:

# Open WebSocket connection to http://xoa/api/
→ {"id":0,"jsonrpc":"2.0","method":"session.signIn","params":{"username":"jsmith","password":"passw0rd"}}
← {"id":0,"jsonrpc":"2.0","result":{"id":"ad2593b0-97c7-446f-bc0e-f5558c013d52","email":"jsmith","groups":[],"permission":"admin","preferences":{}}}

If all you need is a CLI, you can use xo-cli.

mco-system

@julien-f Thank you for your help.

I finally figured out that I missed the API endpoint trailing / which seems mandatory.

My test with openssl s_client worked fine because the trailing / was in place:

GET /api/ HTTP/1.1

To answer your advice about xo-cli, I don't need a CLI tool and xo-cli is fine when I need one.

My goal is to determine what would be the needed efforts to code something like an xo-api Python connector that would allow requesting xo-server from Python apps and scripts.

Such a connector would be useful to me but also to all Python coders.
I'm seriously considering it.

Thank you again for your time, it could have take me days to figure out this missing trailing /.

mbaron

Hi @mco-system,

I propose to you a solution to communicate from a Python program to xo-server api.

import json
import aiohttp
import asyncio

from jsonrpc_websocket import Server

async def routine():
    async with aiohttp.ClientSession() as client:
        server = Server('ws://XO_SERVER_IP/api/', client)

        await server.ws_connect()

        # No signIn required
        methodsInfoResult = await server.system.getMethodsInfo()
        print('\n'.join([str(e) for e in methodsInfoResult.keys()]))

        # signIn required
        result = await server.session.signIn(username='YOUR_LOGIN', password='YOUR_PASSWORD') # email attribute is working in place of username
        result = await server.xo.getAllObjects(filter={"type": "VIF"}, limit=10)

        print('[')
        print(', \n'.join([str(json.dumps(e, indent=4)) for e in result.values()]))
        print(']')

asyncio.get_event_loop().run_until_complete(routine())

This code is using the jsonrpc_websocket library.

A full description is available on my blog => https://mickael-baron.fr/blog/2021/05/28/xo-server-websocket-jsonrcp (in french). A Java version is also available.

mco-system

Hello @mbaron,

Your blog helped me to get a better understanding of the different steps involved in jsonrpc.

Thank you for sharing your code samples.