• 0 Votes
    29 Posts
    5k Views
    A
    Apologies for the long silence on my end — real life got in the way and I couldn't dig into this for a while. Finally found the time now to go through it again properly, helped a lot by the details others have already posted here in the meantime. I don't think this adds a new root cause, but it does corroborate what @kagbasi-wgsdac described — same "mass collapse onto a single anchor UUID" pattern, and in my case too that anchor is a real, still-in-use base disk, not a dangling/non-existent OpaqueRef (referencing vatesfr/xen-orchestra#9578). Environment: 2-host pool, shared NFS SR, XO recently updated to the latest version (didn't change anything regarding this issue, as expected since it's XAPI-side). Ran a read-only xe vdi-list params=uuid,name-label,is-a-snapshot,snapshot-of,snapshot-time directly against the pool master. Result: 47 of 196 VDIs match the pattern (is-a-snapshot: false but snapshot-of populated). All 47 point to the exact same single anchor UUID — which resolves to one specific, actively-used base VDI, not a missing reference. snapshot-time on most affected VDIs is the epoch default (1970-01-01T00:00:00Z), but a handful show plausible real dates (e.g. late 2025 / early 2026) — suggesting those entries originally had a legitimate snapshot-of relationship that got overwritten by whatever corrupted the metadata. Currently-running VMs affected (100% of each VM's disks affected in every case): VM OS Disks Sizes VM-1 Windows Server 2016 1 50 GB VM-2 Windows Server 2016 3 50 / 200 / 450 GB VM-3 Windows Server 2019 3 100 / 100 / 100 GB VM-4 Windows 11 Pro 2 50 / 100 GB VM-5 (decommissioned) Windows Server 2016 2 100 / 50 GB VM-6 Ubuntu 24.04 2 10 / 10 GB VM-7 Ubuntu 24.04 2 10 / 10 GB VM-8 Windows Server 2025 3 25 / 25 / 64 GB VM-9 Ubuntu 24.04 2 10 / 10 GB On top of that, ~28 more affected VDIs are orphaned/unattached objects ("base copy" leftovers, old ISO references) — same pattern, no VBD attached. Not touching any of this (no snapshot-fixer.py, no manual vdi-param-set) given the risk of severing legitimate snapshot relationships that's already been flagged here. Happy to provide a full anonymized xe vdi-list dump if that's useful for tracking down the root cause. Big thanks to everyone who kept digging into this and shared their findings here — especially @kagbasi-wgsdac for the detailed write-up that pointed me in the right direction, and of course @poddingue and the whole Vates team for staying on top of this and keeping us updated despite no clear timeline yet. Really appreciate the effort that goes into this, especially for something as tricky as a metadata corruption bug across production pools. Kind Regards and thx again Alex
  • sr iso disconnect and crashed my hosts

    XCP-ng
    12
    0 Votes
    12 Posts
    2k Views
    H
    @olivierlambert We are still seeing similar issues on fully updated xcp-ng 8.3 host. We do reboot every time it happens. Is there anything we can do to fix this otherwise or prevent this from happening (beside manually disconnecting the cifs iso storage before the san goes for a reboot)?
  • XCP-ng Windows PV tools announcements

    Moved News
    114
    0 Votes
    114 Posts
    39k Views
    Y
    @dinhngtu Yes, I've submitted a false positive report here: https://www.bitdefender.com/en-us/business/submit
  • 🛰️ XO 6: dedicated thread for all your feedback!

    Pinned Xen Orchestra
    272
    7 Votes
    272 Posts
    125k Views
    C
    @julienXOvates Great, thanks Julien!
  • Error: Can't init vhd directory without using alias

    Solved Backup
    12
    1 Votes
    12 Posts
    368 Views
    A
    @pierrebrunet Tested and working again, thanks
  • Master, commit a3139 failing backups

    Solved Backup
    23
    0 Votes
    23 Posts
    3k Views
    P
    @jr-m4 Hi, We have found this issue in another thread, it is now merged
  • 0 Votes
    18 Posts
    1k Views
    A
    Hello @mike.potapov , We found a race-condition happening on Qcow2 driver that explains this issue. We developed a fix that will be ship in the next release of blktap.
  • 2 Votes
    16 Posts
    1k Views
    M
    @acebmxer Very interesting. Thanks for sharing! I will wait until Veeam issued a new (hotfix) release before evaluating Veeam as a replacement for Xen Orchestra Backup.
  • Can't init vhd directory without using alias

    Backup
    1
    0 Votes
    1 Posts
    41 Views
    No one has replied
  • VDI export to VMDK results in a corrupted disk

    Solved Xen Orchestra
    15
    0 Votes
    15 Posts
    283 Views
    A
    @Emmanuel-V In my opinion, it would make more sense if a standalone disk exported in VMDK format were exported directly as monolithicSparse, so that it would not need to be converted. This makes more sense to me because when only the disk is exported, rather than the entire VM (OVA), it can be assumed that the disk will be attached directly to some VM.
  • Troubleshooting "TCP: out of memory" - Possible memory leak?

    Unsolved Xen Orchestra
    12
    1
    0 Votes
    12 Posts
    525 Views
    J
    @florent & @poddingue I now have two heap snapshots. Could you provide somewhere to upload them, please? And I will do so as soon as I can.
  • XO NFS option sec=krb5p encrypted transport

    Xen Orchestra
    3
    0 Votes
    3 Posts
    107 Views
    BytevenidosB
    I do have active directory setup in my lab environment. I just need to set aside some time to try it out between two domain joined hosts. Thanks for trying that! That's some good debugging.
  • 0 Votes
    114 Posts
    17k Views
    poddingueP
    Thanks a lot for your feeback, folks!
  • 3 Votes
    4 Posts
    204 Views
    acebmxerA
    v0.8.0–v0.9.1 — multi-user accounts, self-update, built-in HTTPS, and date ranges. Three releases since the last update, so bundling them here. Date ranges. Collections, extractions, findings runs and the support package can now be scoped to a window instead of always covering the whole bundle — last 24h/7d/30d, since last reboot, or a custom range. XO's own routes still have no date filter, so the first download is unchanged size — the range only narrows what gets kept and reported afterwards. Redact on demand. A collection no longer has to redact immediately with whatever rules happen to be on at that moment. There's now a checkbox to store the raw bundle only, and a "Redact now" button on the card afterwards using whichever rules are switched on when you press it. Multiple user accounts, roles, and an activity log. This was on the "what is coming" list in the first post and it's in now. Any number of accounts, three roles — admin, operator, viewer (read-only, but can still download what's already stored) — and an Activity page logging logins, settings changes, and job runs. Optional 2FA. Per-user TOTP, off by default, each account turns it on for itself. QR code setup, backup codes. Self-update from the UI, opt-in and off by default. Checking needs nothing extra; applying an update needs the Docker socket mounted in explicitly, since that's effectively host root, so it's a separate deliberate step — uncomment the socket mount and group_add block in docker-compose.yml, and it needs its own .env file (just DOCKER_GID=..., next to docker-compose.yml, not the same file as xcp-pulse.env) or docker compose up -d fails with unable to find group. .env.example has the one-liner to generate it. Built-in HTTPS, no reverse proxy required. Bundles nginx into the image to terminate TLS — self-signed cert on first start, or upload your own from Settings. A reverse proxy still works fine too. A user manual in the app itself, under /help — no need to leave XCP Pulse or check out the repo to read how something works. Fixed The account-requirements note from the first two posts was wrong in a way I only found by testing it properly: export:logs — the privilege log downloads actually need — isn't in any built-in XO role, but it can be granted through a custom role. "Test connection" was reading a catalogue that can't tell you that, and told every restricted account it needed full admin regardless. It now actually probes the download endpoint, and the docs walk through creating the custom role via three REST API calls (no UI for it yet on either XO version). A real concurrency bug: the shared SQLite connection wasn't locked across fetches, only execute, so two requests landing close together could crash a page reading jobs — reproduced it under the test suite's own concurrency test. Base-OS CVEs patched in the image build (perl-base, libc6, others); CI now runs a Trivy scan on every push. Still not fixed: the truncated-download-behind-a-reverse-proxy issue from the last post. Haven't found the actual cause yet. Also looking for anyone who can test against a remote proxy in a lab setup. Also open to any other suggestions, features, improvements, UI changes, etc... If any chance someone on Vates would test on their own time. Things like the support bundle and or the logs themselves are not being manipulated in any unwanted ways (for Vates or the project) https://github.com/acebmxer/xcp_pulse
  • MS-01 performance issues w/ Intel 226 NICs

    Hardware
    11
    0 Votes
    11 Posts
    3k Views
    B
    @Andrew said: pcie_aspm=disable Just to help anyone who would run into it. Disabling ASPM via dom0 settings/kernel did not resolve the issue. Had to disable it in BIOS (NUC13) After kernel level disable it did show: lspci -vv -s 55:00.0 | grep -E 'LnkCap|LnkCtl|LnkSta' LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <2us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Only after BIOS disable it showed lspci -vv -s 55:00.0 | grep -E 'LnkCap|LnkCtl|LnkSta' LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <2us, L1 <4us LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- Running on NUC13ANBi7 Hope it helps anyone running into this problem.
  • Date format on web interface: Only US format available?

    Unsolved Compute
    11
    2 Votes
    11 Posts
    1k Views
    J
    @julienXOvates said: @acomav We will actually propose to change the date and time format in XO6 settings, so you would be able to choose between : YYYY-MM-DD MM/DD/YYYY DD/MM/YYYY and 12h or 24h time format. Hopefully in one of the next 3 months ! I hope that will answer your need, otherwise let me know ! Thanks for putting this one the roadmap, for one of the next 3 months. That date order issue, has been bugging me as if you are rushed off your feet, may not remember that it’s the US date order. Also fits in more with the rest of European software used, or configured for the European localisation.
  • 0 Votes
    18 Posts
    2k Views
    J
    @johnnezero said: @john.c Thank you for all the great ideas, keep em' comming! Your welcome. If your other plugin was vibe coded it may be worth doing the same for the other. If you’re still working on it! @julienxovates In regards to the XO updates with the upcoming encrypted XO data store, by default (opt in for now). Is it ready for plugins like the one done by this author, and others in the future please?
  • 0 Votes
    4 Posts
    157 Views
    dthenotD
    @Dan Hello, So our investigation found that this change was introduced with commit https://github.com/xcp-ng/sm/commit/00637dd52e845d6016add9718fc9cc694aec9f0d It was aimed at EXTSR in particular, it appear that the first SR to be plugged is the one choosing the mode of sr-mount since util.makedirs also create the parent directory with the given mode. I have created a card for the issue on our side.
  • Backup remote repository structure

    Backup
    2
    0 Votes
    2 Posts
    89 Views
    P
    @abudef Hi, Do you think this doc can help you? Link to doc I think it needs some improvements, feel free to enhance it if you feel it is necessary!
  • VIDs are showing up as snapshot, but they are not.

    Unsolved XOSTOR
    7
    1
    0 Votes
    7 Posts
    477 Views
    K
    @olivierlambert I'm seeing a recurrence of the snapshot_of / hidden-disks issue after patching to xapi-26.1.16-1.2 (both hosts, rebooted). Unusual element this time: snapshot-fixer.py dry-run flags ~260 VDIs across the SR that all cite a single OpaqueRef which resolves to no VDI (xe vdi-list | grep <ref> returns nothing). Holding off on rewrite until I understand whether that single-dangling-ref pattern is safe, given the set appears to include base VDIs of legitimate snapshots. Full technical detail and questions posted on GitHub: https://github.com/xcp-ng/xcp/issues/844.