XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. kagbasi-wgsdac
    K Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 13
    • Posts 63
    • Groups 0

    kagbasi-wgsdac

    @kagbasi-wgsdac

    10
    Reputation
    12
    Profile views
    63
    Posts
    0
    Followers
    0
    Following
    Joined
    Last Online

    kagbasi-wgsdac Unfollow Follow
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue Bug report filed as requested — https://github.com/xcp-ng/xcp/issues/825 — and tagging @Team-Storage per your suggestion.

      Full evidence bundle is attached to the issue (versions, sweep output, vhd-util vs xe comparison, SMlog). Summary of what I found:

      One correction to the mechanism, and I think it matters. The recap describes is-a-snapshot being flipped to true. On my system that isn't what's happening — is-a-snapshot is false on every affected VDI. The field being wrongly written is snapshot-of, which is getting populated on base disks that aren't snapshots at all. XO's disappearing-disks symptom is consistent with either (it filters on a non-empty snapshot-of), but if the storage team is hunting for a bad is-a-snapshot write, that may be the wrong field. Every affected VDI here looks like:

      is-a-snapshot: false      <-- correct
      snapshot-of:   <populated with an unrelated VDI's UUID>   <-- wrong
      

      A VDI that is a snapshot of itself. The clearest single artifact:

      uuid:          806f7f42-083f-4a40-b3f1-0700d00bab5a
      name-label:    WinSrv2022SHB_Disk1_Data
      is-a-snapshot: false
      snapshot-of:   806f7f42-083f-4a40-b3f1-0700d00bab5a   <-- itself
      snapshot-time: 20260709T11:19:15Z
      sm-config:     vhd-parent: c86e3247-...   <-- bears no relation to the snapshot-of value
      

      No valid code path produces snapshot-of = self. Whatever writes this field isn't validating the target.

      It's still actively corrupting new VDIs — this is not just legacy damage. That self-referential VDI was created 2026-07-09, a week after my patch + reboot. Sweeps 9 days apart went from ~180 → 191 affected VDIs on one SR, and a fourth anchor UUID appeared that didn't exist in the first sweep. Newly created VHDs keep landing in the affected set. So "stop it happening again" is the urgent half of the two-part fix, at least in my case.

      The bogus targets cluster onto a tiny anchor set, and the anchors point at each other:

      Count Anchor
      97 937c3945 (→ a893fdb4)
      50 a893fdb4 (→ ea150883)
      37 ea150883
      7 806f7f42 (→ itself, new since Jul 9)

      That looks less like corrupted lineage and more like the field being filled from an incorrect/uninitialised source.

      On-disk VHDs are completely healthy. vhd-util check says valid, parent locators are consistent, GC reports no work. The two VDIs the DB calls parent/child are, on disk, siblings under a common parent. The corruption is purely in the XAPI database — which is good news for recoverability.

      The VDI_IN_USE is not a real lock. current-operations is empty, xe task-list is empty, no tapdisk holds it. VM.start fails because it's walking a snapshot relationship that doesn't exist on disk. Reproduces from xe on the pool master with XO entirely out of the path — which is why I filed against xcp-ng/xcp rather than the XO tracker.

      Versions: XCP-ng 8.3.0, xapi 26.1.11 (xapi-core-26.1.11-1.2), sm-3.2.12-17.9, sm-fairlock-3.2.12-17.9, blktap 3.55.5-9.1, build 20260618.

      I have not attempted to bulk-clear the fields — on-disk data is intact and I'd rather not do a mass write against the XAPI DB on a live SR without guidance. Backing store snapshotted as a safety net.

      Happy to run whatever diagnostics would help. And +1 to the hand-grenade feeling — the affected set growing on its own is the part that worries me.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XCP-ng v8.3 Host Crashing Upon Console Login and Performing Any Action

      Good evening all,

      Just a quick update. It's been a couple of days now after the rebuild and everything seems to be humming along fine, so I believe this topic can be marked as resolved. I can confidently conclude that this wasn't an XCP-ng issue, although the error message seems a bit misleading.

      posted in XCP-ng
      K
      kagbasi-wgsdac
    • RE: Need Help Understanding the VM Suspend Process

      @olivierlambert Yes sir, it is and I'm glad I confirmed this for myself. Thanks also for helping me understand how the VM Suspend process works. Hopefully this post helps other newbies with the same understanding in the future.

      posted in Management
      K
      kagbasi-wgsdac
    • RE: Cannot Download Exported VM After Export Task Completes Successfully

      @olivierlambert I didn't time it. I'm leaving home to drop the kids off at school, then off to work afterwards. I'll run it again and time it and report back.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: ISO Import to Local Storage via XO Not Working

      @olivierlambert Oh nice, hope it's not too cold over there. It's 9AM over here in Maryland and a bit nippy but not too cold.

      Nope, no host or pool logs available (both are empty). Then again, I cleared all the alerts from the dashboard earlier this morning, so not sure if that's what wiped all the logs. I'm not too worried, as this is a testing environment so I'm using this host to do a lot of learning.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue You're most welcome, sir. 👍

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue Thanks for the response. I got side tracked with another issue and didn't get that bug report filed, I'll do that as soon as possible.

      It feels like I'm playing with a hand grenade...lol, quite scary!

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @john.c Unfortunately, I cannot answer your question in the affirmative. I only use XCP-ng with TrueNAS.

      @Team-Storage have y'all had a chance to take a look at the bug report I filed? This issue is quite serious.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue You're most welcome, sir. 👍

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue Bug report filed as requested — https://github.com/xcp-ng/xcp/issues/825 — and tagging @Team-Storage per your suggestion.

      Full evidence bundle is attached to the issue (versions, sweep output, vhd-util vs xe comparison, SMlog). Summary of what I found:

      One correction to the mechanism, and I think it matters. The recap describes is-a-snapshot being flipped to true. On my system that isn't what's happening — is-a-snapshot is false on every affected VDI. The field being wrongly written is snapshot-of, which is getting populated on base disks that aren't snapshots at all. XO's disappearing-disks symptom is consistent with either (it filters on a non-empty snapshot-of), but if the storage team is hunting for a bad is-a-snapshot write, that may be the wrong field. Every affected VDI here looks like:

      is-a-snapshot: false      <-- correct
      snapshot-of:   <populated with an unrelated VDI's UUID>   <-- wrong
      

      A VDI that is a snapshot of itself. The clearest single artifact:

      uuid:          806f7f42-083f-4a40-b3f1-0700d00bab5a
      name-label:    WinSrv2022SHB_Disk1_Data
      is-a-snapshot: false
      snapshot-of:   806f7f42-083f-4a40-b3f1-0700d00bab5a   <-- itself
      snapshot-time: 20260709T11:19:15Z
      sm-config:     vhd-parent: c86e3247-...   <-- bears no relation to the snapshot-of value
      

      No valid code path produces snapshot-of = self. Whatever writes this field isn't validating the target.

      It's still actively corrupting new VDIs — this is not just legacy damage. That self-referential VDI was created 2026-07-09, a week after my patch + reboot. Sweeps 9 days apart went from ~180 → 191 affected VDIs on one SR, and a fourth anchor UUID appeared that didn't exist in the first sweep. Newly created VHDs keep landing in the affected set. So "stop it happening again" is the urgent half of the two-part fix, at least in my case.

      The bogus targets cluster onto a tiny anchor set, and the anchors point at each other:

      Count Anchor
      97 937c3945 (→ a893fdb4)
      50 a893fdb4 (→ ea150883)
      37 ea150883
      7 806f7f42 (→ itself, new since Jul 9)

      That looks less like corrupted lineage and more like the field being filled from an incorrect/uninitialised source.

      On-disk VHDs are completely healthy. vhd-util check says valid, parent locators are consistent, GC reports no work. The two VDIs the DB calls parent/child are, on disk, siblings under a common parent. The corruption is purely in the XAPI database — which is good news for recoverability.

      The VDI_IN_USE is not a real lock. current-operations is empty, xe task-list is empty, no tapdisk holds it. VM.start fails because it's walking a snapshot relationship that doesn't exist on disk. Reproduces from xe on the pool master with XO entirely out of the path — which is why I filed against xcp-ng/xcp rather than the XO tracker.

      Versions: XCP-ng 8.3.0, xapi 26.1.11 (xapi-core-26.1.11-1.2), sm-3.2.12-17.9, sm-fairlock-3.2.12-17.9, blktap 3.55.5-9.1, build 20260618.

      I have not attempted to bulk-clear the fields — on-disk data is intact and I'd rather not do a mass write against the XAPI DB on a live SR without guidance. Backing store snapshotted as a safety net.

      Happy to run whatever diagnostics would help. And +1 to the hand-grenade feeling — the affected set growing on its own is the part that worries me.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue Thanks for the response. I got side tracked with another issue and didn't get that bug report filed, I'll do that as soon as possible.

      It feels like I'm playing with a hand grenade...lol, quite scary!

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @olivierlambert Thanks for the response, much appreciated.

      I'm getting ready to file a bug report, as I noticed this morning that this issue is now causing a VDI-IN-USE error; preventing me from starting a VM. Fortunately, that VM isn't critical, so I want to report it and help with the troubleshooting that will lead to a fix before it spreads.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @olivierlambert Any chance you can have someone please take a look at this thread? The issue persists and is creating problems for me. If someone out there has figured out the solution, kindly share, thanks.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      Another confirmed data point, with package delta and the specific malformed field.

      Host: XCP-ng 8.3.0, xapi 26.1 (build 26.1.4), Xen 4.17.6-9.

      Setup: XO from sources (community). All VDIs vanished from the per-VM Disks tab (XO 5 and XO 6); xe and the SR Disks tab show them fine; VMs run normally. Trigger was the 8.3 host update + reboot this morning — XO build unchanged since May 28, disks visible yesterday.

      Host update delta (today): all 26.1.3-1.10 → 26.1.4-3.1 (xapi-core, xenopsd, sm-cli, sm-fairlock, xapi-storage-script, vhd-tool, message-switch, etc.), plus sm 3.2.12-17.8 → 17.9 as an independent bump.

      The malformed field. An affected live OS disk (VM running):

      is-a-snapshot: false
      snapshot-of:   <populated, points to another VDI>
      snapshot-time: <populated>
      

      A normal base VDI should have an empty snapshot-of. After the update, snapshot-of/snapshot-time are populated on real, non-snapshot base VDIs, and XO filters anything with a non-empty snapshot-of out of the per-VM Disks view — which is the disappearance.

      The VDI that snapshot-of points to is a legitimate base image in my environment (a heavily-reused Win2022 build template with a large genuine snapshot/clone lineage), so I can't tell from the host side whether the parentage links themselves changed or only the snapshot-of on live VDI labeling did. Either way, the consumer-visible effect is the same.

      REST confirms: /rest/v0/vms/<uuid>/vdis → []; /rest/v0/vdis/<uuid> → "no such VDI" for the VBD's referenced UUID, while xe vdi-list shows it.

      Caution for others: since live disks now carry snapshot-like metadata, be careful with Health-dashboard "orphan" cleanup and snapshot deletion on affected VMs until this is understood.

      Workaround that restored the per-VM Disks view: snapshot → revert → delete-snapshot (tested on a powered-off VM, immediate).

      Happy to provide more diagnostics.


      Quick Follow-up:

      Additional symptom, same root cause: ISO-SR VDIs are also affected. Pre-existing ISOs disappeared from the XO ISO picker (only ISOs uploaded after the patch still show). An affected ISO's vdi-param-list shows:

      is-a-snapshot: false
      snapshot-of:   937c3945-...   (same anchor UUID as an affected VM disk on a different SR)
      snapshot-time: 19700101T00:00:00Z   (Unix epoch — clearly synthetic)
      

      Notably the spurious snapshot-of on both an ISO VDI and an unrelated VM OS disk points to the same anchor UUID, with an epoch timestamp — so this looks like the update is stamping pre-existing VDIs with a bogus snapshot-of rather than any real lineage. VHD chains/GC are clean (GC reports no work).

      Tagging a related GitHub Issue for easy correlation - https://github.com/vatesfr/xen-orchestra/issues/9578

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: RPU Failure on 8.3: Yum HTTPS 500 Error for xo-lite package (Hung Task)

      @julienXOvates Got it.

      I didn't get a chance to work on this over the weekend, so sometime this week, I'll just bounce the host and see if it comes back. Right now, while I can SSH into it, the admin console appears to be frozen and unresponsive (over iLO).

      posted in Management
      K
      kagbasi-wgsdac
    • RPU Failure on 8.3: Yum HTTPS 500 Error for xo-lite package (Hung Task)

      Hello Folks,

      I encountered a failure while attempting a Rolling Pool Update (RPU) on my two-node pool on Wednesday and wanted to report the logs to see if this is a known mirror issue or an orchestration edge case.

      Environment:

      Hypervisor: XCP-ng 8.3 (Two-node pool)
      Management: Xen Orchestra (Community Edition / From Sources | Commit 5af3268 )
      Update Method: Rolling Pool Update (RPU) feature via XO

      Issue Description:
      I initiated the RPU task via XOCE. The process failed during the host update phase. The RPU task in the XO UI appears hung (running since 2026-04-29 00:39), likely because the underlying yum process exited with an error that wasn't gracefully handled by the RPU state monitoring mechanism.

      The Error:
      The logs show a yum failure due to an HTTPS 500 Internal Server Error specifically when trying to fetch the xo-lite package from the mirrors. I can confirm there were no network outages at the time (since I was working remotely and my VPN connection to the site network dropped).

      Relevant Log Snippet:

      stderr: 'http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/Packages/xo-lite-0.20.0-1.xcpng8.3.noarch.rpm: [Errno 14] HTTPS Error 500 - Internal Server Error
      Trying other mirror.
      
      Error downloading packages:
        xo-lite-0.20.0-1.xcpng8.3.noarch: [Errno 256] No more mirrors to try.'
      

      Full JSON Log:

      forum won't allow the full log, so I put it on pastebin.  Here's the link:  https://pastebin.com/raw/ueL86MiL
      

      Notes:

      It seems like a transient mirror issue, but I wanted to report it as it left the RPU task in a hung state in the UI for over 24 hours. I intend to clear the task and attempt a manual yum update on the hosts as a workaround, however, I'm going to hold off until the end of day today. Perhaps this will provide a live production environment, should the XCP-ng team wish to leverage it, for testing a fix.

      Has anyone else seen 500 errors on the xo-lite package recently, or is there a specific mirror I should check?

      As it stands now - 2026-05-01 04:30 - the RPU task is still hung (as shown in screenshot #1) and the first host (VMH01 - the pool master) is still in a disabled state (screenshot #2).

      Thanks for all the work on 8.3!

      SCREENSHOT #1 - Showing hung RPU task
      chrome_cA6EnyuQ4x.png

      SCREENSHOT #2 - Showing master host in disabled state
      chrome_0xxoYK4fzU.png

      posted in Management
      K
      kagbasi-wgsdac
    • RE: XenClean | Cleanup Failed on Windows Server 2022 VM

      @dinhngtu Thanks for the fast response, as always.

      posted in Management
      K
      kagbasi-wgsdac