@MajorP93 said: What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer. Yes, that's it - sorry - I should have been more explicit, even if 'lumbers along' wasn't a hint //EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool. Pedantic, but true I technically meant 'not hosts in the source pool' I guess.

@TrapoSAMA said: @john.c I have this iso but how to know what is de package? Thx It’s going to be an msi installer package or on the guest tools iso, which is on the XCP-ng or Xen Server your running likely in Local Storage SR or Shared Storage SR (if moved). Named the same as the file linked to above, likely. Anyway did you know that the UEFI SecureBoot certificates (at least one of them from 2011) issued by Microsoft are expiring fully during October 2026. Around that time likely at sometime after then those certificates will be placed into dbx. This means the Windows Server 2012 R2 if operating as a UEFI SecureBoot VM, will if restarted (or reboot following a crash), fail to boot. So your client will go from having a working app that they can use, but not upgrade at the moment to one which won’t work and can’t be used! They really need to upgrade or migrate from Windows to Linux (released during 2025 or 2026) along with the app if needed. Alternatively upgrade to Windows Server 2022 or 2025 and the app!

Hello! So I created a new job and a small test vm to gather info. I did 3 replications runs and the last 2 failed. Backups are working thoug. Since this would be a large post, I created a pdf with the test cycle. ... Only to find out that those are not allowed on this forum! So I captured the pdf pages... [image: 1784297275075-535e6b06-678a-4228-b109-e6fad5ec97e5-image.jpeg] [image: 1784297294513-9c5c5933-3dc8-4223-983e-7a9c3d884a72-image.jpeg] [image: 1784297307874-e199e8fa-4335-412a-924b-67f482b8bdd2-image.jpeg] [image: 1784297320970-7e8a9103-1a5f-4b7c-b886-2c9364cdd3af-image.jpeg] [image: 1784297331437-18fedc6b-3f83-4165-a6e3-f1586b97c127-image.jpeg] [image: 1784297343300-7c917ab7-31f7-441a-bca8-397fd8653a29-image.jpeg] [image: 1784297353287-921b2af6-5276-4272-818b-6f7a025d6c59-image.jpeg] And the logs from the jobs: { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml" }, "id": "1784293017099", "jobId": "10d0331c-694e-4681-8860-33fef3790552", "jobName": "TESTCRANDBK", "message": "backup", "scheduleId": "bc7b29ce-25b2-4c66-9186-b1d4c314b979", "start": 1784293017099, "status": "success", "tasks": [ { "id": "0mroxztk5-9yuipq34src", "start": 1784293020005, "status": "success", "tasks": [ { "id": "0mroxztkj-izqz7v0ufn", "start": 1784293020019, "status": "success", "end": 1784293020027, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mroxztz1-v8ycdmtgo9k", "start": 1784293020541, "status": "success", "end": 1784293036537, "result": "20683621-7819-7661-91a2-93c377a7345a", "message": "snapshot" }, { "id": "0mroy06bf-9frdqsgnpym", "start": 1784293036539, "status": "success", "tasks": [ { "id": "0mroy0aun-bscsuoyzrpu", "start": 1784293042415, "status": "success", "tasks": [ { "id": "0mroy3r14-2sk7dgyqrbl", "start": 1784293203352, "status": "success", "end": 1784293207572, "message": "target snapshot" } ], "end": 1784293207573, "result": { "size": 9254731776 }, "message": "transfer" } ], "end": 1784293207664, "message": "export", "data": { "id": "a729e916-4cd8-2208-7dce-329e6b4af892", "isFull": true, "name_label": "xxxxx", "type": "SR" } }, { "id": "0mroy06bg-eyd2s4tidgg", "start": 1784293036540, "status": "success", "end": 1784293207664, "message": "export", "data": { "id": "b0b77301-4625-a13b-7022-9512f927944a", "isFull": true, "name_label": "xxxxxx", "type": "SR" } }, { "id": "0mroy06bh-ra3dxpusxr8", "start": 1784293036541, "status": "success", "end": 1784293207664, "message": "export", "data": { "id": "7054c407-d620-2c9f-5810-1e192a04ffe2", "isFull": true, "name_label": "xxxxxxxxx", "type": "SR" } }, { "id": "0mroy06bi-tdyzypsg58", "start": 1784293036542, "status": "success", "end": 1784293207664, "message": "export", "data": { "id": "621c9b87-4f58-987b-2c07-e8dee67da73a", "isFull": true, "name_label": "xxxxxxxxxx", "type": "SR" } }, { "id": "0mroy06be-u3p9nky3uwd", "start": 1784293036538, "status": "success", "tasks": [ { "id": "0mroy0aus-ax44kkpim1", "start": 1784293042420, "status": "success", "end": 1784293200916, "result": { "size": 9257013760 }, "message": "transfer" }, { "id": "0mroy434q-md07n6vujw", "start": 1784293219034, "status": "success", "end": 1784293219099, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1784293219102, "message": "export", "data": { "id": "c1fdab1d-5535-42e4-b282-1a0282df1081", "isFull": true, "type": "remote" } } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:080d2018-c2a3-aa6f-89a1-b3eaf72c3de5" }, "message": "Snapshot data has been deleted" } ], "end": 1784293219102, "message": "backup VM", "data": { "id": "d771731a-f132-7365-be50-71f19424f96e", "type": "VM", "name_label": "TESTREPL", "progress": 0 } } ], "end": 1784293219104, "infos": [ { "data": { "vms": [ "d771731a-f132-7365-be50-71f19424f96e" ] }, "message": "vms" } ] } { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml" }, "id": "1784293990343", "jobId": "10d0331c-694e-4681-8860-33fef3790552", "jobName": "TESTCRANDBK", "message": "backup", "scheduleId": "bc7b29ce-25b2-4c66-9186-b1d4c314b979", "start": 1784293990343, "status": "success", "tasks": [ { "id": "0mroykonr-nktgebhtb4k", "start": 1784293993431, "status": "success", "tasks": [ { "id": "0mroykoo5-171n48gwq3r", "start": 1784293993445, "status": "success", "end": 1784293993493, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mroykp31-4r5wrf9blru", "start": 1784293993981, "status": "success", "end": 1784294007573, "result": "9ff54ec5-0247-4692-945b-c183206ae6dd", "message": "snapshot" }, { "id": "0mroykzkm-ykzk53um8l", "start": 1784294007574, "status": "success", "tasks": [ { "id": "0mroyl3qn-vzompxsw79o", "start": 1784294012975, "status": "success", "end": 1784294018591, "result": { "size": 396481024 }, "message": "transfer" }, { "id": "0mroyljgo-nlch68rkbil", "start": 1784294033352, "status": "success", "end": 1784294033444, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1784294033446, "message": "export", "data": { "id": "c1fdab1d-5535-42e4-b282-1a0282df1081", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "the writer AggregatedIncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'get'). It won't be used anymore in this job execution." } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:77605638-5e58-4a10-c2de-bc6e77f1ad61" }, "message": "Snapshot data has been deleted" } ], "end": 1784294033447, "message": "backup VM", "data": { "id": "d771731a-f132-7365-be50-71f19424f96e", "type": "VM", "name_label": "TESTREPL", "progress": 0 } } ], "end": 1784294033449, "infos": [ { "data": { "vms": [ "d771731a-f132-7365-be50-71f19424f96e" ] }, "message": "vms" } ] } { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml" }, "id": "1784295416670", "jobId": "10d0331c-694e-4681-8860-33fef3790552", "jobName": "TESTCRANDBK", "message": "backup", "scheduleId": "bc7b29ce-25b2-4c66-9186-b1d4c314b979", "start": 1784295416670, "status": "success", "tasks": [ { "id": "0mrozf8yr-lix8jxbm6pd", "start": 1784295419428, "status": "success", "tasks": [ { "id": "0mrozf8zg-zddbxm901mc", "start": 1784295419452, "status": "success", "end": 1784295419537, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mrozf9dv-7nj082i92kp", "start": 1784295419971, "status": "success", "end": 1784295426890, "result": "18312e82-adb6-2dc5-b13e-2fd9a4b2bf77", "message": "snapshot" }, { "id": "0mrozfeq3-w3y5xrwtqse", "start": 1784295426891, "status": "success", "tasks": [ { "id": "0mrozfrvy-4g1gefr14hx", "start": 1784295443950, "status": "success", "end": 1784295445177, "result": { "size": 92319744 }, "message": "transfer" }, { "id": "0mrozfw5f-mx0xt5f7wjm", "start": 1784295449475, "status": "success", "end": 1784295449563, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1784295449565, "message": "export", "data": { "id": "c1fdab1d-5535-42e4-b282-1a0282df1081", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "the writer AggregatedIncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'get'). It won't be used anymore in this job execution." } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:6a833d28-2d2c-5fe6-858b-02d2fd41b919" }, "message": "Snapshot data has been deleted" } ], "end": 1784295449566, "message": "backup VM", "data": { "id": "d771731a-f132-7365-be50-71f19424f96e", "type": "VM", "name_label": "TESTREPL", "progress": 0 } } ], "end": 1784295449567, "infos": [ { "data": { "vms": [ "d771731a-f132-7365-be50-71f19424f96e" ] }, "message": "vms" } ] } Hope it helps

@pierrebrunet said: Hi, just to be sure, are you saying it is working ok apart one task 8 days ago? Yes, after I adjusted the time it's working

@acebmxer I can confirm that, we'll fix it, thanks

@pierrebrunet Hello! I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens? I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ? REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -

Perfect, thanks for your feedback!

@tmk I was able to reproduce the problem, so it seems the modifications we brought a couple of weeks ago didn't fix it. I'm working on a fix at the moment.

@kagbasi-wgsdac said: @poddingue Bug report filed as requested — https://github.com/xcp-ng/xcp/issues/825 — and tagging @Team-Storage per your suggestion. Full evidence bundle is attached to the issue (versions, sweep output, vhd-util vs xe comparison, SMlog). Summary of what I found: One correction to the mechanism, and I think it matters. The recap describes is-a-snapshot being flipped to true. On my system that isn't what's happening — is-a-snapshot is false on every affected VDI. The field being wrongly written is snapshot-of, which is getting populated on base disks that aren't snapshots at all. XO's disappearing-disks symptom is consistent with either (it filters on a non-empty snapshot-of), but if the storage team is hunting for a bad is-a-snapshot write, that may be the wrong field. Every affected VDI here looks like: is-a-snapshot: false <-- correct snapshot-of: <populated with an unrelated VDI's UUID> <-- wrong A VDI that is a snapshot of itself. The clearest single artifact: uuid: 806f7f42-083f-4a40-b3f1-0700d00bab5a name-label: WinSrv2022SHB_Disk1_Data is-a-snapshot: false snapshot-of: 806f7f42-083f-4a40-b3f1-0700d00bab5a <-- itself snapshot-time: 20260709T11:19:15Z sm-config: vhd-parent: c86e3247-... <-- bears no relation to the snapshot-of value No valid code path produces snapshot-of = self. Whatever writes this field isn't validating the target. It's still actively corrupting new VDIs — this is not just legacy damage. That self-referential VDI was created 2026-07-09, a week after my patch + reboot. Sweeps 9 days apart went from ~180 → 191 affected VDIs on one SR, and a fourth anchor UUID appeared that didn't exist in the first sweep. Newly created VHDs keep landing in the affected set. So "stop it happening again" is the urgent half of the two-part fix, at least in my case. The bogus targets cluster onto a tiny anchor set, and the anchors point at each other: Count Anchor 97 937c3945 (→ a893fdb4) 50 a893fdb4 (→ ea150883) 37 ea150883 7 806f7f42 (→ itself, new since Jul 9) That looks less like corrupted lineage and more like the field being filled from an incorrect/uninitialised source. On-disk VHDs are completely healthy. vhd-util check says valid, parent locators are consistent, GC reports no work. The two VDIs the DB calls parent/child are, on disk, siblings under a common parent. The corruption is purely in the XAPI database — which is good news for recoverability. The VDI_IN_USE is not a real lock. current-operations is empty, xe task-list is empty, no tapdisk holds it. VM.start fails because it's walking a snapshot relationship that doesn't exist on disk. Reproduces from xe on the pool master with XO entirely out of the path — which is why I filed against xcp-ng/xcp rather than the XO tracker. Versions: XCP-ng 8.3.0, xapi 26.1.11 (xapi-core-26.1.11-1.2), sm-3.2.12-17.9, sm-fairlock-3.2.12-17.9, blktap 3.55.5-9.1, build 20260618. I have not attempted to bulk-clear the fields — on-disk data is intact and I'd rather not do a mass write against the XAPI DB on a live SR without guidance. Backing store snapshotted as a safety net. Happy to run whatever diagnostics would help. And +1 to the hand-grenade feeling — the affected set growing on its own is the part that worries me. Has this issue been validated on a storage server built around Debian 13, LVM and ext4 or just TrueNAS when connected to XCP-ng version 8.3.0. As part of the XAPI DB corruption. Can anyone answer this please or give a clue?

@simonp , thank you very much and all of your team ! ! ! Keep up the outstanding work. Regards, Marc

Just FYI, we had some issues with some Debian VMs for instance, with old kernel where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive. It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore. Newer kernel don't have this issue and are able to wake up. I'm quoting Debian here but it might very well happen on other distro

Hi @dvdwx, RBAC is available only at the REST API level, not through the XO6 UI. UI compatibility with RBAC is currently under development. For the topology, this is something that has been here for a long time, so I may be wrong since I don't have the full history. However, if I look at the code, creating the topology selector requires knowing how many CPUs are available in the pool. Since a self-service user doesn't have visibility into the pool, they can't access this kind of information.

Also a question for @Team-XAPI-Network

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes. Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 8,700 log lines around the panic suggests dom0 was under real pressure right then. If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side. I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful. Fingers crossed it stays boring from here.

@christopher-petzel Thanks!

I converted the topic to a question, then marked it solved. Thanks!

Ping @julienXOVates

Hi @acebmxer, thanks for reporting, we identified this bug internally already, we're going to fix it once root cause is found.

Hi Everyone, If I may advertise what I did you have a dedicated thread about my project, this is an easy to setup XCP-ng and XO from the source solution. Sorry sharing the link to the topic as is couldn't find better way to do that : XCP-ng HL — an ISO for home labs, deploy any XOA image from XO Lite directly Essentially you deploy XCP-ng ISO and then in XO Lite deploy XOA section you can deploy your prebuilt or XOA VM directly. Hope that helps Home labers to get in to XCP-ng easily. Same here this is still work in progress, also I'm personally already using this version in my lab. Any feedback is welcome ! With regards.