our solution to the problem we want to share a post-incident analysis of a data-loss event on XCP-ng 8.3 LTS (shared block storage over FC, Veeam B&R 13.1 with CBT enabled on the pool) — consolidated from our own incident, topic 12402, the CBT feedback thread 9268, and the still-open blktap PR #17. The pieces fit together into one coherent failure chain, and we believe it may be worth a sticky/KB article. A note on the trigger, to be fair and complete: in our case the interrupted jobs (step 2) were caused by expired/invalid worker tokens on the XCP-ng hosts, so the backups died mid-run with the locks in place. We consider that our availability problem. However, an interrupted backup must never be able to damage production VM data — cleaning up VDI locks on job abort is the hypervisor's job, and that is the part that turned a backup hiccup into guest data loss. Failure chain (as we understand it) Veeam backup job with CBT runs and sets locks on VDIs — paused: true + host_OpaqueRef:xxx: RW entries in the VDI sm_config (XAPI state.db). The job is interrupted (timeout / crash / restart). The locks are never cleaned up → stale paused: true remains in sm_config. These entries are MRO, so xe vdi-param-remove can't clear them. A later leaf coalesce / commit runs into a cbtlog disk: per PR #17, tapdisk_vbd_first_image returns the cbtlog disk on td_commit, and the cbtlog driver has no commit action → commit fails early. Result: broken VDI chains, CBT metadata VDIs without a vhd parent, hundreds of orphaned VDIs, and .cbtlog files hanging coalesces (as reported in thread 9268). SR rescan believes a GC is already running and aborts; a host reboot was the only way to force the coalesce through (also reported in 9268). Storage cleaning freezes the VM disk briefly, but the un-freeze fails on the stale lock → failed to unpause tapdisk ... VMs using this tapdisk have lost access to the corresponding disk(s). The guest keeps writing on a frozen/lost disk → NTFS corruption inside the guest and, in our case, actual SQL Server data loss. Step 5 matches exactly the theory Veeam R&D is currently investigating ("storage cleaning freezes VM disks briefly during backup and sometimes fails to un-freeze them"). The stale paused:true lock appears to be the missing "why" behind the failed unpause. What helped us recover Patching the stale lock out of XAPI state.db (stop xapi, backup state.db, remove paused + host_OpaqueRef entries from the affected VDI's sm_config, start xapi). Then: reset CBT on the affected VDIs and trigger a full backup so CBT re-initializes cleanly — otherwise the next interrupted job re-creates the same situation. For the coalesce backlog: with the affected VMs powered off and CBT disabled, snapshot-create-then-delete to kick the GC, watch SMlog, iterate. (Same recipe a user documented in 9268.)

Rocky 10 is affected and has no fix in it. I pulled the source RPM for kernel-6.12.0-211.16.1.el10_2.0.1, which is your -211: jiffies.c still ends on core_initcall(init_jiffies_clocksource) with no cs_jiffies_registered, so f24df84cbe05 hasn't landed, and max_raw_delta sits in clocksource.h with nothing setting it early, so the regression is still there. So tsc_mode=2 and nomigrate stay the answer on Rocky until Red Hat picks it up. One more thing, because this thread reads like an AMD problem if you skim it. @dvinni measured the same stalled clock on Intel Xeon Gold in the sibling thread, and the upstream fix came from @teddyastie bisecting it on a Xen HVM guest. I read source rather than booting a Rocky VM, so if you have one behaving differently I'd like to hear it.

Question for @teddyastie but I'm not 100% sure about Intel Coffee Lake iGPU passthrough (iGPU is always far more difficult to passthrough than a discrete GPU)

@poddingue Thanks for running the -31 numbers — good to have it confirmed that the ttyS0 removal stays worth ~3-4s even with the clock fixed. Agreed on not rushing -proposed to production; we'll pick up -31 when it promotes and keep the cloud-init tweak permanently.

Hey that sucks after catching up 23 commits. Looks like something broke in the log module during the update. I’d try a clean reinstall of the dependencies first, or drop back to an older Node version for a bit – Node 24 can be fussy with these packages.

Hi bikemuch, For unregistered XOA the template auto-update is limited. Easiest way is via CLI: download the latest Debian 13 and CentOS Stream 10 ISOs, then use xe vm-import or create new templates from them. You can also check the XOA “Templates” section and force a refresh if available. Works fine for me this way.

@seanmcg182 's post might be the answer to the original question, two years late. Your lspci -s 0000:05:00.0 -v at post 8 shows (rev ff) and !!! Unknown header type 7f, which is the same pair he had, and in his case it came from bifurcating the wrong PCIe port in the BIOS rather than from anything XCP-ng was doing. He says the unknown header is what then produced the device or resource busy error on VM start, which is the error you opened with. I don't know whether your board exposes bifurcation the way his X10DRH-CT does, so it might not transfer at all. @kuznetcoff777 if that machine is still around, it'd be worth a look either way.

@poddingue said: What I can't tell you is what set that particular combination on your VM in the first place. Does it ring a bell? I have no Idea. I had it on "Protect from accidental shutdown" but turned that off again, later. Doing this again (on, off) helped, as you said. Thank you so much!

@poddingue Thank you for the analysis. I'd give you a rep if I could

If the list is only complete after you've been through the Storage view first, that smells like the VM view not loading the whole SR collection, so the dialog only knows about the SR the disk already sits on. I could easily be wrong about the mechanism though. 6.7.0 went out on 30 July, so it would help to know whether it still does this there, and whether anyone else on 6.6.2 sees the same thing. Might be worth a mention to @Team-XO-Frontend, since they'd know straight away whether that dropdown is meant to come from a shared collection or get fetched per view. The nearest thing I found in the tracker is https://github.com/vatesfr/xen-orchestra/issues/7392, but that one is ISO SRs turning up in the wrong list rather than SRs going missing, so probably not the same thing.

Hello @mike.potapov , on n_reqs=32 and n_reqs_free=32, I think you're watching td_blktap_t structure. This structure is only used if the pvdriver are not install in the VM, so it's normal this ring is not used. You should watch struct td_xenblkif that is the ring use normally and that must contains 256 requests on max_order=3. The tapdisk will wakeup periodically, you could put a breakpoint on tapdisk_xenio_ctx_ring_event to confirm that the ring is check periodically. You can investigate td_xenblkif here. Ultimatly, you can try to send a notification from the tapdisk under gdb to the linux kernel with call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port); to see if it unlock something on the guest side. It it's the case it will confirm a notification was lost. If not you will get a message about spurious interrupt in the dmesg of the guest kernel.

Your xe host-list output kills my guess. All three addresses are real LAN IPs with nothing on loopback, so the loopback is probably coming from the XO side rather than from XAPI. I did test one thing: given a name that resolves to both families, Node reports connect ECONNREFUSED ::1:443 and connect ECONNREFUSED 127.0.0.1:443 together, which is what your log shows, while the literal 127.0.0.1 only ever names one address. That suggests something is handing XO a name rather than an IP, and localhost is the obvious candidate, so it's worth checking what you have registered for that pool under Settings then Servers, which is where the address comes from in the first place: https://docs.xen-orchestra.com/xo5/manage_infrastructure#add-a-host . The migration failure I'd keep separate for now, and could you paste the exact error XO gives when one fails? I don't know whether the two are the same problem and I'd rather have the text than keep making wild guesses.

@yllar said: @yannsionneau Hi, any news on the new iso? Hello @yllar In case you missed it, the ISO was released a few days ago: https://xcp-ng.org/blog/2026/08/14/xcp-ng-8-3-lts-refreshed-installation-isos/

After a good sleep, I resolved half of my issue. I got my LSI Cards messed up, I accidentally hidan internal LSI3108 (Address 01:00.0) instead of the PCI Card LSI3008 (Address 05:00.0)... My Boot Drives run off of the internal card. From the fallback kernel, I was able to Modify /etc/grub-efi.cfg to remove the internal card form the hidden list. I now have a separate issue, but will make another post.

@MajorP93 Ok. Thanks!

Any update on this? It is a drawback. It is also in the wrong format under snapshots. XO 5 has the format: July 17, 2026 or 2026-07-17 depending on the screen.