Veeam scheduled a remote call with me and pulled more log files. Of coarse when we ran the backup job twice in a row both times al vms were successful. Veeam needs to baby sit our backups :). The call was cut short do to internet going down. I have uploaded the logs and waiting to hear back.

@poddingue Thanks for running the -31 numbers — good to have it confirmed that the ttyS0 removal stays worth ~3-4s even with the clock fixed. Agreed on not rushing -proposed to production; we'll pick up -31 when it promotes and keep the cloud-init tweak permanently.

@poddingue said: What I can't tell you is what set that particular combination on your VM in the first place. Does it ring a bell? I have no Idea. I had it on "Protect from accidental shutdown" but turned that off again, later. Doing this again (on, off) helped, as you said. Thank you so much!

@poddingue Runtime confirmation for your source read: booted CentOS Stream 10 itself — GenericCloud x86_64, kernel 6.12.0-260.el10, 8 vCPUs, XCP-ng 8.3 UEFI. sched_clock correction is -84s with the image's default cmdline (it carries console=ttyS0,115200n8 too), -10s with it removed — same ~8x amplification as on Ubuntu. Backport request sent to devel@lists.centos.org: https://lists.centos.org/hyperkitty/list/devel@lists.centos.org/thread/OLHN4WZR3RYHHXDPPTJR2UDNAFFLCEJF/

@MajorP93 Ok. Thanks!

@poddingue Here is what is in the Settings -> Servers [image: 1787162145156-1233d201-1ffc-4be8-b18e-666f53e00ee9-image-resized.jpeg] It looks like the migration is working now. I'm not positive what caused it. I did make some changes after I found a discrepancy with the time between the 3 systems. It was 30-50 seconds different between them.

Hello @anthoineb, We captured another occurrence and repeated the diagnostics you suggested. This time we inspected the active struct td_xenblkif, not the legacy td_blktap_t. The incident affected os-ott-data-2-4 (172.30.52.185) on hypervisor 172.30.50.181. The guest had xen_blkfront.max_ring_page_order=3 active. Guest state: no completed xvdb I/O progress; 249 requests in flight; 250 of 256 blk-mq tags busy; I/O PSI full approximately 97%; 11 tasks in D state; OpenSearch remained locally responsive for part of the incident, but the node had left cluster membership. Tapdisk remained responsive and reported: reqs_outstanding=0 xenbus reqs=[8105673748,8105673748] tap/image/VBD/xenbus errors=0 The active td_xenblkif state was: domid=20 devid=832 port=325 ring_n_pages=8 ring_size=256 n_reqs_free=256 req_prod=3810706702 req_cons=3810706452 rsp_prod=3810706452 rsp_prod_pvt=3810706452 Therefore, req_prod - req_cons = 250: the 250 pending requests were present in the active Xen PV ring, but tapdisk had not consumed them and still reported zero outstanding requests. Additional state: in_polling=false chkrng_event=57 stoppolling_event=58 poll_duration=8000 poll_idle_threshold=50 We placed a breakpoint on tapdisk_xenio_ctx_ring_event for eight seconds. It did not fire. GDB then detached and tapdisk remained responsive. After preserving the original ring and guest state, we executed the suggested notification: call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port) The call returned 0, but it did not restore I/O. Three subsequent guest samples remained at 249 inflight / 250 busy tags with no disk progress. The td_xenblkif ring indices were unchanged in a second GDB capture, and no new spurious-interrupt message appeared in the guest dmesg. A forced VM reboot restored I/O and the node rejoined the cluster. This appears to show that requests are present in the active PV ring but are not being consumed by tapdisk, and a backend-to-guest event-channel notification alone does not unblock the condition. Could you advise which callback or scheduler state we should inspect next? In particular, should we also place breakpoints on tapdisk_xenblkif_cb_chkrng or related polling callbacks during the next occurrence? We have the complete before/after td_xenblkif dumps, breakpoint log, notify log, guest snapshots and hypervisor data available for upload.

I went and checked a couple of the factual bits here rather than take them on trust, and the xscontainer one holds up: xscontainer-10.0.4-1.xcpng8.3.noarch.rpm is still sitting in the 8.3 base repo, and back in topic 6845 Olivier said it was meant to be removed at some point, which evidently hasn't happened. The wiki page is the bit I couldn't reproduce. There's no Docker or container page in the current docs that I can find, and nothing matching in the docs repo either, so I might be looking in the wrong place. On the proposal, I'm not the right person to say whether a containers collector fits the agent's scope. Two things I can tell you. xen-guest-agent lives on GitLab rather than GitHub, so the design conversation would need to happen there. And there's no Feeder entry for any of this yet, which surprised me given how far back the requests go; worth putting one up so the votes have somewhere to land. That's my read on where it should go rather than on whether it's a good idea, and someone closer to the agent will correct me if I've sent you the wrong way.

Question for @teddyastie but I'm not 100% sure about Intel Coffee Lake iGPU passthrough (iGPU is always far more difficult to passthrough than a discrete GPU)

@seanmcg182 's post might be the answer to the original question, two years late. Your lspci -s 0000:05:00.0 -v at post 8 shows (rev ff) and !!! Unknown header type 7f, which is the same pair he had, and in his case it came from bifurcating the wrong PCIe port in the BIOS rather than from anything XCP-ng was doing. He says the unknown header is what then produced the device or resource busy error on VM start, which is the error you opened with. I don't know whether your board exposes bifurcation the way his X10DRH-CT does, so it might not transfer at all. @kuznetcoff777 if that machine is still around, it'd be worth a look either way.

After a good sleep, I resolved half of my issue. I got my LSI Cards messed up, I accidentally hidan internal LSI3108 (Address 01:00.0) instead of the PCI Card LSI3008 (Address 05:00.0)... My Boot Drives run off of the internal card. From the fallback kernel, I was able to Modify /etc/grub-efi.cfg to remove the internal card form the hidden list. I now have a separate issue, but will make another post.

Question for @Team-XO-Backend

It's more like "partially"/loosely based on Alma (10, not 9). And yes, the kernel will be far more recent, same for Xen and so on.

Hey that sucks after catching up 23 commits. Looks like something broke in the log module during the update. I’d try a clean reinstall of the dependencies first, or drop back to an older Node version for a bit – Node 24 can be fussy with these packages.

Hi bikemuch, For unregistered XOA the template auto-update is limited. Easiest way is via CLI: download the latest Debian 13 and CentOS Stream 10 ISOs, then use xe vm-import or create new templates from them. You can also check the XOA “Templates” section and force a refresh if available. Works fine for me this way.

@poddingue Thank you for the analysis. I'd give you a rep if I could

If the list is only complete after you've been through the Storage view first, that smells like the VM view not loading the whole SR collection, so the dialog only knows about the SR the disk already sits on. I could easily be wrong about the mechanism though. 6.7.0 went out on 30 July, so it would help to know whether it still does this there, and whether anyone else on 6.6.2 sees the same thing. Might be worth a mention to @Team-XO-Frontend, since they'd know straight away whether that dropdown is meant to come from a shared collection or get fetched per view. The nearest thing I found in the tracker is https://github.com/vatesfr/xen-orchestra/issues/7392, but that one is ISO SRs turning up in the wrong list rather than SRs going missing, so probably not the same thing.