Intermittent Xen blkfront I/O stalls: all guest tags busy while tapdisk reports zero outstanding requests

olivierlambert

anthoineb

Hello @kazual82,
Lot of question here, I'll try to answer my best:

Is there a known XCP-ng 8.3, blktap, or Xen issue matching this frontend/backend state mismatch?
No, I never saw this kind of issue but I appreciate all your investigation.
Can tapdisk or the Xen backend lose a completion while both sides remain in state Connected?
It'd surprise me because my opinion is tapdisk is sending to much completion notifications. My guess is a race condition somewhere that loose the notification.
Is xen_blkfront.max_ring_page_order=3 considered safe and recommended with this backend?
Yes, it's safe but there is no recommendation on that because any max_ring_page_order should work.
Could increasing the ring size reduce the probability of this failure, or would it only mask it temporarily?
I guess it reduce the probability.
Is the fact that persistent grants and indirect descriptors are not negotiated relevant?
No, it's expected, tapdisk doesn't support those features.
Are there additional tapdisk, xenstore, event-channel, or ring diagnostics that we should enable before the next incident?
I don't see what we can do. If it appends again I'm curious to investigate. Install blktap-debuginfo and devtoolset-11-gdb to attach the debugger on tapdisk and see the state of the blkif ring, that could be an info if the requests are still in the ring. If you can open a support tunnel, I will investigate.
Are there candidate packages or patches that would be useful to test on one canary host?
The next update, yet to be release, will add a bunch of fixes for QCOW2 support in tapdisk and sm. I recommend this update.
Could the use of a multi-terabyte QCOW2 VDI on an LVM SR be relevant to this failure mode?
I fear this is a QCOW2 issue only linked to the recent release of the feature but no real clue here.

The fact that tap-ctl stats answer is the proof that there is no deadlock on the tapdisk side, that's a small reason of satisfaction.

kazual82

@anthoineb Thank you. We have now installed the required packages on all three hypervisors hosting the OpenSearch data VMs:

blktap-3.55.5-6.7.xcpng8.3.x86_64
blktap-debuginfo-3.55.5-6.7.xcpng8.3.x86_64
devtoolset-11-gdb

The debuginfo package was not exposed by our internal yum mirror metadata, but we found the exact matching RPM in the official XCP-ng 8.3 stable-updates repository.

No services or hypervisors were restarted during the installation.

Could you please provide the recommended gdb commands for attaching to the affected tapdisk process and inspecting the blkif ring? In particular, it would be helpful to know which structures, fields and threads should be captured.

We plan to test the attach procedure manually on a healthy tapdisk first. After that, we can add a short pre-reboot diagnostic pause to our recovery controller so that the GDB data can be collected during the next occurrence before the VM is rebooted.

kazual82

@anthoineb We captured another occurrence, this time on os-ott-data-3-2, running on hypervisor 172.30.50.191.

The guest-side state was:

31 requests in flight
all 32 blkfront tags busy
I/O PSI full around 97%
6 tasks in D state
no completed xvdb I/O progress

The VM was still running and its local OpenSearch HTTP endpoint was responsive, but the node had already disappeared from OpenSearch cluster membership.

On the hypervisor:

tapdisk was responsive to tap-ctl
reqs_outstanding: 0
tap request counters: 0/0
all xenbus request counters were balanced
no tapdisk errors were reported

We attached GDB to the affected tapdisk before reboot. The main thread was sleeping in scheduler_wait_for_events(), and the captured blktap/shared-ring state was:

req_prod = 0
req_cons = 0
rsp_prod = 0
rsp_prod_pvt = 0
n_reqs = 32
n_reqs_free = 32

The complete shared ring was empty. Therefore, at capture time, there were no requests waiting in the tapdisk ring, while the guest still believed that 31 requests were in flight and all frontend tags were occupied.

GDB detached normally. After the diagnostic window, the VM was rebooted and successfully rejoined the cluster.

At the time of the failure, this VM was still running with xen_blkfront.max_ring_page_order=0 and 32 tags. After reboot, the persistent setting became active:

xen_blkfront.max_ring_page_order=3
nr_tags=256

Could this evidence indicate that requests or completion notifications are being lost between blkfront and the backend before they become visible in the tapdisk shared ring?

We can provide the complete GDB backtraces and ring dump if useful.

poddingue

Your GDB capture looks to me like it lands where @anthoineb already pointed in post #2, when he guessed at a race condition losing the notification somewhere.
An empty ring on the tapdisk side while the guest still believes 31 tags are in flight seems to fit that, though I might be reading more into it than is there.
He mentioned too that the next update brings a batch of QCOW2 fixes in tapdisk and sm, which he recommended you take.

anthoineb

Hello @kazual82 Sorry for the delay, I was working on a gdb script for you before I was preempt. Yes, I'm curious about GDB backtraces and ring dump. I'm puzzled because even if we miss a notification from the frontend, tapdisk periodically checks blkif ring, so he must see the request with a delay at least. Can you also share your daemon.log and SMlog?

mike.potapov

Hello @anthoineb,

Thank you. I have attached the requested evidence as separate text files because the forum does not accept ZIP archives:

two complete GDB/tap-ctl captures taken during the same incident, before reboot;
the corresponding daemon.log excerpt from the hypervisor;
the corresponding SMlog excerpt from the hypervisor;
a README containing the timeline and identifiers;
SHA-256 checksums.

Both host-log excerpts cover 2026-07-25 04:35:00-05:10:00 MSK (UTC+03:00).
The first GDB capture started at 04:45:44, the independent repeat capture
started at 04:48:03, and the forced VM reboot was requested at 04:58:33.

Both captures show the same state:

the tapdisk main thread was in scheduler_wait_for_events();
n_reqs=32 and n_reqs_free=32;
req_prod=req_cons=rsp_prod=rsp_prod_pvt=0;
tap-ctl reported reqs_outstanding=0;
the guest still reported 31 requests in flight and all 32 blkfront tags busy.

The host logs contain no tapdisk error before the reboot. At 04:58:34-04:58:36
they show the expected sring disconnect, tapdisk close/detach and clean shutdown
after the forced reboot request. The new QCOW2 tapdisk was opened at 04:59:57
and its sring connected at 05:00:27.

Please let me know if you need another structure printed from GDB or a wider
host-log interval. Our recovery controller can preserve the same pre-reboot
diagnostic window during the next occurrence.
SHA256SUMS.txt README.txt osott-193-SMlog-20260725-0435-0510.txt osott-193-daemon-20260725-0435-0510.txt 20260725-044732-gdb-172.30.52.193.txt 20260725-044544-gdb-172.30.52.193.txt

mike.potapov

Hello @anthoineb,

We captured another occurrence on the same VM, os-ott-data-3-2
(172.30.52.193), on the same hypervisor and QCOW2 VDI as the incident
reported on 2026-07-25.

This occurrence is useful because xen_blkfront.max_ring_page_order=3 was
definitely active:

xenstore showed ring-page-order=3 with eight ring references;
the guest queue depth was 256;
255 write requests remained in flight;
all 256 blk-mq scheduler tags were busy;
wbt_lat_usec=0;
I/O PSI full was approximately 98%;
completed xvdb counters did not advance between samples.

The backend state was nevertheless the same as in the previous capture:

tapdisk was sleeping in scheduler_wait_for_events();
tap-ctl reported reqs_outstanding=0;
tap request counters were 0/0;
req_prod=0 and rsp_prod=0;
no tap, image, VBD, map or xenbus errors were reported.

One detail may be relevant: GDB still printed n_reqs=32 and
n_reqs_free=32, although the frontend had negotiated an order-3 ring and the
guest exposed 256 tags. Is this expected because n_reqs represents a separate
fixed tapdisk request pool, or could it indicate that tapdisk is not observing
the complete multi-page ring?

The controller captured GDB and the guest/hypervisor state before reboot. After
the ten-minute diagnostic window, the same signature was confirmed three more
times and the VM was rebooted. It rejoined the cluster successfully.

I have attached:

the complete GDB/tap-ctl capture;
the guest blk-mq, diskstats, PSI, D-state stack and kernel-journal snapshot;
the corresponding tapdisk/xenstore, daemon.log and SMlog excerpt;
a README and SHA-256 checksums.

The important new result appears to be that increasing
max_ring_page_order from 0 to 3 did not prevent the stall; the same mismatch
recurred with 255/256 frontend requests/tags occupied while the backend ring
and tapdisk remained empty.
20260730-005220-gdb-172.30.52.193.txt 20260730-pre-reboot-guest-172.30.52.193.txt 20260730-pre-reboot-hypervisor-172.30.50.191.txt README.txt SHA256SUMS.txt

anthoineb

Hello @mike.potapov , on n_reqs=32 and n_reqs_free=32, I think you're watching td_blktap_t structure. This structure is only used if the pvdriver are not install in the VM, so it's normal this ring is not used. You should watch struct td_xenblkif that is the ring use normally and that must contains 256 requests on max_order=3.

The tapdisk will wakeup periodically, you could put a breakpoint on tapdisk_xenio_ctx_ring_event to confirm that the ring is check periodically. You can investigate td_xenblkif here.

Ultimatly, you can try to send a notification from the tapdisk under gdb to the linux kernel with call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port); to see if it unlock something on the guest side. It it's the case it will confirm a notification was lost. If not you will get a message about spurious interrupt in the dmesg of the guest kernel.

mike.potapov

Hello @anthoineb,

We captured another occurrence and repeated the diagnostics you suggested. This time we inspected the active struct td_xenblkif, not the legacy td_blktap_t.

The incident affected os-ott-data-2-4 (172.30.52.185) on hypervisor 172.30.50.181. The guest had xen_blkfront.max_ring_page_order=3 active.

Guest state:

no completed xvdb I/O progress;
249 requests in flight;
250 of 256 blk-mq tags busy;
I/O PSI full approximately 97%;
11 tasks in D state;
OpenSearch remained locally responsive for part of the incident, but the node had left cluster membership.

Tapdisk remained responsive and reported:

reqs_outstanding=0
xenbus reqs=[8105673748,8105673748]
tap/image/VBD/xenbus errors=0

The active td_xenblkif state was:

domid=20
devid=832
port=325
ring_n_pages=8
ring_size=256
n_reqs_free=256

req_prod=3810706702
req_cons=3810706452
rsp_prod=3810706452
rsp_prod_pvt=3810706452

Therefore, req_prod - req_cons = 250: the 250 pending requests were present in the active Xen PV ring, but tapdisk had not consumed them and still reported zero outstanding requests.

Additional state:

in_polling=false
chkrng_event=57
stoppolling_event=58
poll_duration=8000
poll_idle_threshold=50

We placed a breakpoint on tapdisk_xenio_ctx_ring_event for eight seconds. It did not fire. GDB then detached and tapdisk remained responsive.

After preserving the original ring and guest state, we executed the suggested notification:

call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port)

The call returned 0, but it did not restore I/O. Three subsequent guest samples remained at 249 inflight / 250 busy tags with no disk progress. The td_xenblkif ring indices were unchanged in a second GDB capture, and no new spurious-interrupt message appeared in the guest dmesg.

A forced VM reboot restored I/O and the node rejoined the cluster.

This appears to show that requests are present in the active PV ring but are not being consumed by tapdisk, and a backend-to-guest event-channel notification alone does not unblock the condition.

Could you advise which callback or scheduler state we should inspect next? In particular, should we also place breakpoints on tapdisk_xenblkif_cb_chkrng or related polling callbacks during the next occurrence?

We have the complete before/after td_xenblkif dumps, breakpoint log, notify log, guest snapshots and hypervisor data available for upload.