Hello @anthoineb,
We captured another occurrence and repeated the diagnostics you suggested. This time we inspected the active struct td_xenblkif, not the legacy td_blktap_t.
The incident affected os-ott-data-2-4 (172.30.52.185) on hypervisor 172.30.50.181. The guest had xen_blkfront.max_ring_page_order=3 active.
Guest state:
no completed xvdb I/O progress;
249 requests in flight;
250 of 256 blk-mq tags busy;
I/O PSI full approximately 97%;
11 tasks in D state;
OpenSearch remained locally responsive for part of the incident, but the node had left cluster membership.
Tapdisk remained responsive and reported:
reqs_outstanding=0
xenbus reqs=[8105673748,8105673748]
tap/image/VBD/xenbus errors=0
The active td_xenblkif state was:
domid=20
devid=832
port=325
ring_n_pages=8
ring_size=256
n_reqs_free=256
req_prod=3810706702
req_cons=3810706452
rsp_prod=3810706452
rsp_prod_pvt=3810706452
Therefore, req_prod - req_cons = 250: the 250 pending requests were present in the active Xen PV ring, but tapdisk had not consumed them and still reported zero outstanding requests.
Additional state:
in_polling=false
chkrng_event=57
stoppolling_event=58
poll_duration=8000
poll_idle_threshold=50
We placed a breakpoint on tapdisk_xenio_ctx_ring_event for eight seconds. It did not fire. GDB then detached and tapdisk remained responsive.
After preserving the original ring and guest state, we executed the suggested notification:
call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port)
The call returned 0, but it did not restore I/O. Three subsequent guest samples remained at 249 inflight / 250 busy tags with no disk progress. The td_xenblkif ring indices were unchanged in a second GDB capture, and no new spurious-interrupt message appeared in the guest dmesg.
A forced VM reboot restored I/O and the node rejoined the cluster.
This appears to show that requests are present in the active PV ring but are not being consumed by tapdisk, and a backend-to-guest event-channel notification alone does not unblock the condition.
Could you advise which callback or scheduler state we should inspect next? In particular, should we also place breakpoints on tapdisk_xenblkif_cb_chkrng or related polling callbacks during the next occurrence?
We have the complete before/after td_xenblkif dumps, breakpoint log, notify log, guest snapshots and hypervisor data available for upload.
