@anthoineb We captured another occurrence, this time on os-ott-data-3-2, running on hypervisor 172.30.50.191.
The guest-side state was:
31 requests in flight
all 32 blkfront tags busy
I/O PSI full around 97%
6 tasks in D state
no completed xvdb I/O progress
The VM was still running and its local OpenSearch HTTP endpoint was responsive, but the node had already disappeared from OpenSearch cluster membership.
On the hypervisor:
tapdisk was responsive to tap-ctl
reqs_outstanding: 0
tap request counters: 0/0
all xenbus request counters were balanced
no tapdisk errors were reported
We attached GDB to the affected tapdisk before reboot. The main thread was sleeping in scheduler_wait_for_events(), and the captured blktap/shared-ring state was:
req_prod = 0
req_cons = 0
rsp_prod = 0
rsp_prod_pvt = 0
n_reqs = 32
n_reqs_free = 32
The complete shared ring was empty. Therefore, at capture time, there were no requests waiting in the tapdisk ring, while the guest still believed that 31 requests were in flight and all frontend tags were occupied.
GDB detached normally. After the diagnostic window, the VM was rebooted and successfully rejoined the cluster.
At the time of the failure, this VM was still running with xen_blkfront.max_ring_page_order=0 and 32 tags. After reboot, the persistent setting became active:
xen_blkfront.max_ring_page_order=3
nr_tags=256
Could this evidence indicate that requests or completion notifications are being lost between blkfront and the backend before they become visible in the tapdisk shared ring?
We can provide the complete GDB backtraces and ring dump if useful.
