Hello @anthoineb,
We had another occurrence and this time tested the exact GDB method you suggested. It successfully unlocked the ring.
The incident affected os-ott-data-3-2 (172.30.52.193) on hypervisor 172.30.50.191.
Before the test, the guest state was:
254 requests in flight
255/256 blk-mq tags busy
I/O PSI full: approximately 94-97%
3 tasks in D state
no completed xvdb I/O progress
Tapdisk remained responsive and reported:
reqs_outstanding=0
tap/image/VBD/xenbus errors=0
The untouched active td_xenblkif state was:
domid=26
devid=832
port=343
ring_n_pages=8
ring_size=256
n_reqs_free=256
in_polling=false
req_prod=2243281141
req_cons=2243280886
rsp_prod=2243280886
rsp_prod_pvt=2243280886
Therefore, exactly 255 requests were waiting in the active PV ring.
After preserving the baseline, we executed once:
call (void)tapdisk_xenblkif_sched_chkrng(blkif)
The indices were unchanged immediately inside GDB, but within approximately five seconds:
tapdisk reqs_outstanding: 0 -> 256
guest write inflight: 254 -> 0
I/O PSI full: approximately 98% -> 24% and continuing to decrease
xvdb completed-I/O counters started advancing
The post-test GDB capture showed:
in_polling=true
req_prod=2243366603
req_cons=2243366603
rsp_prod=2243366600
rsp_prod_pvt=2243366600
n_reqs_free=253
The previously stuck requests had been consumed and normal new I/O was flowing.
OpenSearch recovered its transport communication and the node rejoined the cluster without restarting the OpenSearch service and without rebooting the VM. The cluster completed shard recovery and became green at 17:05 MSK.
We also checked SMlog, xensource.log, and daemon.log for this incident. There was no VM or storage pause/unpause immediately before or during the stall.
This appears to confirm that tapdisk was not performing the scheduled ring check, and explicitly calling tapdisk_xenblkif_sched_chkrng() restored processing.
Would it be reasonable to use this call as an automated recovery action when the same exact signature is detected, before falling back to a VM reboot? We have the complete before/after GDB captures and guest/hypervisor diagnostics available if useful.
