Hello @anthoineb,
Following your suggestion to test a newer blktap release, we installed blktap-3.55.5-9.4.xcpng8.3.x86_64 and matching debuginfo on all three hypervisors. We then fully shut down and started all 21 OpenSearch data VMs, completing this on September 11. We verified that their running tapdisk executables matched the installed binary.
Unfortunately, the stall recurred on September 13 on os-ott-data-1-5.
For the affected tapdisk process (PID 2797568), we verified during the incident:
/proc/2797568/exe pointed to /usr/libexec/tapdisk, without (deleted);
its SHA-256 matched the installed executable;
GDB loaded matching debug symbols, build ID dc98a78aff623c1cf12518663c859e3efa32413c.
Before recovery:
guest I/O made no progress;
255 requests were in flight and 256/256 tags were busy;
I/O PSI full was approximately 96–97%, with four tasks in D state;
tapdisk reported zero outstanding requests.
The active td_xenblkif ring showed:
req_prod = 866947066
req_cons = 866946810
rsp_prod = 866946810
rsp_prod_pvt = 866946810
nr_ents = 256
Thus, 256 requests were pending in the active PV ring but had not been consumed.
After preserving the baseline GDB capture, our recovery controller executed your suggested call once:
call (void)tapdisk_xenblkif_sched_chkrng(blkif)
The call completed at 09:40:14 MSK (UTC+03:00). By 09:40:25, guest I/O was progressing again, inflight requests and busy tags were zero, and the local OpenSearch API was responding. The post-recovery GDB capture showed all four ring counters equal to 866967322.
The node rejoined the cluster at 09:42:00 without restarting OpenSearch or rebooting the VM.
So the same ring-processing stall still occurs with the running 9.4 binary, and the explicit ring-check call still restores I/O. We have complete before/after GDB captures and the recovery-call log available.
Is there a newer build or a specific scheduler/event-channel state you would like us to capture during the next occurrence?