Intermittent Xen blkfront I/O stalls: all guest tags busy while tapdisk reports zero outstanding requests

anthoineb

Hello @mike.potapov , on n_reqs=32 and n_reqs_free=32, I think you're watching td_blktap_t structure. This structure is only used if the pvdriver are not install in the VM, so it's normal this ring is not used. You should watch struct td_xenblkif that is the ring use normally and that must contains 256 requests on max_order=3.

The tapdisk will wakeup periodically, you could put a breakpoint on tapdisk_xenio_ctx_ring_event to confirm that the ring is check periodically. You can investigate td_xenblkif here.

Ultimatly, you can try to send a notification from the tapdisk under gdb to the linux kernel with call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port); to see if it unlock something on the guest side. It it's the case it will confirm a notification was lost. If not you will get a message about spurious interrupt in the dmesg of the guest kernel.

mike.potapov

Hello @anthoineb,

We captured another occurrence and repeated the diagnostics you suggested. This time we inspected the active struct td_xenblkif, not the legacy td_blktap_t.

The incident affected os-ott-data-2-4 (172.30.52.185) on hypervisor 172.30.50.181. The guest had xen_blkfront.max_ring_page_order=3 active.

Guest state:

no completed xvdb I/O progress;
249 requests in flight;
250 of 256 blk-mq tags busy;
I/O PSI full approximately 97%;
11 tasks in D state;
OpenSearch remained locally responsive for part of the incident, but the node had left cluster membership.

Tapdisk remained responsive and reported:

reqs_outstanding=0
xenbus reqs=[8105673748,8105673748]
tap/image/VBD/xenbus errors=0

The active td_xenblkif state was:

domid=20
devid=832
port=325
ring_n_pages=8
ring_size=256
n_reqs_free=256

req_prod=3810706702
req_cons=3810706452
rsp_prod=3810706452
rsp_prod_pvt=3810706452

Therefore, req_prod - req_cons = 250: the 250 pending requests were present in the active Xen PV ring, but tapdisk had not consumed them and still reported zero outstanding requests.

Additional state:

in_polling=false
chkrng_event=57
stoppolling_event=58
poll_duration=8000
poll_idle_threshold=50

We placed a breakpoint on tapdisk_xenio_ctx_ring_event for eight seconds. It did not fire. GDB then detached and tapdisk remained responsive.

After preserving the original ring and guest state, we executed the suggested notification:

call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port)

The call returned 0, but it did not restore I/O. Three subsequent guest samples remained at 249 inflight / 250 busy tags with no disk progress. The td_xenblkif ring indices were unchanged in a second GDB capture, and no new spurious-interrupt message appeared in the guest dmesg.

A forced VM reboot restored I/O and the node rejoined the cluster.

This appears to show that requests are present in the active PV ring but are not being consumed by tapdisk, and a backend-to-guest event-channel notification alone does not unblock the condition.

Could you advise which callback or scheduler state we should inspect next? In particular, should we also place breakpoints on tapdisk_xenblkif_cb_chkrng or related polling callbacks during the next occurrence?

We have the complete before/after td_xenblkif dumps, breakpoint log, notify log, guest snapshots and hypervisor data available for upload.

anthoineb

Hello @mike.potapov , ok that's interesting, it seems that the lack of communication is in the other way, from guest to tapdisk.

To try unlock the ring on tapdisk side you can either:

send a SIGUSR2 to force the polling of tapdisk: kill -s SIGUSR2 <TAPDISK_PID> . Problem is the polling will start only if the cpu load is low enough.
or with gdb, call (void)tapdisk_xenblkif_sched_chkrng(blkif). Same command the polling is doing above but without the cpu load check.

Can you confirm one of the method check the ring and unlock it?

SMlog is doing a pause/unpause at the moment of the lock or just before?

I will investigate on how a event channel could be lost in tapdisk because I trust the kernel to send it and the hypervisor to deliver it.

mike.potapov

@anthoineb,

We checked both SMlog and xensource.log for the two latest incidents on hypervisor 172.30.50.181.

There was no VM or storage pause/unpause operation before either stall. The only matching pause/unpause pairs were part of our later forced VM.hard_reboot:

2026-08-19, VM 172.30.52.185: stall confirmed around 20:24; pause at 20:36:37 and unpause at 20:38:12.
2026-08-20, VM 172.30.52.182: stall confirmed around 20:24; pause at 20:30:07 and unpause at 20:31:26.

The first related VDI operations in SMlog were also the vdi_deactivate actions initiated by those reboots. Therefore, we do not see a pause/unpause immediately before or at the onset of either lock.

mike.potapov

Hello @anthoineb,

We had another occurrence and this time tested the exact GDB method you suggested. It successfully unlocked the ring.

The incident affected os-ott-data-3-2 (172.30.52.193) on hypervisor 172.30.50.191.

Before the test, the guest state was:

254 requests in flight
255/256 blk-mq tags busy
I/O PSI full: approximately 94-97%
3 tasks in D state
no completed xvdb I/O progress

Tapdisk remained responsive and reported:

reqs_outstanding=0
tap/image/VBD/xenbus errors=0

The untouched active td_xenblkif state was:

domid=26
devid=832
port=343
ring_n_pages=8
ring_size=256
n_reqs_free=256
in_polling=false

req_prod=2243281141
req_cons=2243280886
rsp_prod=2243280886
rsp_prod_pvt=2243280886

Therefore, exactly 255 requests were waiting in the active PV ring.

After preserving the baseline, we executed once:

call (void)tapdisk_xenblkif_sched_chkrng(blkif)

The indices were unchanged immediately inside GDB, but within approximately five seconds:

tapdisk reqs_outstanding: 0 -> 256
guest write inflight: 254 -> 0
I/O PSI full: approximately 98% -> 24% and continuing to decrease
xvdb completed-I/O counters started advancing

The post-test GDB capture showed:

in_polling=true
req_prod=2243366603
req_cons=2243366603
rsp_prod=2243366600
rsp_prod_pvt=2243366600
n_reqs_free=253

The previously stuck requests had been consumed and normal new I/O was flowing.

OpenSearch recovered its transport communication and the node rejoined the cluster without restarting the OpenSearch service and without rebooting the VM. The cluster completed shard recovery and became green at 17:05 MSK.

We also checked SMlog, xensource.log, and daemon.log for this incident. There was no VM or storage pause/unpause immediately before or during the stall.

This appears to confirm that tapdisk was not performing the scheduled ring check, and explicitly calling tapdisk_xenblkif_sched_chkrng() restored processing.

Would it be reasonable to use this call as an automated recovery action when the same exact signature is detected, before falling back to a VM reboot? We have the complete before/after GDB captures and guest/hypervisor diagnostics available if useful.

anthoineb

@mike.potapov yes, it is safe but it's a bit a heavy procedure to recover. I will open a ticket on our side to investigate now we have the root-cause.

anthoineb

@mike.potapov Can you upgrade to the latest blktap-3.55.5-9.3.xcpng8.3 to check is the issue is still there?

mike.potapov

Hello @anthoineb,

Following your suggestion to test a newer blktap release, we installed blktap-3.55.5-9.4.xcpng8.3.x86_64 and matching debuginfo on all three hypervisors. We then fully shut down and started all 21 OpenSearch data VMs, completing this on September 11. We verified that their running tapdisk executables matched the installed binary.

Unfortunately, the stall recurred on September 13 on os-ott-data-1-5.

For the affected tapdisk process (PID 2797568), we verified during the incident:

/proc/2797568/exe pointed to /usr/libexec/tapdisk, without (deleted);
its SHA-256 matched the installed executable;
GDB loaded matching debug symbols, build ID dc98a78aff623c1cf12518663c859e3efa32413c.

Before recovery:

guest I/O made no progress;
255 requests were in flight and 256/256 tags were busy;
I/O PSI full was approximately 96–97%, with four tasks in D state;
tapdisk reported zero outstanding requests.

The active td_xenblkif ring showed:

req_prod     = 866947066
req_cons     = 866946810
rsp_prod     = 866946810
rsp_prod_pvt = 866946810
nr_ents      = 256

Thus, 256 requests were pending in the active PV ring but had not been consumed.

After preserving the baseline GDB capture, our recovery controller executed your suggested call once:

call (void)tapdisk_xenblkif_sched_chkrng(blkif)

The call completed at 09:40:14 MSK (UTC+03:00). By 09:40:25, guest I/O was progressing again, inflight requests and busy tags were zero, and the local OpenSearch API was responding. The post-recovery GDB capture showed all four ring counters equal to 866967322.

The node rejoined the cluster at 09:42:00 without restarting OpenSearch or rebooting the VM.

So the same ring-processing stall still occurs with the running 9.4 binary, and the explicit ring-check call still restores I/O. We have complete before/after GDB captures and the recovery-call log available.

Is there a newer build or a specific scheduler/event-channel state you would like us to capture during the next occurrence?

anthoineb

Hello @mike.potapov , We found a race-condition happening on Qcow2 driver that explains this issue. We developed a fix that will be ship in the next release of blktap.

anthoineb

@mike.potapov blktap-3.55.5-11.1.xcpng8.3 was released today in xcp-ng-testing with the fixed to avoid stalled IO. (Announce in https://xcp-ng.org/forum/post/108803). Can you give it a try?