mike.potapov

Hello @anthoineb,

We captured another occurrence on the same VM, os-ott-data-3-2
(172.30.52.193), on the same hypervisor and QCOW2 VDI as the incident
reported on 2026-07-25.

This occurrence is useful because xen_blkfront.max_ring_page_order=3 was
definitely active:

• xenstore showed ring-page-order=3 with eight ring references;
• the guest queue depth was 256;
• 255 write requests remained in flight;
• all 256 blk-mq scheduler tags were busy;
• wbt_lat_usec=0;
• I/O PSI full was approximately 98%;
• completed xvdb counters did not advance between samples.

The backend state was nevertheless the same as in the previous capture:

• tapdisk was sleeping in scheduler_wait_for_events();
• tap-ctl reported reqs_outstanding=0;
• tap request counters were 0/0;
• req_prod=0 and rsp_prod=0;
• no tap, image, VBD, map or xenbus errors were reported.

One detail may be relevant: GDB still printed n_reqs=32 and
n_reqs_free=32, although the frontend had negotiated an order-3 ring and the
guest exposed 256 tags. Is this expected because n_reqs represents a separate
fixed tapdisk request pool, or could it indicate that tapdisk is not observing
the complete multi-page ring?

The controller captured GDB and the guest/hypervisor state before reboot. After
the ten-minute diagnostic window, the same signature was confirmed three more
times and the VM was rebooted. It rejoined the cluster successfully.

I have attached:

• the complete GDB/tap-ctl capture;
• the guest blk-mq, diskstats, PSI, D-state stack and kernel-journal snapshot;
• the corresponding tapdisk/xenstore, daemon.log and SMlog excerpt;
• a README and SHA-256 checksums.

The important new result appears to be that increasing
max_ring_page_order from 0 to 3 did not prevent the stall; the same mismatch
recurred with 255/256 frontend requests/tags occupied while the backend ring
and tapdisk remained empty.
20260730-005220-gdb-172.30.52.193.txt 20260730-pre-reboot-guest-172.30.52.193.txt 20260730-pre-reboot-hypervisor-172.30.50.191.txt README.txt SHA256SUMS.txt

Hello @anthoineb,

Thank you. I have attached the requested evidence as separate text files because the forum does not accept ZIP archives:

• two complete GDB/tap-ctl captures taken during the same incident, before reboot;
• the corresponding daemon.log excerpt from the hypervisor;
• the corresponding SMlog excerpt from the hypervisor;
• a README containing the timeline and identifiers;
• SHA-256 checksums.

Both host-log excerpts cover 2026-07-25 04:35:00-05:10:00 MSK (UTC+03:00).
The first GDB capture started at 04:45:44, the independent repeat capture
started at 04:48:03, and the forced VM reboot was requested at 04:58:33.

Both captures show the same state:

• the tapdisk main thread was in scheduler_wait_for_events();
• n_reqs=32 and n_reqs_free=32;
• req_prod=req_cons=rsp_prod=rsp_prod_pvt=0;
• tap-ctl reported reqs_outstanding=0;
• the guest still reported 31 requests in flight and all 32 blkfront tags busy.

The host logs contain no tapdisk error before the reboot. At 04:58:34-04:58:36
they show the expected sring disconnect, tapdisk close/detach and clean shutdown
after the forced reboot request. The new QCOW2 tapdisk was opened at 04:59:57
and its sring connected at 05:00:27.

Please let me know if you need another structure printed from GDB or a wider
host-log interval. Our recovery controller can preserve the same pre-reboot
diagnostic window during the next occurrence.
SHA256SUMS.txt README.txt osott-193-SMlog-20260725-0435-0510.txt osott-193-daemon-20260725-0435-0510.txt 20260725-044732-gdb-172.30.52.193.txt 20260725-044544-gdb-172.30.52.193.txt