Intermittent Xen blkfront I/O stalls: all guest tags busy while tapdisk reports zero outstanding requests

kazual82

Hi,

We are investigating recurring complete I/O stalls affecting Linux data VMs running on XCP-ng 8.3.

We have experienced five incidents on different physical hypervisors. All affected hosts run the same XCP-ng, Xen, blktap, SM, and dom0 versions.

The most recent incident was captured in detail and appears to show a discrepancy between the Linux Xen blkfront state and the XCP-ng backend/tapdisk state:

The guest believed that all blkfront requests and tags were still in flight.
The backend/tapdisk reported zero outstanding requests.
Both Xen frontend and backend remained in the Connected state.
Only a VM reboot restored I/O.

We do not yet have a deterministic reproducer. The failures happen under a high-I/O OpenSearch workload with frequent fsync, flush, shard recovery, and merge operations.

Environment

Hypervisor stack, identical on the affected hosts:

XCP-ng 8.3 LTS
Build date: 20260430T09:28:41Z
xcp-ng-release-8.3.0-37
Xen: 4.17.6-6
Package: xen-hypervisor-4.17.6-6.2.xcpng8.3
XAPI: 26.1.3
dom0 kernel reported by uname: 4.19.0+1
blktap: 3.55.5-6.7.xcpng8.3
SM: 3.2.12-17.8.xcpng8.3

Guest:

Debian 12
Kernel: 6.1.0-47-amd64 (6.1.170-3)
12 vCPUs
64 GiB RAM
OpenSearch 3.4.0
Data filesystem: XFS
Data device: approximately 6.1 TiB
QCOW2 VDI on an LVM SR backed by HW raid with raid10 arrays

The OpenSearch cluster has 21 data nodes and approximately 18,800 active shard copies.

Symptoms in the guest

The VM remained reachable over SSH and the OpenSearch TCP ports remained open, but the local OpenSearch API stopped responding.

Observed guest metrics:

Load average around 38
Approximately 30 blocked processes
Approximately 91% I/O wait
I/O PSI:

some avg10=100.00
full avg10≈91.00

Kernel stacks contained calls such as:

blk_mq_get_tag
folio_wait_writeback
xfs_vm_writepages
xfs_file_fsync
xfs_log_force
fsync
fdatasync

There was no wbt_wait or rq_qos_wait in this incident.

WBT and THP had already been disabled:

/sys/block/xvdb/queue/wbt_lat_usec = 0
transparent_hugepage/enabled = never
transparent_hugepage/defrag = never

No completed I/O was observed in /proc/diskstats during a five-second interval, although approximately 31 requests remained in flight.

blk-mq state in the guest

For xvdb, debugfs showed:

hctx state: SCHED_RESTART

Driver tags:

nr_tags=32
nr_reserved_tags=0
active_queues=0
busy=32

Scheduler tags:

nr_tags=64
nr_reserved_tags=0
active_queues=0
busy=64

There were 32 requests marked as in flight, including a FLUSH request, and another request waiting in the dispatch queue.

The relevant queue settings were:

nr_requests=64
scheduler=[mq-deadline] none
wbt_lat_usec=0
max_sectors_kb=44
max_segments=11
max_segment_size=4096

The guest’s Xen blkfront module was configured with:

max_ring_page_order=0
max_queues=4
feature_persistent=Y

However, boot negotiation reported:

barrier: enabled
persistent grants: disabled
indirect descriptors: disabled
bounce buffer: disabled

State on the XCP-ng host

At the same time:

The physical LUN was in the running state.
Host-side latency and I/O wait were low.
There were no SCSI resets, timeouts, or I/O errors in dom0 logs.
Xen frontend state was 4 (Connected).
Xen backend state was 4 (Connected).
The frontend had one ring page/ring reference.
The backend advertised max-ring-page-order=3.
The tapdisk process was running.
tap-ctl stats reported:

reqs_outstanding=0
No message, map, VBD, image, or ENOSPC errors were reported by tapdisk.

Therefore, at the same point in time:

Guest blkfront: 32 requests still in flight, all driver tags exhausted.
Backend/tapdisk: zero requests outstanding.

Our current interpretation is a possible lost completion or Xen frontend/backend ring desynchronization. XFS and OpenSearch appear to be downstream victims rather than the original cause.

Recovery behavior

Stopping OpenSearch did not help because the requests and processes were stuck inside the kernel.

A normal guest reboot also hung during shutdown. We had to use:

xe vm-reboot uuid=<VM_UUID> force=true

After the VM reboot, XFS log recovery completed normally and the node returned to the OpenSearch cluster.

Canary test with a larger Xen ring

Because the backend advertised max-ring-page-order=3, we tested the following on the affected VM:

options xen_blkfront max_ring_page_order=3

We rebuilt initramfs and rebooted the VM.

After reboot, xenstore showed:

ring-page-order = 3
ring-ref0 ... ring-ref7
state = 4

The guest queue then showed:

driver tags: 256
scheduler tags: 256
nr_requests: 256

The VM successfully started OpenSearch, completed shard recovery, and the cluster returned to green. No blkfront tags remained busy after recovery.

This is only a canary result. We do not consider it proof that the original issue is fixed. It may only provide more queue capacity and delay the same failure if a completion is actually being lost.

Questions

Is there a known XCP-ng 8.3, blktap, or Xen issue matching this frontend/backend state mismatch?
Can tapdisk or the Xen backend lose a completion while both sides remain in state Connected?
Is xen_blkfront.max_ring_page_order=3 considered safe and recommended with this backend?
Could increasing the ring size reduce the probability of this failure, or would it only mask it temporarily?
Is the fact that persistent grants and indirect descriptors are not negotiated relevant?
Are there additional tapdisk, xenstore, event-channel, or ring diagnostics that we should enable before the next incident?
Are there candidate packages or patches that would be useful to test on one canary host?
Could the use of a multi-terabyte QCOW2 VDI on an LVM SR be relevant to this failure mode?

We can provide the complete blk-mq request list, xenstore frontend/backend trees, tapdisk statistics, and sanitized host logs if useful.

Thank you.

olivierlambert

Ping @Team-Storage