Backup failures after VHD to QCOW2 conversion

SeriouZ

Hi everyone,

After migrating VMs between pools and converting their disks from VHD to QCOW2, some VMs repeatedly fail during backup export and snapshot cleanup. Other QCOW2 VMs and VHD-backed VMs back up successfully.

Environment

Two XCP-ng 8.3 hosts, both updated and rebooted
Shared LVM-over-iSCSI storage on Synology (approximately 8.6 TiB free)
NBD enabled
XO Server 5.213.0, commit a50776cb7, running in the Ronivay Docker image

Recurring errors

QCOW2 export times out and falls back to a full:

BodyTimeoutError: Body Timeout Error
code: UND_ERR_BODY_TIMEOUT
can't compute delta …, fall back to a full

Other export attempts receive a host storage-backend error:

invalid HTTP header in response body
HTTP/1.1 500 Internal Error
SR_BACKEND_FAILURE_1200

The response includes a failed disk-close operation:

tapdisk_control_close_image: device qcow2:… still open
tap-ctl close … -t 120
Connection timed out
status=110

Snapshot cleanup then fails:

Couldn't disconnect … from dom0
OPERATION_NOT_ALLOWED: VBD … still attached to …
VDI_IN_USE(…, destroy)

We confirmed that the affected attachments belong to snapshot VDIs in dom0. Repeated VBD.unplug_force operations appear, and backup retries encounter the same problem.

The problem persists after updating and rebooting both hosts and updating XO.

Has anyone experienced similar QCOW2 backup and snapshot cleanup failures after a cross-pool migration? Is there a known fix or recommended recovery procedure?

poddingue

I don't know the QCOW2 side well enough to say for sure, but your symptoms look close to a thread from last month: https://xcp-ng.org/forum/post/108888 .
There, @dthenot explained that with QCOW2 the tapdisk process holding the disk isn't stopped the way it is for VHD, and that the fixes landed in sm-3.2.12-25.1.

Could you post the output of rpm -q sm blktap from both hosts? "Updated" can mean a few different package sets, and that would show whether you already have those fixes or are hitting something new.
The SR_BACKEND_FAILURE_1200 and the VBDs still attached to snapshot VDIs in dom0 sound like something @Team-Storage will want to look at either way.

SeriouZ

@poddingue said:

rpm -q sm blktap

Both hosts have the following versions installed:

sm-3.2.12-25.2.xcpng8.3.x86_64
blktap-3.55.5-12.3.xcpng8.3.x86_64