Hello @kazual82 Sorry for the delay, I was working on a gdb script for you before I was preempt. Yes, I'm curious about GDB backtraces and ring dump. I'm puzzled because even if we miss a notification from the frontend, tapdisk periodically checks blkif ring, so he must see the request with a delay at least. Can you also share your daemon.log and SMlog?

Hello @kazual82,
Lot of question here, I'll try to answer my best:

• Is there a known XCP-ng 8.3, blktap, or Xen issue matching this frontend/backend state mismatch?
  No, I never saw this kind of issue but I appreciate all your investigation.

• Can tapdisk or the Xen backend lose a completion while both sides remain in state Connected?
  It'd surprise me because my opinion is tapdisk is sending to much completion notifications. My guess is a race condition somewhere that loose the notification.

• Is xen_blkfront.max_ring_page_order=3 considered safe and recommended with this backend?
  Yes, it's safe but there is no recommendation on that because any max_ring_page_order should work.

• Could increasing the ring size reduce the probability of this failure, or would it only mask it temporarily?
  I guess it reduce the probability.

• Is the fact that persistent grants and indirect descriptors are not negotiated relevant?
  No, it's expected, tapdisk doesn't support those features.

• Are there additional tapdisk, xenstore, event-channel, or ring diagnostics that we should enable before the next incident?
  I don't see what we can do. If it appends again I'm curious to investigate. Install blktap-debuginfo and devtoolset-11-gdb to attach the debugger on tapdisk and see the state of the blkif ring, that could be an info if the requests are still in the ring. If you can open a support tunnel, I will investigate.

• Are there candidate packages or patches that would be useful to test on one canary host?
  The next update, yet to be release, will add a bunch of fixes for QCOW2 support in tapdisk and sm. I recommend this update.

• Could the use of a multi-terabyte QCOW2 VDI on an LVM SR be relevant to this failure mode?
  I fear this is a QCOW2 issue only linked to the recent release of the feature but no real clue here.

The fact that tap-ctl stats answer is the proof that there is no deadlock on the tapdisk side, that's a small reason of satisfaction.

@limezest You're right, we'll fix this.

With this typo, the elif branch will always be false and not executed, it's dead code. As it is a less common issue than the first branch of the if, the script will left behind this particular case but will still fix the majority of snapshot_of problems.
The script can still be used safely.

Hello all,

Here we (the XCP-ng team) share a script that will help workaround the snapshot-of issue, also known as VDI not showing in XO. But first, a bit of context.

The issue

This issue has existed for quite some time, first report was late August / September 2025. Sadly, it gets more and more visible over time.

The issue is that there are 2 database fields relating to snapshot that are redundant, and the issue is that sometimes they contradict each other.

The database field that gets changed (snapshot_of), can be done so both by xapi, or SM (smapi). To fix this design issue, a new revert operation is needed but designing a new API takes time and a lot of care.

On the XAPI side, this change is implemented. The smapi part is work in progress and high on the priority list.

The script

This script only treats the symptoms of the issue and not the cause.

This script allows users to fix XAPI database records with incongruent snapshot metadata.

Changing XAPI database is always a risky operation and should be considered carefully

To operate, the script temporarily disables HA and stops XAPI to apply the changes to the database. This means that the pool is not running operations like handling backups, migrating, starting or stopping VMs, and HA is disabled during the operation.

The script needs to be run on the master host of the affected pool.

In the unlikely case the script corrupts the database, the script creates a backup before modifying the database and provides an option to restore the database from this backup.

Usage

This script is not compatible with XCP-ng 8.2.1

Download the snapshot-fixer.py script here.

Check that content of the file is correct:

# sha256sum snapshot-fixer.py 
3aad01563f813571364357364f803c61cc59049fee6e8b24cfa964a03444f609  snapshot-fixer.py

snapshot-fixer.py usage:

# python3 snapshot-fixer.py -h
usage: snapshot-fixer.py [-h] {dry-run,restore-backup,rewrite} ...

Rewrite erroneous VM snapshot links.

positional arguments:
  {dry-run,restore-backup,rewrite}
    dry-run             Prints invalid values in the database, does not modify
                        any file
    restore-backup      Find a previous backup and restore it
    rewrite             Backup xapi's database, and rewrite it.

optional arguments:
  -h, --help            show this help message and exit

First run the script in dry-run mode. This will not disable the HA nor the XAPI, only print the invalid values detected that could be corrected by the script:

# python3 snapshot-fixer.py dry-run

Check the output and confirm that the VDIs are the ones you need to fix before continuing with the next steps.

The rewrite operation could take 10 seconds or more because disabling HA and XAPI are slow operations. Be patient and don't stop the script. Remember that this script should run on the master host of the affected pool.

# python3 snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...

The last command available is restore-backup. As rewrite this command will disable HA and XAPI to operate and that could take seconds, be patient.

# python3 snapshot-fixer.py restore-backup

The next section will list error messages that could be displayed if something goes wrong.

Troubleshooting

Starting xapi timed out. Please make sure it's working by running `systemctl status xapi`

After a rewrite or restore-backup operation, the script will wait 15 seconds to re-enable the XAPI. If the timeout is reached, you will see this message asking you to re-enable it yourself.

HA was disabled and needs to be enabled back again manually. Please re-enable it by running `xe pool-ha-enable`

After a rewrite or restore-backup operation, the script will wait 30 seconds to re-enable the HA (it it was enabled before). If the timeout happens, you will see this message with the command to re-enable the HA by hand.

File '/var/lib/xcp/state.db.snapshot_of.backup' already exists, aborting. If you are sure you want to run the command again, please delete the file

This message means that the command rewrite was already run and a backup was created during the operation.

Anything else?

Don't hesitate to share your feedback on this thread to get help on this issue.

Hello @igorglock, Damien is in holiday this week but he identified the issue and a patch should be tested for the next release of SM.

@andrewperry Hi, we are working on 2 things:

1. a proper fix under development to avoid new issue where snapshot-of field is garbage.
2. a script to fix the wrongly set snapshot-of already present on affected pools.
   Both are necessary and we hope we can release the script soon to at least workaround if the issue happen again before we release the fix.

@olivierlambert Yes, we saw this before, we are investigating.

@wilsonqanda qcow2 packages are in a separate repository. You should have setup the repo, a grep -r "qcow2" /etc/yum.repos.d/ should tell you if it was setup on your host.

@wilsonqanda, can you share the /var/log/SMlog? Do you installed qcow2 release and use some qcow2 VDIs?

@acp Here is the procedure to re-insert your host in the SR.

Make sure you have the required packages by running these commands on it:

yum install -y xcp-ng-release-linstor
yum install -y xcp-ng-linstor

It should be the case because the node was running the services before, but better to check.

And then restart the toolstack to detect the LINSTOR driver:

xe-toolstack-restart

Ensure you have the same configuration on each PBD of your XOSTOR SR using this command.

xe pbd-list sr-uuid=<UUID>

All device-config must be the same.

Then, use this command with the correct <GROUP_NAME> and <HOST_UUID> to add the node to the SR:

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=addHost args:groupName=<GROUP_NAME>

For a short description, this command (re)create a PBD, open DRBD/LINSTOR ports, start specific services and add the node in the LINSTOR database.

A storage layer is not directly added to this node. You can verify the storage state like this:

linstor sp list

You shouldn't see the storage pool of your node yet.

Run the correct command where the controller is running to add the volume group in the LINSTOR database:

# For thin:
linstor storage-pool create lvmthin <NODE_NAME> <SP_NAME> <VG_NAME>

# For thick:
linstor storage-pool create lvm <NODE_NAME> <SP_NAME> <VG_NAME>

A new linstor sp list should show you the node now.