VDI to coalesce number should decrease over time yes. If it's not, you might have a coalesce issue in your storage, check the /var/log/SMlog in the relevant host (master for shared storage, the host attached to its local storage otherwise).
Rescan will scan for the LV/VHD files. But coalesce is automatically checked/started every 5 minutes. Coalesce can take time (depending on SR read/write and quantity to "merge"). Next releases will have a visible task for pending coalesce (next major XCP-ng release). Until then, you can do a ps axf | grep coalesce to see if there's something happening.
