Just to close the loop on this: XO 6.4 (released on April 29) includes a specific fix for VNC reconnection, which may have been causing keyboard focus loss and, in turn, modifier keys to get stuck (PR #9727).
At least, that’s my understanding; I could be completely wrong, so please take it with a grain of salt.
If you’re on XO 6 and still seeing the issue, upgrading to 6.4 may help.