Yes that has been an ongoing issue from the start of AH3. If the any of the buildings on the port are destroyed, and a GV is able to get inside, when the building Restores, they are able to hide in there and shoot out, unseen.
The only work around currently, that I know of is to destroy the building again, and kill the gv inside. IIRC that has happened in the Towns on occasion, albeit not very often.