How about making the buttons work like buttons on most OSes? More specifically, if you move the mouse pointer off of a button and release the mouse button, the button should not "activate".
Try scrolling the map and moving the cursor over one of the surrounding buttons to see why this can be annoying.
