I've never had the issue, but I also use an SST profile. I wonder if it might be due to the stick treating the pinky as a shift state button even when there's no profile loaded, and having the test window open bypasses that?
I'd be curious if someone who has this issue were to use a blank profile with SST with no shift states and see if the button starts working without the test window open.
Wiley.