Lazs,
I had the same problem. In my case, it turned out to be the Mic had the wrong...oh, hell what's the word....IMPEDENCE. I think that's what it was.
Either way, I bought a mic that was 9 buck from radio chack, and it had the right impedence, and I've not had a problem since.
One good way to test your VOX without pissing off your squaddies. You may already know this, but instead of typing a VOX channel in Radio 5, type in your callsign. You will hear exactly what your squaddies hear when you transmit.
Good Luck!