I fly with two sometimes but yes I fly with a glove. The trick is to get a thin glove I have used bmx gloves nomex flight gloves and thin made deer skin gloves.
Now I dont just wear em to be cool, I wear em because when I get in a fight I take a firm grip on my controls and my hands sweat then my hand will slip...typically at the very worst moment. Now if you dont have that issue and dont want a touch more control then i wouldnt worry about it.