No need to. Most of us realize that adding stuff on slows us down. Heck, supposedly even taking the 30mms on the 190a8 takes off 1mph.
However, you can't let that stop you. Take what you need. You're not running a race, and most of the time a couple of MPH from a drop tank shackle aren't going to mean life or death.
Rocket tubes, on the other hand....
Well, don't take 'em if you only plan on dogfighting.