More drag, lets start with that. The 1940 Spit IAs (and later) had a "bubble canopy", a mirror, an armored windscreen, and an ariel (for the radio) as standard equipment. Radioless fighters were maybe big in Russia? I have no idea, the RAF kinda liked them.
The first Spit Vs were engine conversions from Spit IBs and Spit IAs, and the equipment was otherwise the same.
What increased drag there was from a Spit VB resulted from the 20mm cannon in the wings, where the Spit VA had no cannons.
The rest of the differences are attributed to the full throttle height differences between the Merlin III and the Merlin 45/46 series.
Also, you have to compare apples to apples. The BoB Spit I, Spit II and 1941 Spit V (early) all ran at either +9 (normal full throttle), or +12 lbs boost ( 5min wep on 100 octane). So you have to compare them that way.
There was not a big difference in the sea level speeds of the 3 types running at +12 lbs "flat out", the speed differences were minor, the biggest advantage a Spit V had was in #1 rate of climb and #2 service ceiling. Both considered important by the end of the BoB.
The Spit V Merlin 46 had the slightly larger impeller blower, compared to the Merlin 45. Both versions were @250 hp more than the Merlin IIIs, at least in 1941.
...Prop type, snow guard, armament and specific engine type would give slight differences in top speed of the various Mk Vs.