It's not about weight or aerodynamics... the engine is simply geared for higher alts. Simple as that. It's geared to run better higher, at the expense of lower alts. Late in the war they decided to lower the altitude gearing on the P-51D for various reasons, resulting in better performance lower down.
You'll notice the P-51D is somewhat faster than the B up to about 24k, where its power drops off.