Most of the time when a vehicle, be it aircraft or ground based, is designed for multiple roles it is going to be more expensive because it has to cover more areas of expertise. Also, when trying to cover all the bases it typically is not able to do them all well, either.
A tank destroyer, be it an M10, M18, Jagdpanzer V, Su-85/Su-100, Firefly, Archer, etc, is usually less able to deal with infantry due to fewer MG's (if any) and a less then average HE shell (typically). As such, they stood off and searched for enemy armor to engage, not enemy infantry.