The existance of TDs in an army does not make other tanks "infantry support tanks". The Marder II didnt make the Panzer IV an infantry support tank. The Su-100 didnt make the T-34/85 an infantry support tank, and the M-10 didnt make the Sherman one either. The lack of AP power (or any other drawback) in its gun does not alter its designation or intended use on the battlefield.
The infantry tank concept was used by the British in the form of tanks like the Churchill and deployed into independant tank brigades. The Germans used assault guns like the StuG III in the same role. The Shermans that went into action at El Alemain, Tunisia, Sicily, Italy, Normandy and Germany in the US, British and Canadian armies were deployed as multi-role main battle tanks. The role in the US Army of the tank destroyer was the same as other armies in most respects; they were intended to rapidly deploy to the area of an enemy armored attack and thwart it with the help of artillery and anti-tank guns. There were certainly M4 Sherman support variants but thats another matter.
Although it perhaps leaned towards being a TD The Firefly was not a designated tank destroyer. TDs (called Self Propelled guns in the British Army, but a rose by any other name I guess...) were issued to and manned by members of the Royal Artillery Branch. The Firefly Sherman was issued, deployed and used within standard medium tank regiments as an mbt by members of the Royal Armored Corps. They were issued as a % to units in 1944-45 because they could not be manufactured in time to completely replace the Shermans with the 75mm guns. The fact that they were better able to handle German armor doesn't classify them as tank destroyers any more than a US Army Sherman M4A3 (76) was or the later M26 Pershing was, or the later British Comet was. That being said a Firefly equipped troop would certainly get a "tap on the shoulder" if enemy armor was expected as they were there to improve an armored units ability to deal with Panthers and Tigers. All that said I get why its sometimes referred to as a TD being that it was designed with German armor in mind, but like I stated no more than other later Allied tanks were.