The PT boats, MGB, MTB are basically made of wood, when you compare it to a 5 inch AAA shell think tinfoil and since there was no overhead protection, open cockpit, exposed gunnery positions the shrapnel from the shell could take out the majority of the crew with a near miss.
The PT,MTB,MGB all used high octane aviation fuel and a piece of hot shrapnel into a fueltank could result in a loud bang or at the very good possibilityof a severe shipboard fire, under combat with someone shooting at you.
The hull and deck were basically made of wood with some armor plating added to try and protect the bridge and in other significant areas.
German Eboats were better armed and armoured and had diesel engines, so the flash fire effect associated with Aviation fuel was absent.
Using AAA/AA and heavier guns against coastal forces (PT,MGB,MTB) was a perfectly accepted doctrine in any Naval force, since the Boats were very vulnerable to any damage and the crews were the most vulnerable and a PT almost doesn't have the neccessary mass to actually activate the delayed fuse on the AP shell, unless you struck an engine or gun mounting.
"Granted thats a airburst 5" round that works pretty good at killing planes. But it should NOT kill PT boats. "
See above comment.
Even if the shells were not to arm at closer range (AAA) the shells still can be considered solid shot and would be garaunteed to make it an exceedingly bad day to be in a PT boat, should one strike your plywood boat.
Even land forces have used shrapnel shells (both designed and the improvised use of AA/AAA) to take out troops and since a PT,MTB,MGB cannot fight effectively or at all without a crew, the use of a 5 inch AAA shell would be a viable option to take care of a close in torpedo attack or gun attack in the case of an MGB (Motor Gun Boat, British).
PTs and the like were generally used against enemy shipping which was either escorted by an armed trawler or obsolete destroyer/corvette or PT's,MGB,MTB or were hugging the shoreline and moving at night, hardly if ever during the day.
To go up against a Battle Group was the height of foolishness or desperation and immediately draw a lot of unwelcome attention.