Based on the ANA numbers at Simmer's Paint Shop, Insignias White and Blue are matte, while Insignia Red is a gloss. However there's also an Insignia Red on that site with an FS number that points to a non-specular color (US Army Quartermaster Specification #45). Black is for the most part matte, though there's one semi-gloss (Instrument Black) and one gloss, however SPS doesn't specify which black was used for which purpose. That said, I think the gloss black would have been used for night camouflage (P-61s, Black Cat PBYs, P-38Ms, etc.) since the gloss paint would be less prone to giving away the aircraft visually if a spotlight is shined on it.
Some text, such as the U.S. Army stencils on the undersides of pre/early-war USAAF aircraft was done in Insignia Blue, so would have been matte.
However keep in mind, that regardless of whether a color is specular, semi-gloss, or non-specular, there's always going to be a SLIGHT specularity due to the way light interacts with the paint surface, so you don't want to make any of the paint 100% black on your spec map. I'd give lighter colors a slightly higher spec than darker colors for two colors of the same specularity (matte vs. gloss).