There was a guy I met a few years ago. He had a friend who flew the SR-71. Now I dont have any proof, but he told me stories about this pilot that flew the SR-71. Based on his stories, no one really knows for a fact how fast the Blackbird can go. No one has ever tried to run on full power to see how much faster it can fly. Also, the SR-71 can fly much higher than 100,000 feet. The pilot had taken the plane so high up that the engines quit running because of lack of atmosphere for combustion in the engines.
Dont take my word on this, but what Im telling you guys is what the SR-71 pilot had spoken of.
Mach 3.5 is probably a pretty realistic number. The limiting factor is thermal, not drag. The windscreen frame and LE of the vertical tails hit some righteous temperatures at those speeds.
BTW, no one has mentioned the X-15. Granted it was rocket-powered and air-launched, with minimal endurance, but the records it set were absolutely staggering for it's time (and now

) Looking strictly at Speed/Altitude performance, everything else that's ever flown (with the possible exception of the mythical "Aurora") is a Piper Cub by comparison.