Shrimp,
You're right. But don't forget a few things...
Every everest climber who goes up without O2 has trained for many many years to acclimatize for that environment. There are massive physiological changes that have to occur to be able to do that.
The experiences(and mysterious deaths) of those WWII pilots and the early jet pilots directly led to a lot of very strict rules. Another issue with high altitude flying is the potential to get the bends. It took years of research and collaboration between diving and flying pioneers to figure out what was going on there and to set limits. We know now that 25,000 is the max safe altitude for unpressurized aircraft because bad things happen to even highly trained and acclimatized pilots above that alt.
I know of highly experienced T-37 pilots who nearly died following just 2 1.5 hour sorties above 23,000 ft, due to the bends. As a student, I personally experienced moderate hypoxia at 17,000 ft following an oxygen regulator failure, and the only thing that saved me was the instructor in the seat next to me because I wasn't experienced enough to recognize the early symptoms. I sure as hell know those symptoms now, but the first time you are in that situation it is unlikely you'll recognize the problem until it's far too late.
Our training shows us that even doing simple math (what is 5+7?) for more than 30 seconds above 23,000 ft without oxygen can be impossible. The first time I went through altitude chamber training, I thought I was doing fine for 2 minutes but after I went back on oxygen, the instructor showed me what I'd written... Apparently at 23,000 ft without oxygen, 2+2 equals four thousand plus some other meaningless squiggles