I prefer to render an intermediate file out of Sony Vegas, saving to Uncompressed AVI format. Then use Handbrake (
http://handbrake.fr) on the intermediate file to transcode it into an .MP4 for uploading.
WMV is pretty bad now-a-days, I used to use it for my WoW videos, and it doesn't hold a candle quality-wise. Its also bad to make YouTube do more transcoding than necessary, because that's how you lose quality. So to retain quality, give them the .MP4 format they ask for.
If you use Fraps, be sure to "Force lossless RGB capture" and go "full-size". I'd rather drop my FPS down as low as 15 before sacrificing on either of the above.
This technique will use a "lot" of hard drive space. Not really a lot any more IMO, since TB hard drives are fairly common now. But 50 to 500 GB should be anticipated to hold the Fraps files and intermediate files, depending on how much footage you take, the length of your video, etc. Once you have the final product made and uploaded and you're satisfied with it, you get all that space back because you can delete intermediate and Fraps files.
Here's a video that I found helpful:
www.youtube.com/watch?&v=rWMX5lSvEgY . I find it easier to just use Uncompressed AVI rather than DNxHD that they recommend for the intermediate file, but their way works as well, and is probably a little more forgiving if you don't have a lot of hard drive space available.
