I do all my work with 852x480@24fps (my personal preference)
I record at 1360x768, then use Vegas to edit footage and output to 852x480.
Here are my settings:
I set fraps to custom fps of 24 and "Full-size" (Resolution depends on the source of recording)
In Vegas, go to File, then Properties: I use a custom resolution and frame rate, again @ 24fps
When I perform the "Render As" option under File in Vegas, here are my custom settings:
I use CBR 2-pass for BEST image results. This is a constant bit rate, so the image file is determined by duration and Mega-bits per second setting. I set it for 2.5Mbps, but this number is just my personal preference based on final image size, fps, resolution, whether video has lots of fast moving images and finally my personal opinion on final output.
Play with this number to determine image quality and file size. The higher the number, the bigger the file and better the image quality. The 2-pass option is better than single pass, but takes twice as long to render. 2-pass will improve your image quality tremendously without the need to further increase bit rate/file size.
Youtube and other video hosting sites can reduce image quality significantly if you do not upload in the specified resolution/fps/filesize parameters. It has to do with their encoding software. Each time you encode your work, image quality will be reduced.