sounds to me like you need a new system anyhow

essentially the film making will run as smooth as your game does.
using fraps is good, always remember to capture in the film window, not fullscreen. not totaly sure if this helps but it certainly makes better files on my machine.
and with fraps check the box to ignore sound.
recording the lame film viewer sounds will only take longer, take more system power, and produce lower fps in captured files.
thats all i know.