I only ever have problems in the film viewer displaying sprites/graphics for the first time. If I play through, get the stutter, then rewind and play again it's no longer there (as long as you don't hit stop).
I only get these massive stutters after recording 3-5 fraps clips (unregistered version) and usually it gets progressively worse. I can then close fraps, close the film viewer, restart both, go back to the exact same spot in the film, and record smooth as silk.
Considering this never happens when running the film viewer alone, and that the stutters get progressively worse the more fraps clips I record in a row, I am drawn to the conclusion that it is mostly fraps creating the problem (but also has to do with fraps and ahfilm interaction, perhaps both want/need the same resource?)