I've bumped into that as well in the past, the only way around it I've found is simple trial and error.  Best to do this with a short clip so you don't waste a bunch of time waiting for rendering.  Just play around with the different codecs and settings till you find one that works.  FRAPS is an option as well.