I've bumped into that as well in the past, the only way around it I've found is simple trial and error. Best to do this with a short clip so you don't waste a bunch of time waiting for rendering. Just play around with the different codecs and settings till you find one that works. FRAPS is an option as well.