Please give some details about your system, video card, OS, drivers, etc. That may help to identify the problem.
What video card are you running, and what resolution and color depth did you select? Are you runnind DX8? What video driver version are you running?
Also, if you are running Roger Wilco, try disabling it, removing it, or at least set it to use WAV recording if using DX8.