A couple of weeks ago, I overclocked my GTX260 and after some tests everything looked fine. While playing AH (and during some video processing with Premiere Pro, I got the same exact thing - the screen "popped" and my system locked up and I had to hard-reset and reboot the PC.
I backed off the overclocking settings a little and everything has been rock solid since. So, I think Skuzzy nailed it...heat related. Could also be flaky video ram or GPU that gets wonky at higher temps.