My guess is you are overloading the video card's memory. You've probably been playing with the right settings, just not in the right combinations.
Also, I'm not sure how much my advice could help since the only ATI card I ever used was an old Rage Pro 10 years ago. Still, your system should be kickin mine's rear and its not, so maybe something I got can help you.
First of all, basics. Driver install/uninstall. Uninstall your current driver. Completely. Reboot and let Windows detect the card as a generic VGA Compatible. Install your new drivers for the card. Reboot again. Reinstall Direct X just to be safe. Run the Direct X Diagnostic and make sure everything is working properly. I'd use the 4.12 drivers, because I know those are safe. If you want to experiment with newer ones be my guest. Just follow the directions above every time you change drivers to prevent Windows from getting confused.
Go into your BIOS at startup and make sure your AGP Aperture setting is high enough (it normally is by default, and you have a GB of RAM; I doubt this is an issue but it doesnt hurt to check). With a GB of RAM you should be able to set it as high as 512MB. Every system is different on what it likes. They have run so many tests and benchmarks, and no one can come to a really good decision. Best guess for most modern systems is 128MB, but again individual systems seem to sometimes want more or less. Play with it and see what happens. It may be a moot point, you may not even be able to adjust it. Worth a shot though.
Now that you can be sure of your hardware interface being stable, time to play with software. As I said, I'm not real familiar with the ATI interface, so instead of possibly putting my foot in my mouth, I'm going to provide you with some links I have used in the past that helped me setting mine up (most are useful for either ATI or Nvidia).
Nice little tweak utility
http://guru3d.com/rivatuner/Driver cleaner to prevent any leftovers from gumming up your fresh installs
http://driverheaven.net/cleaner/The best guides I have ever seen on getting the most out of your stuff are here at Tweakguides.com. I linked directly to the ATI Catalyst tweak guide. Enjoy.
http://www.tweakguides.com/ATICAT_1.html