One thing you may want to check is:
If you running in 1024 x 768 in game, do you have your desktop settings in 1024 x 768? make sure the desktop size matches whatever size you request in the game.
Try lowering the in-game textures to 256..if that works, then bump em up to where you can get away with it.
Not sure what you mean by weird graphics.. may also want to increase your Hz on the monitor (assuming you have it on the default 60 Hz) but ONLY do this if your monitor AND video card allow the configuration. Can damage things if you don't know what to do, or what not to do.
Not sure by reading your post, but it seems you might be saying that game runs fine in 600 x 480 mode?
But again, I really do believe that your biggest problem is that your old system cannot keep up with the speed of that new card you got.