Go down to 512 on your textures and knock you AA down to x4 (my best guess).
Here is a copy of Skuzzy's Video Memory optimization formula. I tried it. Was getting ok frame rates, but would get drops and hesitations I couldnt figure out. Skuzzy set me straight, and I have had really good frame rates and stable video ever since.
At the time I was running 1024x768 resolution, 1024 max textures, 4x AA. If your number you come up with is higher than the amount of video memory you have, you will be swapping in and out of memory, which eats up the card's resources. You make the card work harder, you get drops in frame rates and stutters under a load. You can see by the results, if I had a video card with 256MB of RAM I would have been fine. My card only has 128MB. I had to adjust my settings. I havent regretted it since.
Here is a simple formula Star. It is about the amount of video ram needed for any given frame. This is an approximation.
Drawn_Frame_Buffer = (Horizontal Resolution * Vertical Resolution) * 4;
Working_Frame_Buffers = Drawn_Frame_Buffer * 2;
Total_Frame_Buffer_Space = Drawn_Frame_Buffer + (Drawn_Frame_Buffer * AA level);
Aces_High_Texture_Space = (Maximum Texture Size / 5) * 1MB
----
Now, add it all up.
Approximate Video Memory Required = Total_Frame_Buffer_Space + Aces_High_Texture_Space;
(1024 * 768) * 4 = 3.07MB;
3.07MB * 2 = 6.14MB;
3.07MB + (3.07MB * 4) = 15.35MB
(1024 / 5) * 1MB = 204.8MB
220.15MB
Now this is a rough estimate, but it gives you an idea of the amount of video ram needed for any given scenario.
This also does not take into account the amount of scratch pad video ram required by the video card and drivers. You can toss in another 4 to 8MB for that, depending on the video card.
In a best case scenario, you want that total number to be less than the total amount of video ram you have available.