A P3 500 is not fast enough to warrant dumping money into a GF 3, that's for sure. You are entirely CPU bound on that system.
Adding that much ram to Windows 98 (or 95 or ME) will make no difference. In fact 768 MB is the absolute limit that Windows 98 and ME can handle without tweaking system files to prevent, of all things, an "Out of Memory" error.
That frame rate sounds about right with those system specs. The game *should* be very smooth for you as long as you aren't in dense smoke areas. (My secondary system is a P3 450 with a Tnt 2 Ultra video card and it gets around 35 fps and is very playable.)
Modem speed makes no difference in framerate.