Too late 
Not totally: Even if you're using another browser than IE you should keep IE fully patched. Why, if you're not going to use it? Because IE affects to Windows' networking components. IE is behind Messenger and other Live thingies, and I guess it also works with Windows Update. It can't be fully uninstalled.
So, if you're not going to use it, put the security settings on High. Tips for safer browser settings are i.e. on this
Cert site. For a browser that is not in use the rules can be set even tighter.