Actually, depending on how you have security set in Internet Explorer, there are sites that can quietly download spyware on your computer, without your knowledge.
The default security setting Internet Explorer uses leaves your computer wide open for little programs to plant files and read data from your computer.