Turn it off since it's an intrusion point. For instance we can discuss many ways for this to be exploited. The speed is usually due to HTML emails that need to go get graphics from a web site. This is also one of the exploits that leak information back to the email sender that "hey, that's a live email address" and will result in more spam.
I have a user that has extremely slow response when clicking an email message in Outlook 2003.
If the message is 10-60kb it's almost instant as most users, but if there is any kind of attachement, he clicks on the the email and it takes Outlook some time to show the message.
The system retrieves email from Exchange server and he's been setup the same way as the other users.