An easy trick. Once you know it, you won't forget it.
Use the arrow-keys (down and up) to go to the address you want to delete. Use the delete-key to delete it. That's all.
I have Office 2003 and want to delete my old e-mail address from the drop-down that opens in Outlook each time I type the first few letters of my current e-mail address (it begins with the same letters as the old one).
I have removed the old one from Outlook Contacts already. Outlook sometime shows the new one in the drop-down as well as the old but not always???