I would guess that either your router or your ISP is doing some kind of QoS (Quality of Service) packet prioritization.

Trouble is that getting your ISP to admit to this is usually next to impossible. The low level flunkies have no idea, and those who do know will have certainly learned from Comcast's example and will keep their mouths shut.

This can also be caused by use of file sharing programs like bittorrent and kazaa. They have a remarkable talent for sucking up all available bandwidth if you don't know how to reign them in. Even if it's only running on one system, it can affect all others on your LAN. Though USUALLY if it's a file sharing program it's going to cause web pages to lag a bit too.