Cable internet speeds are affected by a wide variety of factors. One of the primary factors is that cable is a shared medium, i.e., you are sharing the 20 Mbs with everyone else who is on your segment of cable. Cable runs are shared by multiple end points, the number depending on population density and the capabilities of the head-end hardware. What this means is that the more end points that are active at the same time reduces your slice of the bandwidth that you get. The heaviest internet usage in the U.S. is usually in the evening hours. At one time, the rest of the world could tell when Americans got off work and got on the internet, because their bandwidth would suddenly start decreasing (at one time, all internet traffic went through the U.S. internet backbone. This is no longer the case.)
Other factors include the number of hops your traffic has and the throughput speed of the hops your internet traffic takes. Each hop potentially has different speed bandwidth and utilization. Some of the hops may have a lot of bandwidth but also be heavily utilized, while other hops may not have a lot of utilization, but not have as much bandwidth. The ISP's, regional NAP's, and backbone providers use routing protocols and network design to try to balance the utilization of internet segments as much as possible.
Another factor that has been alluded to above is the bursty nature of internet traffic. Network utilization is never steady but varies widely at any moment in time. It is easy to see when you are downloading a large file. If your download manager tracks the download speed, you will see the speed fluctuate, sometimes by large amounts. There is nothing that can change that.
One final factor is the server/s that you are connected to and how many people are on them at the same time. Most sites have multiple servers running in multiple locations and/or use Akamai to insure that all users get good performance at all times. However, there are times and circumstances where a particular site's servers can get overloaded and response times and download speeds go in the toilet (Healthcare.gov anyone?). All you can do then is just wait until the problem is fixed.

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic