Bots dominate small Web site traffic, research shows
Shared hosting providers can lose more than 50 percent of their capacity from parasitic drag generated by bot traffic, according to a new report.
A surprising 51 percent of traffic to an average Web site -- one with 50,000 to 100,000 monthly visitors -- is potentially bot generated, according to new research from Web security and performance company Incapsula.
On top of that, 31 percent of overall traffic to these such sites is malicious.
The news is worse for very small sites -- or those with fewer than 2,500 monthly visitors. Incapsula examined more than a thousand small sites and found that 83 percent of each site's traffic comes from non-human (bad bots and good bots) with bad bots accounting for 49 percent of traffic.
So what is the overhead represented by this automated bot traffic? Malicious bots will try to steal data, of course. But that's only some of the harm they pose to your Web site. Hosting providers are realizing that there is increased capacity overhead for each site deployed on any server because bots waste capacity, bandwidth, and power.
Most this automated traffic is fixed, completely unrelated to the Web site's genuine human traffic volume. In other words, each Web site spun up by a hosting provider will automatically get a set level of bot traffic no matter how many real visitors it attracts. As a result, the lower any Web site's genuine human traffic, the higher the Web site's bot traffic percentage is going to be. This rule is similar to a phenomenon seen in aerodynamics known as parasitic drag. This parasitic drag occurs when moving a solid object through a gaseous medium -- for example, an airplane wing's drag during flight.
Web hosting companies tend to offer small site owners shared hosting with unlimited bandwidth plans. Any small Web site owner can quickly set up a site at a fixed cost and not be charged according to their storage, bandwidth, or CPU consumption. This is financially possible for the hosting provider because they can cram hundreds of sites onto a single server and enjoy the economies of scale.
The paradox created is that the more Web sites the hosting provider adds to the server -- in an attempt to increase overall utilization and drive more revenue -- the more the server's utilization decreases due to the increase in bot traffic, or parasitic drag.
In aerodynamics, drag is determined by fancy equations that show how drag changes dynamically with speed. Not so when it comes to automated traffic. In this case, the drag is linear and proportionate to the number of sites on the server. Ironically, the drag may not be apparent to Web site owners who believe they have unlimited capacity.
Additional points from Incapsula's research:
- Shared hosting providers can lose over 50 percent of their server capacity.
- Small Web site owners' page load times are increased by more than 50 percent.
- 25 to 50 percent of Web site visitors abandon a site when load times exceed 4 seconds.
- SEO ranking are affected (Although impossible to quantify, parasitic drag will impact SEO rankings.)
The security community talks a lot about the cost of a data breach but this typically assumes that hackers only incur a cost when they're successful. This assumption misses much of the real damage and cost. In reality, nearly all Web sites -- large and small -- need to sustain about 30 percent more traffic from visitors who will never click the buy button.
This parasitic drag incurs a tangible cost in utilization and even an intangible cost when it comes to customer experience. Imagine, for example, the impact of removing 30 percent of cars on a heavily trafficked road. The commute becomes a lot more enjoyable.