Spikes in the traffic reports
We noticed two unusual trends in traffic to our customer sites over recent months. One was a surprising increase in traffic generally on some sites, and the other was an increase in the proportions of visitors using Internet Explorer 6 and a corresponding decrease in popularity of IE7. We investigated further.
There are two factors accounting for this apparent resurgence of Internet Explorer 6, and neither of them are desirable.
The first culprit we identified was webform spam. I have written before about spammers who use automated robots which masquerade as humans and post spam messages into the forms they find on websites. This continues to be a problem and we continue to develop defences against this type of spam, blocking anything up to 500 spams attempts per day. Our investigations show that this spam usually disguises itself as a user of Internet Explorer 6 and, in some cases, certainly accounts for the huge jump in traffic to some of our smaller websites.
The second culprit was more surprising. It was, we believe, the anti-virus software, AVG. I have long been a fan of AVG but the latest version, AVG8, included a link scanner feature which has caused headaches for webmasters. On an AVG8-protected machine, when you searched Google, AVG scanned the results so that it could warn you which sites might contain malware, to discourage you from clicking on them. It did this by downloading every page listed on the Google results screen and scanning each one, whether you clicked on any of those links or not. And of course, normally you DON'T click on most of the links.
As you can imagine, this approach not only slowed down the user's PC, it also created a lot of redundant network traffic for website owners, especially on sites such as ours which are well-optimised for search engines and frequently appear in the first few pages of results. These AVG scans were disguised to look like a human user, using Internet Explorer 6, and so were impossible for us to detect and block. With tens of millions of people using AVG, this quickly became a significant problem, not only because of the additional load it was placing on web servers but also because it makes it so much harder to draw any meaningful statistics from traffic log analysis.
Webmasters worldwide were vocal in objecting to this ill-conceived feature. I'm pleased to report that after much online debate and criticism, AVG has thankfully withdrawn the link-scanner component, so hopefully traffic figures will return to a more normal pattern once the AVG updates reach the users.
18th July 2008
This article comes from the SKILLZONE email newsletter, published monthly since January 2008, and covering topics related to technology and the internet. All articles and artwork in the SKILLZONE newsletter are orignal content.