How Safe are Internet Website Directories?
Recently, we told you that Dmoz.org, one of the largest user-edited directories on the Internet, is also one of the safest directories. Directories such as Dmoz.org contain links to hundreds of thousands to millions of sites. These directories are categorized by volunteers or through automated means. Many search engines, including Google, Hotbot and others, potentially use data from these directories. These directories are also used as efficient lookup services by thousands of web-surfers who want to locate sites which belong to a very specific category.
Given the important role that these directories play in the Internet, one would expect that they would make an attempt to point only to websites which are “safe.” By “safe,” we mean sites which have not been injected with malware, via code-injection attacks or other attack vectors.
We are not picking on Dmoz.org here. We were very impressed to see that none of the 2.8 million sites we profiled, were present on the Google Safe Browsing List. This could indicate that sites listed on Dmoz.org are concerned about their image, hence care about their visitors, and take appropriate precautions against malware.
To follow up on our previous article, we have further analyzed 10,000 sites, randomly chosen from the Dmoz.org corpus of nearly 2.8 million websites. Each of the 10,000 sites was tested against each of the below website reputation services.
Note: When analyzing a domain-name or URL, for verification with the Google Safe Browsing List, we have calculated the hash of the website name to match against the list. The test was conducted between January 19th and January 21st, 2010. The list of domain names tested are presented at the end of this article.
We identify the most interesting results below:
- McAfee SiteAdvisor marked 0.39% of domains as Unsafe, 84.23% as Safe, 15.08% as Untested and 0.3% as Potentially-Unsafe.
- Norton Safe Web marked 0.39% of domains as Unsafe, 59.02% as Safe, 39.79% as Untested and 0.8% as Potentially-Unsafe.
- Google Safe Browsing marked 0.02% of domains as Unsafe, 99.98% as Safe.
Note: The presence of the hash of the domain name being tested, on the Google Safe Browsing List, is interpreted as “Unsafe” while its absence is interpreted as “Safe.” - Microsoft Bing marked 0.06% of domains as Unsafe, 93.2% as Safe, and 6.74% as Untested.
- Comodo Site Inspector marked 0.08% of domains as Unsafe, 99.46% as Safe, and 0.44% as Unreachable.
Note: We were only able to test the first 5000 URLs with Comodo Site Inspector.
McAffee SiteAdvisor and Norton SafeWeb seem to detect nearly 19 times more websites as “Unsafe to Visit” than Google, and nearly 6 times more websites as “Unsafe to Visit” than Bing. It is interesting to note that it is an order of magnitude difference in the number of websites marked as “Unsafe to Visit” by these competing services.
We would like to know how long McAfee, Norton or Bing cache results for a particular site. Google allows webmasters to request reviews when they believe the site has been disinfected, and Comodo’s service seems to be an On-Demand service. This makes an interesting place to start for a future experiment. Further, it would be interesting to see whether sites listed on Yahoo the Directory and other directories are classified by these services.
Read more…
