We have recently published statistics comparing various website reputation services and have received good feedback over private channels regarding our article. In this sequel we add Microsoft’s Bing, malware filter along with comparison to other website reputation services.

At StopTheHacker.com (Jaal LLC) we have conducted tests of 721 URLs, all of which have been reported as malicious by volunteers of various blacklists. We follow a similar format for presentation of results as in the last post.

Website Reputation services: agree to disagree.

Note: All 721 domains/URLs, were reported as malicious, and were collected from malware.com.br on January 14, 2010. The blue column (maximum 100) indicates the percentage of sites that the website-reputation service correctly identified as unsafe. The orange column (maximum 100) indicates the percentage of sites that the website-reputation services incorrectly identified as safe.

The aim of the test:

1. Identify the accuracy of the website reputation service
2. Identify the overlap in terms of safe/unsafe websites

We present the most interesting results in this article. First we detail the parameters of the testing procedure to provide an idea of how the test was set up.

First, 721 URLs were collected from malware.com.br (mbr) on January 14, 2010. These URLs are reported for listing by one or more of the following: individuals, organizations, agencies and software products or services.  For the purposes of this test we assume that all the URLs obtained from the “regular” list on mbr are malicious and hence deemed “unsafe” to visit.

We compare the reputation provided by each website-reputation service and observe how many websites are marked unsafe, safe, untested, maybe-unsafe/caution/potentially-unsafe, and unreachable.

Website-reputation services tested:

• McAfee SiteAdvisor
• Norton Safe Web
• Google Safe Browsing
• Microsoft Bing
• Comodo SiteInspector

Note, that when analyzing a domainname/URL, for checking with the Google safebrowsing API, we have calculated the MD5 hash of the website name to match with the malware hash list. The date that we conducted this test was: January 15, 2010. The list of domain names tested are presented below and a graph representing the statistics for the 721 sites tested is above.

We identify the most interesting results below:

1. McAfee SiteAdvisor marked 36.75% of domains as Unsafe, 27.18% as Safe, 32.32% as Untested and 3.74% as Potentially-Unsafe.
2. Norton Safe Web marked 41.75% of domains as Unsafe, 45.49% as Safe, 4.3% as Untested and 8.32% as Potentially-Unsafe.
3. Google Safe Browsing marked 5.96% of domains as Unsafe, 94.04% as Safe.
   Note: The presence of the hash of the domain name  being tested, on the google malware hash list, is interpreted as “unsafe” while the absence is interpreted as “safe.”
4. Microsoft Bing marked 0.69% of domains as Unsafe, 34.26% as Safe, and 65.05% as Untested
5. Comodo SiteInspector marked 0.19% of domains as Unsafe, 95.82% as Safe, and 4.08% as Unreachable.

This follow-up experiment also shows that the variance between website reputation services that are currently being offered by large Internet-services/security companies continues to be very large indeed.

After discussions with representatives of the companies mentioned in this article, and getting a better idea of their behind the scenes methodologies. It seems that these website reputation services will continue to “agree to disagree.” We welcome their comments.

A note on differences between website reputation services:

Some of the services scan pages and some scan parts of a site. Some scan for potential “signs” of an infection, while others scan for the “postmortem” effect of an infection, such as an exploit being launched. Furthermore, the time difference between one of the services testing a web page or site versus when another one tests the same web page can also complicate issues. At StopTheHacker.com we recognize the current limitations of website reputation services that being offered by the industry.

In conclusion, while website reputation services have come a long way, they still have an even longer path to tread in order to become something that users should trust implicitly.

News, Report, Security bing, comodo, google, mcafee, norton, safebrowsing, safeweb, siteadvisor, website reputation

Websites on the Internet have now become the standard modus operandi for spreading malicious software to infect personal and corporate environments. A large number of benign and well-meaning websites are compromised everyday by hackers inserting malicious code to, in turn, infect the computers used by visitors to the hacked site. One of the ways to combat this is to develop a website reputation mechanism which can warn of potential threats before visiting a compromised site.

Website-reputation services vary wildly in their opinions.

Note that all 350 domains, were reported as malicious, and were collected from malware.com.br on December 18, 2009. The blue column (maximum 350) indicates the number of sites that the website-reputation service correctly identified reported bad sites. The orange column (maximum 350) indicates the number of sites that the website-reputation services incorrectly identified reported malicious sites as safe.

Website reputation services have been around for nearly 5-7 years now. Initially developing as a niche product line which could serve to provide an opinion of a site’s reputation to full fledged offerings which provide advisories about websites, whether they are distributing malware, and if they are, what kind, and using which Autonomous Systems.

At StopTheHacker.com (Jaal LLC) we have conducted tests with 350 domain names, all of which have been reported as malicious by volunteers of various blacklists.

The aim of the test is to:

1. Identify how accurate the website reputation services are
2. What is the overlap in terms of safe/unsafe websites

We have found some interesting results which we present in this article. First we detail the parameters of the testing procedure to provide an idea of how the test was set up.

350 URLs were collected from malware.com.br (mbr) on December 18, 2009. These URLs are reported to this website for listing by one or more of the following: individuals, organizations, agencies and software products or services.  We assume for the purposes of this test that all the URLs obtained from the “regular” list from mbr are malicious and hence deemed “unsafe” to visit.

We compare the reputation provided by each website-reputation service and observe how many websites are marked as unsafe, safe, untested, maybe-unsafe/caution/potentially-unsafe, unreachable.

• McAfee Siteadvisor
• Norton Safeweb
• Google SafeBrowsing
• Comodo Siteinspector

Note, that when analyzing a domain name, for checking with the Google safebrowsing API, we have had to calculate the MD5 hashes of the website names to match with the malware hash list. The date that we conducted this test was: December 21, 2009. The list of domain names tested are presented below and a graph representing the statistics for the first 350 sites tested is above.

We have identified some of the most interesting results below:

1. McAfee Siteadvisor marked 32.5% of Domains as Unsafe, 22% as Safe, 43% as Untested and 1.7% as Potentially-unsafe.
2. Norton Safeweb marked 50.86% of Domains as Unsafe, 43.71% as Safe, 2.29% as Untested and 3.14% as Potentially-unsafe.
3. Google SafeBrowsing marked 10.86% of Domains as Unsafe, 89.14% as Safe. Note: the presence of the hash of the domain name  being tested, on the google malware hash list, is interpreted as “unsafe” while the absence in interpreted as “safe”.
4. Comodo Siteinspector marked 0.29% of Domains as Unsafe, 98.86% as Safe and 0.86% as Unreachable. Note: after feedback from Comodo, a retest was conducted, accuracy changed from 0.29% -> 1.2%.

This limited test is a first step towards showing how much variance there is website reputation services that are currently being offered by large Internet-services/security companies. To highlight this point we present immediately below the relatively few domains (~6% of the total domains tested) that were marked as bad by all three major services, Norton, McAfee, and Google.

In brief:

• 6% of domains tested were marked as “unsafe” by all 3, McAfee, Norton and Google
• 10% of domains tested were marked as “unsafe” by Norton and Google
• 22% of domains tested were marked as “unsafe” by Norton and McAfee
• 5.7% of domains tested were marked as “unsafe” by Google and McAfee

Update: December 28, 2009

After receiving helpful feedback from representatives at Comodo, we were informed that Comodo’s service could provide more accurate answers if complete web page locations were checked instead of just the domain name. We followed the advice and saw a definite increase in Comodo’s accuracy. Comodo marked 1.2% of the website/pages as malicious. Prior to this re-test, the same service marked 0.2% of the websites as unsafe. The graph at the beginning of this article does not represent the results of this re-test.
Read more…

News, Report, Security comodo, google, mcafee, norton, safebrowsing, safeweb, siteadvisor, website reputation