In recent years, the number of blacklist being published by web-centric organizations have grown by leaps and bounds. Large Internet based companies such as Google, Yahoo and Microsoft have been providing cues to their users about malicious websites in trying to make the Internet a safer place. Google provides much more in-depth information than the other two, Yahoo and Bing, and seems to have sophisticated virtual machine based analysis tools which can detect misbehaving malicious code. Yahoo employs McAfee’s Search scan service while Bing potentially uses Microsoft specific technologies.

Experiment Goal

The aim of this experiment is to compare the coverage for each of the blacklists published by Google, Yahoo and Bing and compare them to what users in the Internet believe. To do this we will compare the results of Google, Yahoo, Bing and Malware Patrol with Web of Trust (WOT). Furthermore, we have also tried to see how many of these malicious URLs are also involved in Phishing. We have done this by looking up each URL/domain via Phishtank’s API.

Blacklists provide an easy mechanism for users (via browsers) and developers (via APIs) to assimilate security information about websites, IPs and such in order to make an informed decision about whether to allow or deny access to an IP or website.

Methodology

We have collected 1095 confirmed malicious links from MalwareURL. Each of these links was tested to determine if they are listed on blacklists supplied by Google, Yahoo and Bing. Note that Yahoo and Bing unlike Google do not provide any direct APIs to probe their databases. Thereby each link, and its associated domain was pushed via an HTTP request to Yahoo and Bing to analyze if the results indicated that the domain/link was infected.

To determine if a website is present in the Google malware blacklist, the domain name along with the link and its variations, as defined here, are converted to MD5 hashes and checked using Google’s Safe Browsing API. For Malware Patrol, the aggressive version of their blacklist is downloaded and comparisons are made locally. For WOT, we employ their XML based API to gather information about the belief of users in the Internet. For Phishtank we have used their XML based API. The tests were conducted on Mar 22 2010.

Popular blacklists cover only a minuscule percentage of malicious sites.

Highlights

• Google marked 0.18% of the URLs as unsafe.
• Yahoo marked 1.0% of the URLs as unsafe.
• Bing marked 0.09% of the URLs as unsafe.
• Malware Patrol marked 0.63% of the URLs as unsafe.
• Phishtank marked 0% of the URLs as unsafe.
• WOT marked 99% of URLs as unsafe.

Note: 1095 unique, malicious URLs were tested with each service.

Observations

Interestingly, Web Of Trust (WOT) marked 99% of the URLs with “poor” or “very poor” or “unsatisfactory” reputation. We have to assume that when users will see such a rating they will not visit the website in question and hence treat this kind of rating as unsafe, for the purposes of this test. It remains to be determined if WOT uses a data feed from a malware URL which we have used to prime the test set. Nonetheless, it is surprising to see that a company which specializes in collating the trust and opinions of web surfers performs better orders of magnitude than large Internet companies and established blacklist providers.

One must keep in mind though that Google’s approach to maintaining an ever changing blacklist is slightly different from the other actors in the game. Google publishes an updated version of its list every 30 minutes or so and specifies which MD5 hashes need to be purged and which ones need to be inserted. Some blacklist services do not take this approach and hence may claim to store information on millions of sites, which were infected at one point in time. The probability of this happening in the Google blacklist is low, because they have opened up a review process via their webmaster central area to update their blacklist.

In contrast, Bing and Yahoo do not provide public APIs for developers and applications to use.

Also, we see that none of the URL/domains were actually listed on Phishtank. It seems that websites which aim to infect users with malware are quite different from the set of sites used for phishing. It does not seem that malware laced websites are also used to commit phishing.

Conclusion

Large Internet companies, some of whom have published effective blacklists, used by many developers and application all over the world, still have a long way to go in order to become truly effective. As we have seen, only minuscule numbers of malicious websites are identified by the blacklist services. WOT seems to be extremely effective at identifying unsafe websites. It remains to be determined whether the data-set used for this test has a large overlap with any of the sources WOT uses to classify websites.

Another interesting result is that it does not seem that websites which aim to infect users with malware are actively involved in phishing campaigns.

Unfortunately, the idea of page caching has not been implemented well. In fact, page caching has opened up new opportunities for malware. The primary problem being that, from a security perspective, when search engines cache copies of websites, they are storing any malware that is present on the site on their own infrastructure as well.

Hackers Exploit Search Engine Page Caches

Most large search engines use some kind of malware analysis to determine if a website is compromised or not. Google for example, has a well tuned system with high accuracy. In our meeting with the Google malware team, some months ago, we were glad to find that they were already aware of this problem. In the weeks following our interaction, cached copies of infected websites were no longer easily available via searches.

Not so long ago, we wrote an article about our efforts to alert Yahoo of the presence of malware in the cached versions of various web pages served up by their search engine. Our efforts were not successful, although the occurrence of malware in Yahoo cached pages seems to have gone down significantly. Perhaps our messages were not entirely ignored.

Recently, an article came up on ISC SANS discussing this very same issue.

Recently, we have found instances of Bing serving up malware in their cached pages. It seems that Bing’s malware detection methods are not able to reliably detect malware on cached web pages. This keeps Bing from securing cached pages which contain malware for its users. We have provided screen shots below as an example of the issue. In this particular case, the strain of malware found in Bing cached pages has been around since 2009.

Search Engines Ignore the Problem

Consider the case where a malicious individual deliberately infects a website with malware and Bing (or another search engine) indexes it. The malicious individual can then send out hyperlinks pointing to the cached web pages hosted by Bing. Any kind of “reputation-checking” for the cached link will confirm that the page is hosted by a reputable company, in this case, Bing (Microsoft). However, the malware will still be able to deliver its payload. Just in case you’re thinking, “my antivirus will protect me from the malware on the cached page,” you may like to read this article.

It is surprising to see that search engines like Bing, which claim to implement malware detection, cannot correctly determine if a cached copy of a web page hosts malware! In these cases, Bing ends up an excellent attack vector for malicious individual.

It remains to be seen if search engine companies will continue to serve up cached pages laced with malware at the same time as they are touting active scan and detection mechanisms. Let’s hope this article can get attention in the upper echelons of management at these large search giants and they start to pay attention to this problem.

Screen shots follow below:

Yahoo, has allowed users, for several years, to use the “cached pages” options displayed along with its search results on Yahoo-Search. Yahoo has partnered with McAfee’s SearchScan to provide safer searches since about May 2008. This is all good. The intention of providing safer searches to visitors is very noble. Google too, has led the pack in this direction by opening up its SafeBrowsing API and by providing visual warnings in search results boldly claiming “Warning visiting this website may harm your computer”.

Stopthehacker.com  has tried to communicate with executives at Yahoo since April 2009 about the potential problems that we have been observing in their cached pages. This has not been met with any real response.

The problem is simple, but very important. Cached versions of web pages displayed on Yahoo Search often contain malware code embedded in them. This is a phenomenon that we have observed repeatedly.

Consider one of our many attempts at communicating this issue to Yahoo (message shortened for brevity).

We have found that Yahoo’s cache results, even with SearchScan on, do not detect the presence of malware on its cached copies of webpages. I have attached some screen shots which prove the point.

Our scanners flagged the code in the cached copies right away. The site in question, for which I looked up Yahoo’s cache is http://www.xxxxxxxx.com

More info on our response to this site is available at http://xxxxxxxxxx.xxx/**stripped**

The screen shots attached with this post show an example of a website which was scraped by Yahoo’s spider, indexed and cached and then when accessed via its search results, pops up the malware code. There does not seem to be any kind of sanitization/scrubbing process going on in the background.

Worryingly, this problem gives rise to a very effective attack vector, where a malicious individual can compromise a site or even simply create a site that contains malicious code. Once the site is crawled by Yahoo’s spider, and is loaded in the cache, the link to this cached page becomes an excellent attack vector to use for social engineering, as it carries the sense of security that comes with Yahoo’s brand name. No need to exploit XSS/CSRF, no back-breaking hours of toil and sweat need to be put in discovering flaws in a site. Just get the infected pages cached in Yahoo! and voila, you have a live exploit launched from official Yahoo property.

Consider the fact that Yahoo search has 18% of the search market in October 2009, the number of visitors to the site is non-trivial! Moreover, Yahoo’s brand image can suffer, if this phenomenon becomes more wide spread or well-known.

Given my failed efforts to discuss this with Yahoo, at this point, I can only hope that this does not become more popular.

I cannot understand how Yahoo is employing SearchScan technology to provide safer search results to visitors, yet fails at the back-end to identify cached pages loaded with malware.

Till next time.