Websites on the Internet have now become the standard modus operandi for spreading malicious software to infect personal and corporate environments. A large number of benign and well-meaning websites are compromised everyday by hackers inserting malicious code to, in turn, infect the computers used by visitors to the hacked site. One of the ways to combat this is to develop a website reputation mechanism which can warn of potential threats before visiting a compromised site.

Website-reputation services vary wildly in their opinions.

Note that all 350 domains, were reported as malicious, and were collected from malware.com.br on December 18, 2009. The blue column (maximum 350) indicates the number of sites that the website-reputation service correctly identified reported bad sites. The orange column (maximum 350) indicates the number of sites that the website-reputation services incorrectly identified reported malicious sites as safe.

Website reputation services have been around for nearly 5-7 years now. Initially developing as a niche product line which could serve to provide an opinion of a site’s reputation to full fledged offerings which provide advisories about websites, whether they are distributing malware, and if they are, what kind, and using which Autonomous Systems.

At StopTheHacker.com (Jaal LLC) we have conducted tests with 350 domain names, all of which have been reported as malicious by volunteers of various blacklists.

The aim of the test is to:

1. Identify how accurate the website reputation services are
2. What is the overlap in terms of safe/unsafe websites

We have found some interesting results which we present in this article. First we detail the parameters of the testing procedure to provide an idea of how the test was set up.

350 URLs were collected from malware.com.br (mbr) on December 18, 2009. These URLs are reported to this website for listing by one or more of the following: individuals, organizations, agencies and software products or services.  We assume for the purposes of this test that all the URLs obtained from the “regular” list from mbr are malicious and hence deemed “unsafe” to visit.

We compare the reputation provided by each website-reputation service and observe how many websites are marked as unsafe, safe, untested, maybe-unsafe/caution/potentially-unsafe, unreachable.

• McAfee Siteadvisor
• Norton Safeweb
• Google SafeBrowsing
• Comodo Siteinspector

Note, that when analyzing a domain name, for checking with the Google safebrowsing API, we have had to calculate the MD5 hashes of the website names to match with the malware hash list. The date that we conducted this test was: December 21, 2009. The list of domain names tested are presented below and a graph representing the statistics for the first 350 sites tested is above.

We have identified some of the most interesting results below:

1. McAfee Siteadvisor marked 32.5% of Domains as Unsafe, 22% as Safe, 43% as Untested and 1.7% as Potentially-unsafe.
2. Norton Safeweb marked 50.86% of Domains as Unsafe, 43.71% as Safe, 2.29% as Untested and 3.14% as Potentially-unsafe.
3. Google SafeBrowsing marked 10.86% of Domains as Unsafe, 89.14% as Safe. Note: the presence of the hash of the domain name  being tested, on the google malware hash list, is interpreted as “unsafe” while the absence in interpreted as “safe”.
4. Comodo Siteinspector marked 0.29% of Domains as Unsafe, 98.86% as Safe and 0.86% as Unreachable. Note: after feedback from Comodo, a retest was conducted, accuracy changed from 0.29% -> 1.2%.

This limited test is a first step towards showing how much variance there is website reputation services that are currently being offered by large Internet-services/security companies. To highlight this point we present immediately below the relatively few domains (~6% of the total domains tested) that were marked as bad by all three major services, Norton, McAfee, and Google.

In brief:

• 6% of domains tested were marked as “unsafe” by all 3, McAfee, Norton and Google
• 10% of domains tested were marked as “unsafe” by Norton and Google
• 22% of domains tested were marked as “unsafe” by Norton and McAfee
• 5.7% of domains tested were marked as “unsafe” by Google and McAfee

Update: December 28, 2009

After receiving helpful feedback from representatives at Comodo, we were informed that Comodo’s service could provide more accurate answers if complete web page locations were checked instead of just the domain name. We followed the advice and saw a definite increase in Comodo’s accuracy. Comodo marked 1.2% of the website/pages as malicious. Prior to this re-test, the same service marked 0.2% of the websites as unsafe. The graph at the beginning of this article does not represent the results of this re-test.

Below we list the websites from which we extract the statistics above

Websites marked as “unsafe” by Norton, McAfee and Google

219.148.34.10
219.148.34.9
4gameranking.com
77.245.61.232
aiongamemeca.com
durantilumi1cao.com.br
golary.cn
hagnuor.cn
igivor.cn
igoudix.cn
igouhxe.cn
ihaegup.cn
ihaerxi.cn
ihagoin.cn
ihoekag.cn
ihouvi.cn
ihuere.cn
ihuqoyr.cn
ijaheuw.cn
ikyigy.cn
iloefe.cn

Websites marked as “unsafe” by Google and Norton

212.99.87.130
219.148.34.10
219.148.34.9
4gameranking.com
61.164.108.213
77.245.61.232
aimeblog.com
aiongamemeca.com
bhactuant.com
durantilumi1cao.com.br
findreaso1ble.org
for23.3322.org
golary.cn
gyfvuxe.cn
hagnuor.cn
ifueme.cn
igivor.cn
igoudix.cn
igouhxe.cn
iguyzmo.cn
ihaegup.cn
ihaerxi.cn
ihagoin.cn
ihoekag.cn
ihogedi.cn
ihouvi.cn
ihuere.cn
ihuqoyr.cn
ijaheuw.cn
ijakony.cn
ijazofy.cn
ijeife.cn
ijelodi.cn
ikyigy.cn
iloefe.cn

Websites marked as “unsafe” by McAfee and Google

219.148.34.10
219.148.34.9
4gameranking.com
77.245.61.232
aiongamemeca.com
durantilumi1cao.com.br
emes.com.br
golary.cn
hagnuor.cn
igivor.cn
igoudix.cn
igouhxe.cn
ihaegup.cn
ihaerxi.cn
ihagoin.cn
ihoekag.cn
ihouvi.cn
ihuere.cn
ihuqoyr.cn
ijaheuw.cn
ikyigy.cn
iloefe.cn

Websites marked as “unsafe” by McAfee and Norton

163.fuckunion.com
206.161.127.72
208.75.230.43
209.205.196.16
218.93.205.250
219.148.34.10
219.148.34.9
4gameranking.com
61.235.117.72
70.148.212.252
77.245.61.232
82.98.235.173
85.92.157.141
91.213.126.100
97feihu.com
adobeflashupdates.com
adwareprotectionsite.com
aiongamemeca.com
amforum.lua.pl
antivirus-live.com
artistinove.it
centralspa.ca
comerciocentral.net
densmail.com
diadoamigo0.myartsonline.com
dimorphothec.com
dl.get-torrent.com
dl.targetsaver.com
dudi11.off.co.il
durantilumi1cao.com.br
ebestsite.co.kr
elogios0.myartsonline.com
exeype.cn
fuck-celebrities-movie.com
gclass.it
generalantivirus.com
ghterwa.com
gokzed.cn
golary.cn
google.netcdn.com
gorazyn.cn
hagnuor.cn
hahdyti.cn
hgtr3.com
hiqtacy.cn
hjyuw2.com
icepot.cn
idoafy.cn
idoape.cn
igafep.cn
igakuot.cn
igeuvat.cn
igivor.cn
igoudix.cn
igouhxe.cn
igycoat.cn
ihaegup.cn
ihaerxi.cn
ihagoin.cn
ihoekag.cn
ihouvi.cn
ihuere.cn
ihuqoyr.cn
ijaheuw.cn
ijepiyq.cn
ijesiam.cn
ijobuaw.cn
ijuebka.cn
ikoiwe.cn
ikorate.cn
ikuaxge.cn
ikyadeh.cn
ikyigy.cn
ileufby.cn
ilixyeq.cn
ilodux.cn
iloefe.cn
iluefot.cn
i1gyve.cn

Interestingly, Comodo’s service marked only 1 website, 218.146.255.156 as malicious. This domain was also marked malicious by Norton, “Untested” by McAfee and was not found on the Google malware hash list. Below follows the complete list of domains that were tested.

Complete list of domains tested

001.bbexe.cn
113.105.175.138
114.207.112.169
119.147.114.163
12.10.157.6
12.24.238.229
12.25.151.68
121.12.127.230
121.205.91.142
121.205.91.145
123.244.30.118
123.244.30.66
123.bbexe.cn
147.163.1.77
148.208.196.2
163.fuckunion.com
174.36.233.59
192.220.110.228
193.104.27.139
193.169.234.27
200.111.155.122
200.242.43.250
200.63.5.78
200.67.103.187
200.69.124.17
202.105.183.104
202.114.181.5
204.12.43.43
204.232.131.12
206.161.127.72
208.75.230.43
209.131.200.246
209.172.35.144
209.205.196.16
209.43.123.143
210.166.220.240
210.206.8.254
210.51.166.217
211.39.130.196
211.78.87.42
212.31.234.155
212.63.132.215
212.88.178.22
212.97.63.156
212.99.87.130
216.24.165.4
216.240.148.175
217.116.46.139
218.146.255.156
218.16.120.253
218.188.0.5
218.6.15.135
218.63.200.196
218.86.118.98
218.93.202.115
218.93.205.250
219.146.128.242
219.146.128.245
219.148.34.10
219.148.34.9
220.90.213.158
220.95.232.28
221.1.204.243
221.143.43.200
222.66.209.98
222.76.243.53
24.1188d.cn
24.65.70.52
3.1188d.cn
3310.net.cn
38.99.91.47
3s.8i9i.com
46.1188d.cn
46.3388a.cn
4gameranking.com
5.1188d.cn
53.1188d.cn
58.147.27.69
58.215.79.176
6.1188d.cn
60.191.39.6
61.108.173.3
61.110.21.192
61.164.108.213
61.235.117.72
62.193.229.83
64.160.216.20
65.109.240.130
65.183.178.92
66.116.229.233
66.152.93.119
66.220.17.157
66.45.235.228
67.19.9.234
67.43.224.77
68.153.57.9
70.148.212.252
72.10.166.195
72.20.6.106
72.237.212.57
72.35.84.6
72.64.146.16
731273265.520815.com
76.162.68.70
76.73.42.43
77.245.61.232
77.92.158.122
78.159.127.254
78.46.151.179
80.153.182.80
81.223.40.244
81.252.31.148
82.114.87.46
82.98.235.173
83.103.59.84
83.206.113.161
83.240.174.136
83.245.62.87
84.20.251.223
85.17.136.139
85.25.81.140
85.92.157.141
91.207.7.116
91.213.126.100
93.174.95.140
95.211.98.136
97feihu.com
98.126.34.250
a.amg777.com
a1964.g.akamai.net
absi2008.netfirms.com
acripino7878.110mb.com
admin.bbexe.cn
adobeflashupdates.com
adwareprotectionsite.com
aha-autoimage.com
aimeblog.com
aiongamemeca.com
album.pagi1s.sapo.pt
alison.wz.cz
alkeichah.com
amforum.lua.pl
amoravela.com.sapo.pt
antivirus-live.com
antivirusadvanced.com
arathas.de
arcade.ya.com
arkbroadcasters.org
artdeli.co.kr
artistinove.it
atencaousuario.webcindario.com
atualizaca-juridica.sitesled.com
ausamedia.berepublic.com
avr-download.com
b.amg777.com
backstaroup.home.sapo.pt
bb.bbexe.cn
bbs.pxtang.cn
bcfpb.com
bchokies.com
bdesata.com
belezademulher.org
best-sale.us
bevaccine.com
bgcomstock.com
bhactuant.com
blog20fc2.com
blogaofotos8.com.sapo.pt
blogfotos2008.com.sapo.pt
blogpesoalpessoal.com.sapo.pt
bmz.horizon.net.pl
brasilterra.com.sapo.pt
c.amg777.com
caixa-cefinstall.sitesled.com
caixaeconomica-gov.sitesled.com
cancelamentt0.googlepages.com
carbys.no.sapo.pt
card2009.com.sapo.pt
cardamorhtml.no.sapo.pt
cardpaixao.esmartdesign.com
cartao8578.com.sapo.pt
cartaoamizade000.com.sapo.pt
cartaoespecial9.com.sapo.pt
cartaovirtual2006.no.sapo.pt
cartoesnovos.250x.com
cartoesuol.com.sapo.pt
cartoeswebapaxo1do.no.sapo.pt
casasbahia.com.sapo.pt
cau.ac.kr
centralspa.ca
chaiyapruekpethospital.com
chamadavideo-1.my3gb.com
chi1oilfactory.cn
chinesefreewebs.com
ciduninstall.com
cinema-film-4you.ru
club.telepolis.com
comerciocentral.net
comunidade777.110mb.com
config.koreamessenger.com
correiosweb.com.sapo.pt
cprzafra.juntaextremadura.net
d.amg777.com
d.kkkmfdy.com
d4.kkkmfdy.com
damnkt.logi1pp.com
db.ms.kr
denizlisurucukursu.com.tr
densmail.com
diadoamigo0.myartsonline.com
dimorphothec.com
di1r-cs.real-host.ru
dindindopv.bravehost.com
ditto.arpa.org
dl.get-torrent.com
dl.qvodir.cn
dl.targetsaver.com
dl.woyo8g.com
dl02.softdown-load.com.cn
dollardream.ru
donghae.ms.kr
dorota288.w8w.pl
down.1vysoft.org
down.woyo8g.com
down.yellowsoft.org
download.gameztar.com
download.iobit.com
download.leeboo.com
download.softpedia.com
downlopaginvisualiz.com.sapo.pt
dtvprosoft.hotbox.ru
dudi11.off.co.il
durantilumi1cao.com.br
dw.idchecker.co.kr
dx.woyo8g.com
e-airkoryo.com
e.amg777.com
ebestsite.co.kr
edirrelojoeiro.com.br
elogios0.myartsonline.com
emes.com.br
empresarial0001.pisem.su
energy-sol.com
exeype.cn
extex-events.ru
f-forge.com
fhblack.com
fideizm.ru
fileanchor.com
findreaso1ble.org
flashplaginsmirror.com
flashplayer.home.sapo.pt
fondbaybakova.ru
for23.3322.org
forrodotchaka.com.br
forum.factor8guild.com
fotoalbumbr.flog.br
fotoemsg.110mb.com
fotosbalada10x.fileave.com
fotoslinks439856.com.sapo.pt
franciszkankiswklary.ofm.pl
freefilehosting.net
freeweb.siol.net
fuck-celebrities-movie.com
galeon.com
gclass.it
generalantivirus.com
ghterwa.com
gizemguvenfa1tikleri.googlepages.com
glla.net
gokzed.cn
golary.cn
goldeninka.ii1a.net
google.netcdn.com
gorazyn.cn
govsaude.110mb.com
grwww.info
gtpq.info
gtz-legalproject.az
gyfsanimados2009.com.sapo.pt
gyfvuxe.cn
gymarqe.cn
hagnuor.cn
hahdyti.cn
haimadhav.googlepages.com
hakaymobilya.com
hgtr3.com
hiqtacy.cn
hjwx3.com
hjyuw2.com
hohu.spacequadrat.de
homecards11.no.sapo.pt
hosting.free2w.com
hotmailtorpedos2008.com.sapo.pt
humano.ya.com
icepot.cn
idfc2.info
idoafy.cn
idoape.cn
ies.bbexe.cn
ifueme.cn
ifypeod.cn
igafep.cn
igakuot.cn
igayzde.cn
igeuvat.cn
igivor.cn
igoudix.cn
igouhxe.cn
iguyzmo.cn
igycoat.cn
ihaegup.cn
ihaerxi.cn
ihagoin.cn
ihoekag.cn
ihogedi.cn
ihouvi.cn
ihuere.cn
ihuqoyr.cn
ijaheuw.cn
ijakony.cn
ijazofy.cn
ijeife.cn
ijelodi.cn
ijepiyq.cn
ijesiam.cn
ijobuaw.cn
ijuebka.cn
ijyadpi.cn
ijyoxri.cn
ikayvo.cn
ikeuqe.cn
ikeysi.cn
ikioda.cn
ikoiwe.cn
ikorate.cn
ikuaxge.cn
ikyadeh.cn
ikyigy.cn
ildapadilha.110mb.com
ileufby.cn
ilipyw.cn
ilixyeq.cn
ilodux.cn
iloefe.cn
iluefot.cn
img242.imageshack.us
img503.imageshack.us
img522.imageshack.us
i1gyve.cn

•