• How Good Are Website-Reputation Services?

    Websites on the Internet have now become the standard modus operandi for spreading malicious software to infect personal and corporate environments. A large number of benign and well-meaning websites are compromised everyday by hackers inserting malicious code to, in turn, infect the computers used by visitors to the hacked site. One of the ways to combat this is to develop a website reputation mechanism which can warn of potential threats before visiting a compromised site.

    Website-reputation services vary wildly in their opinions

    Website-reputation services vary wildly in their opinions.

    Note that all 350 domains, were reported as malicious, and were collected from malware.com.br on December 18, 2009. The blue column (maximum 350) indicates the number of sites that the website-reputation service correctly identified reported bad sites. The orange column (maximum 350) indicates the number of sites that the website-reputation services incorrectly identified reported malicious sites as safe.

    Website reputation services have been around for nearly 5-7 years now. Initially developing as a niche product line which could serve to provide an opinion of a site’s reputation to full fledged offerings which provide advisories about websites, whether they are distributing malware, and if they are, what kind, and using which Autonomous Systems.

    At StopTheHacker.com (Jaal LLC) we have conducted tests with 350 domain names, all of which have been reported as malicious by volunteers of various blacklists.

    The aim of the test is to:

    1. Identify how accurate the website reputation services are
    2. What is the overlap in terms of safe/unsafe websites

    We have found some interesting results which we present in this article. First we detail the parameters of the testing procedure to provide an idea of how the test was set up.

    350 URLs were collected from malware.com.br (mbr) on December 18, 2009. These URLs are reported to this website for listing by one or more of the following: individuals, organizations, agencies and software products or services.¬† We assume for the purposes of this test that all the URLs obtained from the “regular” list from mbr are malicious and hence deemed “unsafe” to visit.

    We compare the reputation provided by each website-reputation service and observe how many websites are marked as unsafe, safe, untested, maybe-unsafe/caution/potentially-unsafe, unreachable.

    Note, that when analyzing a domain name, for checking with the Google safebrowsing API, we have had to calculate the MD5 hashes of the website names to match with the malware hash list. The date that we conducted this test was: December 21, 2009. The list of domain names tested are presented below and a graph representing the statistics for the first 350 sites tested is above.

    We have identified some of the most interesting results below:

    1. McAfee Siteadvisor marked 32.5% of Domains as Unsafe, 22% as Safe, 43% as Untested and 1.7% as Potentially-unsafe.
    2. Norton Safeweb marked 50.86% of Domains as Unsafe, 43.71% as Safe, 2.29% as Untested and 3.14% as Potentially-unsafe.
    3. Google SafeBrowsing marked 10.86% of Domains as Unsafe, 89.14% as Safe. Note: the presence of the hash of the domain name¬† being tested, on the google malware hash list, is interpreted as “unsafe” while the absence in interpreted as “safe”.
    4. Comodo Siteinspector marked 0.29% of Domains as Unsafe, 98.86% as Safe and 0.86% as Unreachable. Note: after feedback from Comodo, a retest was conducted, accuracy changed from 0.29% -> 1.2%.

    This limited test is a first step towards showing how much variance there is website reputation services that are currently being offered by large Internet-services/security companies. To highlight this point we present immediately below the relatively few domains (~6% of the total domains tested) that were marked as bad by all three major services, Norton, McAfee, and Google.

    In brief:

    • 6% of domains tested were marked as “unsafe” by all 3, McAfee, Norton and Google
    • 10% of domains tested were marked as “unsafe” by Norton and Google
    • 22% of domains tested were marked as “unsafe” by Norton and McAfee
    • 5.7% of domains tested were marked as “unsafe” by Google and McAfee

    Update: December 28, 2009

    After receiving helpful feedback from representatives at Comodo, we were informed that Comodo’s service could provide more accurate answers if complete web page locations were checked instead of just the domain name. We followed the advice and saw a definite increase in Comodo’s accuracy. Comodo marked 1.2% of the website/pages as malicious. Prior to this re-test, the same service marked 0.2% of the websites as unsafe. The graph at the beginning of this article does not represent the results of this re-test.

    Below we list the websites from which we extract the statistics above

    Websites marked as “unsafe” by Norton, McAfee and Google

    219.148.34.10
    219.148.34.9
    4gameranking.com
    77.245.61.232
    aiongamemeca.com
    durantilumi1cao.com.br
    golary.cn
    hagnuor.cn
    igivor.cn
    igoudix.cn
    igouhxe.cn
    ihaegup.cn
    ihaerxi.cn
    ihagoin.cn
    ihoekag.cn
    ihouvi.cn
    ihuere.cn
    ihuqoyr.cn
    ijaheuw.cn
    ikyigy.cn
    iloefe.cn
    

    Websites marked as “unsafe” by Google and Norton

    212.99.87.130
    219.148.34.10
    219.148.34.9
    4gameranking.com
    61.164.108.213
    77.245.61.232
    aimeblog.com
    aiongamemeca.com
    bhactuant.com
    durantilumi1cao.com.br
    findreaso1ble.org
    for23.3322.org
    golary.cn
    gyfvuxe.cn
    hagnuor.cn
    ifueme.cn
    igivor.cn
    igoudix.cn
    igouhxe.cn
    iguyzmo.cn
    ihaegup.cn
    ihaerxi.cn
    ihagoin.cn
    ihoekag.cn
    ihogedi.cn
    ihouvi.cn
    ihuere.cn
    ihuqoyr.cn
    ijaheuw.cn
    ijakony.cn
    ijazofy.cn
    ijeife.cn
    ijelodi.cn
    ikyigy.cn
    iloefe.cn
    

    Websites marked as “unsafe” by McAfee and Google

    219.148.34.10
    219.148.34.9
    4gameranking.com
    77.245.61.232
    aiongamemeca.com
    durantilumi1cao.com.br
    emes.com.br
    golary.cn
    hagnuor.cn
    igivor.cn
    igoudix.cn
    igouhxe.cn
    ihaegup.cn
    ihaerxi.cn
    ihagoin.cn
    ihoekag.cn
    ihouvi.cn
    ihuere.cn
    ihuqoyr.cn
    ijaheuw.cn
    ikyigy.cn
    iloefe.cn
    

    Websites marked as “unsafe” by McAfee and Norton

    163.fuckunion.com
    206.161.127.72
    208.75.230.43
    209.205.196.16
    218.93.205.250
    219.148.34.10
    219.148.34.9
    4gameranking.com
    61.235.117.72
    70.148.212.252
    77.245.61.232
    82.98.235.173
    85.92.157.141
    91.213.126.100
    97feihu.com
    adobeflashupdates.com
    adwareprotectionsite.com
    aiongamemeca.com
    amforum.lua.pl
    antivirus-live.com
    artistinove.it
    centralspa.ca
    comerciocentral.net
    densmail.com
    diadoamigo0.myartsonline.com
    dimorphothec.com
    dl.get-torrent.com
    dl.targetsaver.com
    dudi11.off.co.il
    durantilumi1cao.com.br
    ebestsite.co.kr
    elogios0.myartsonline.com
    exeype.cn
    fuck-celebrities-movie.com
    gclass.it
    generalantivirus.com
    ghterwa.com
    gokzed.cn
    golary.cn
    google.netcdn.com
    gorazyn.cn
    hagnuor.cn
    hahdyti.cn
    hgtr3.com
    hiqtacy.cn
    hjyuw2.com
    icepot.cn
    idoafy.cn
    idoape.cn
    igafep.cn
    igakuot.cn
    igeuvat.cn
    igivor.cn
    igoudix.cn
    igouhxe.cn
    igycoat.cn
    ihaegup.cn
    ihaerxi.cn
    ihagoin.cn
    ihoekag.cn
    ihouvi.cn
    ihuere.cn
    ihuqoyr.cn
    ijaheuw.cn
    ijepiyq.cn
    ijesiam.cn
    ijobuaw.cn
    ijuebka.cn
    ikoiwe.cn
    ikorate.cn
    ikuaxge.cn
    ikyadeh.cn
    ikyigy.cn
    ileufby.cn
    ilixyeq.cn
    ilodux.cn
    iloefe.cn
    iluefot.cn
    i1gyve.cn
    

    Interestingly, Comodo’s service marked only 1 website, 218.146.255.156 as malicious. This domain was also marked malicious by Norton, “Untested” by McAfee and was not found on the Google malware hash list. Below follows the complete list of domains that were tested.

    Complete list of domains tested

    001.bbexe.cn
    113.105.175.138
    114.207.112.169
    119.147.114.163
    12.10.157.6
    12.24.238.229
    12.25.151.68
    121.12.127.230
    121.205.91.142
    121.205.91.145
    123.244.30.118
    123.244.30.66
    123.bbexe.cn
    147.163.1.77
    148.208.196.2
    163.fuckunion.com
    174.36.233.59
    192.220.110.228
    193.104.27.139
    193.169.234.27
    200.111.155.122
    200.242.43.250
    200.63.5.78
    200.67.103.187
    200.69.124.17
    202.105.183.104
    202.114.181.5
    204.12.43.43
    204.232.131.12
    206.161.127.72
    208.75.230.43
    209.131.200.246
    209.172.35.144
    209.205.196.16
    209.43.123.143
    210.166.220.240
    210.206.8.254
    210.51.166.217
    211.39.130.196
    211.78.87.42
    212.31.234.155
    212.63.132.215
    212.88.178.22
    212.97.63.156
    212.99.87.130
    216.24.165.4
    216.240.148.175
    217.116.46.139
    218.146.255.156
    218.16.120.253
    218.188.0.5
    218.6.15.135
    218.63.200.196
    218.86.118.98
    218.93.202.115
    218.93.205.250
    219.146.128.242
    219.146.128.245
    219.148.34.10
    219.148.34.9
    220.90.213.158
    220.95.232.28
    221.1.204.243
    221.143.43.200
    222.66.209.98
    222.76.243.53
    24.1188d.cn
    24.65.70.52
    3.1188d.cn
    3310.net.cn
    38.99.91.47
    3s.8i9i.com
    46.1188d.cn
    46.3388a.cn
    4gameranking.com
    5.1188d.cn
    53.1188d.cn
    58.147.27.69
    58.215.79.176
    6.1188d.cn
    60.191.39.6
    61.108.173.3
    61.110.21.192
    61.164.108.213
    61.235.117.72
    62.193.229.83
    64.160.216.20
    65.109.240.130
    65.183.178.92
    66.116.229.233
    66.152.93.119
    66.220.17.157
    66.45.235.228
    67.19.9.234
    67.43.224.77
    68.153.57.9
    70.148.212.252
    72.10.166.195
    72.20.6.106
    72.237.212.57
    72.35.84.6
    72.64.146.16
    731273265.520815.com
    76.162.68.70
    76.73.42.43
    77.245.61.232
    77.92.158.122
    78.159.127.254
    78.46.151.179
    80.153.182.80
    81.223.40.244
    81.252.31.148
    82.114.87.46
    82.98.235.173
    83.103.59.84
    83.206.113.161
    83.240.174.136
    83.245.62.87
    84.20.251.223
    85.17.136.139
    85.25.81.140
    85.92.157.141
    91.207.7.116
    91.213.126.100
    93.174.95.140
    95.211.98.136
    97feihu.com
    98.126.34.250
    a.amg777.com
    a1964.g.akamai.net
    absi2008.netfirms.com
    acripino7878.110mb.com
    admin.bbexe.cn
    adobeflashupdates.com
    adwareprotectionsite.com
    aha-autoimage.com
    aimeblog.com
    aiongamemeca.com
    album.pagi1s.sapo.pt
    alison.wz.cz
    alkeichah.com
    amforum.lua.pl
    amoravela.com.sapo.pt
    antivirus-live.com
    antivirusadvanced.com
    arathas.de
    arcade.ya.com
    arkbroadcasters.org
    artdeli.co.kr
    artistinove.it
    atencaousuario.webcindario.com
    atualizaca-juridica.sitesled.com
    ausamedia.berepublic.com
    avr-download.com
    b.amg777.com
    backstaroup.home.sapo.pt
    bb.bbexe.cn
    bbs.pxtang.cn
    bcfpb.com
    bchokies.com
    bdesata.com
    belezademulher.org
    best-sale.us
    bevaccine.com
    bgcomstock.com
    bhactuant.com
    blog20fc2.com
    blogaofotos8.com.sapo.pt
    blogfotos2008.com.sapo.pt
    blogpesoalpessoal.com.sapo.pt
    bmz.horizon.net.pl
    brasilterra.com.sapo.pt
    c.amg777.com
    caixa-cefinstall.sitesled.com
    caixaeconomica-gov.sitesled.com
    cancelamentt0.googlepages.com
    carbys.no.sapo.pt
    card2009.com.sapo.pt
    cardamorhtml.no.sapo.pt
    cardpaixao.esmartdesign.com
    cartao8578.com.sapo.pt
    cartaoamizade000.com.sapo.pt
    cartaoespecial9.com.sapo.pt
    cartaovirtual2006.no.sapo.pt
    cartoesnovos.250x.com
    cartoesuol.com.sapo.pt
    cartoeswebapaxo1do.no.sapo.pt
    casasbahia.com.sapo.pt
    cau.ac.kr
    centralspa.ca
    chaiyapruekpethospital.com
    chamadavideo-1.my3gb.com
    chi1oilfactory.cn
    chinesefreewebs.com
    ciduninstall.com
    cinema-film-4you.ru
    club.telepolis.com
    comerciocentral.net
    comunidade777.110mb.com
    config.koreamessenger.com
    correiosweb.com.sapo.pt
    cprzafra.juntaextremadura.net
    d.amg777.com
    d.kkkmfdy.com
    d4.kkkmfdy.com
    damnkt.logi1pp.com
    db.ms.kr
    denizlisurucukursu.com.tr
    densmail.com
    diadoamigo0.myartsonline.com
    dimorphothec.com
    di1r-cs.real-host.ru
    dindindopv.bravehost.com
    ditto.arpa.org
    dl.get-torrent.com
    dl.qvodir.cn
    dl.targetsaver.com
    dl.woyo8g.com
    dl02.softdown-load.com.cn
    dollardream.ru
    donghae.ms.kr
    dorota288.w8w.pl
    down.1vysoft.org
    down.woyo8g.com
    down.yellowsoft.org
    download.gameztar.com
    download.iobit.com
    download.leeboo.com
    download.softpedia.com
    downlopaginvisualiz.com.sapo.pt
    dtvprosoft.hotbox.ru
    dudi11.off.co.il
    durantilumi1cao.com.br
    dw.idchecker.co.kr
    dx.woyo8g.com
    e-airkoryo.com
    e.amg777.com
    ebestsite.co.kr
    edirrelojoeiro.com.br
    elogios0.myartsonline.com
    emes.com.br
    empresarial0001.pisem.su
    energy-sol.com
    exeype.cn
    extex-events.ru
    f-forge.com
    fhblack.com
    fideizm.ru
    fileanchor.com
    findreaso1ble.org
    flashplaginsmirror.com
    flashplayer.home.sapo.pt
    fondbaybakova.ru
    for23.3322.org
    forrodotchaka.com.br
    forum.factor8guild.com
    fotoalbumbr.flog.br
    fotoemsg.110mb.com
    fotosbalada10x.fileave.com
    fotoslinks439856.com.sapo.pt
    franciszkankiswklary.ofm.pl
    freefilehosting.net
    freeweb.siol.net
    fuck-celebrities-movie.com
    galeon.com
    gclass.it
    generalantivirus.com
    ghterwa.com
    gizemguvenfa1tikleri.googlepages.com
    glla.net
    gokzed.cn
    golary.cn
    goldeninka.ii1a.net
    google.netcdn.com
    gorazyn.cn
    govsaude.110mb.com
    grwww.info
    gtpq.info
    gtz-legalproject.az
    gyfsanimados2009.com.sapo.pt
    gyfvuxe.cn
    gymarqe.cn
    hagnuor.cn
    hahdyti.cn
    haimadhav.googlepages.com
    hakaymobilya.com
    hgtr3.com
    hiqtacy.cn
    hjwx3.com
    hjyuw2.com
    hohu.spacequadrat.de
    homecards11.no.sapo.pt
    hosting.free2w.com
    hotmailtorpedos2008.com.sapo.pt
    humano.ya.com
    icepot.cn
    idfc2.info
    idoafy.cn
    idoape.cn
    ies.bbexe.cn
    ifueme.cn
    ifypeod.cn
    igafep.cn
    igakuot.cn
    igayzde.cn
    igeuvat.cn
    igivor.cn
    igoudix.cn
    igouhxe.cn
    iguyzmo.cn
    igycoat.cn
    ihaegup.cn
    ihaerxi.cn
    ihagoin.cn
    ihoekag.cn
    ihogedi.cn
    ihouvi.cn
    ihuere.cn
    ihuqoyr.cn
    ijaheuw.cn
    ijakony.cn
    ijazofy.cn
    ijeife.cn
    ijelodi.cn
    ijepiyq.cn
    ijesiam.cn
    ijobuaw.cn
    ijuebka.cn
    ijyadpi.cn
    ijyoxri.cn
    ikayvo.cn
    ikeuqe.cn
    ikeysi.cn
    ikioda.cn
    ikoiwe.cn
    ikorate.cn
    ikuaxge.cn
    ikyadeh.cn
    ikyigy.cn
    ildapadilha.110mb.com
    ileufby.cn
    ilipyw.cn
    ilixyeq.cn
    ilodux.cn
    iloefe.cn
    iluefot.cn
    img242.imageshack.us
    img503.imageshack.us
    img522.imageshack.us
    i1gyve.cn