Hackers are hitting websites hard and fast. Everyday, upwards of 6,000 new websites are compromised by malware due to code injection, FTP credential compromise, weak server security, web-application flaws and the full gamut of other security issues.
In this vein, any system used to determine whether a website is clean or infected, needs to be able to handle large numbers of sites for analysis. This ability ensures a high throughput rate when analyzing “suspect” sites.
One of our goals at StopTheHacker.com, is to target throughput rates in excess of 1,000,000 sites per day. This obviously necessitates an automated process with high reliability and accuracy (we have it). To develop such an automated process, we focus heavily on advanced Machine Learning and Artificial Intelligence techniques which can learn on the fly from compromised websites and update to catch even more bad websites. All on the fly.
In order to develop training sets for machine-learning based automated solutions, one needs to get hold of a massive dataset. We recently profiled over 2.8 million websites (2,800,560 to be exact). What dataset is this? All these profiled sites were sourced from DMOZ. Surprisingly, none of these websites are listed in the Google Safe Browsing List as of January 19, 2010.
Note: DMOZ is a user-edited directory of sites (which provided a good starting point for this experiment).
Each website is classified according to a categorization scheme described here. We used the description to download and analyze around 2.8 million sites. Each site name was entered in a program which calculated a hash of the site name and looked it up on the Google Safe Browsing List to determine if the website was on the malware list or not.
Interestingly, we did not find any of the sites on the Google Safe Browsing List. This definitely adds a feather of sorts to DMOZ Directory’s proverbial hat. I think they might just be able to claim that they are the “largest and safest human-edited directory on the web”!
A graphical representation of the top 50 categories, sorted by those having the most websites is presented, followed by a list of the top 100 categories.
DMOZ categories ranked by the number of associated sites:
Sites Category 654667 Regional/N 654420 Regional/North_America 569801 Regional/E 265961 Regional/Europe 120813 Regional/A 120813 Regional/A 88487 Society/Religion_and_Spirituality 66790 Arts/Music 66790 Arts/Music 49708 Regional/O 49268 Regional/Oceania 48641 Regional/Asia 39457 Reference/Education 35682 Games/Video_Games 34671 Arts/Movies 31420 Science/B 31403 Science/Biology 29998 Computers/Software 28296 Computers/Internet 24826 Regional/C 23427 Sports/S 22570 Arts/Literature 22570 Arts/Literature 22274 Recreation/Pets 20796 Science/S 20016 Society/Law 19946 Arts/Performing_Arts 19651 Science/Social_Sciences 18652 Business/Industrial_Goods_and_Services 17764 Sports/B 17753 Society/Issues 17493 Business/Textiles_and_Nonwovens 17248 Business/Business_Services 17247 Business/Construction_and_Maintenance 17033 Business/Arts_and_Entertainment 16948 Regional/Africa 16010 Computers/Programming 15703 Business/Consumer_Goods_and_Services 15418 Sports/Soccer 14186 Sports/M 13701 Society/People 13696 Sports/E 13672 Society/Organizations 13348 Science/E 13063 Arts/Visual_Arts 13001 Recreation/Outdoors 12976 Health/Conditions_and_Diseases 12696 Sports/F 11815 Science/T 11653 Home/Cooking 11383 Shopping/Home_and_Garden 11332 Regional/M 11332 Regional/M 11190 Regional/Middle_East 11075 Science/Technology 10589 Science/M 10456 Sports/Equestrian 10236 Society/History 10157 Health/Medicine 10030 Arts/Television 9875 Science/Math 9429 Sports/G 8948 Arts/Animation 8581 Sports/W 8216 Science/A 8117 Business/Financial_Services 8117 Business/Financial_Services 8070 Business/Electronics_and_Electrical 8049 Arts/People 7920 Shopping/Crafts 7920 Shopping/Crafts 7771 Business/Transportation_and_Logistics 7764 Business/Agriculture_and_Forestry 7597 Sports/Golf 7597 Sports/Golf 7511 Sports/Football 7481 Shopping/Clothing 7481 Shopping/Clothing 7466 Shopping/Food 7265 Recreation/Food 7189 Sports/C 7072 Sports/Basketball 7028 Shopping/Health 6606 Science/Environment 6398 Regional/U 6390 Computers/Hardware 6390 Computers/Hardware 6372 Business/Food_and_Related_Products 6271 Sports/H 6222 Recreation/Autos 6100 Science/Earth_Sciences 6100 Science/Earth_Sciences 6003 Arts/Crafts 6001 Sports/Martial_Arts 5938 Regional/Caribbean 5904 Sports/Motorsports 5880 Recreation/Travel 5590 Business/Healthcare 5560 Society/Genealogy 5437 Society/Philosophy