In this vein, any system used to determine whether a website is clean or infected, needs to be able to handle large numbers of sites for analysis. This ability ensures a high throughput rate when analyzing “suspect” sites.

One of our goals at StopTheHacker.com, is to target throughput rates in excess of 1,000,000 sites per day. This obviously necessitates an automated process with high reliability and accuracy (we have it). To develop such an automated process, we focus heavily on advanced Machine Learning and Artificial Intelligence techniques which can learn on the fly from compromised websites and update to catch even more bad websites. All on the fly.

In order to develop training sets for machine-learning based automated solutions, one needs to get hold of a massive dataset. We recently profiled over 2.8 million websites (2,800,560 to be exact). What dataset is this? All these profiled sites were sourced from DMOZ. Surprisingly, none of these websites are listed in the Google Safe Browsing List as of January 19, 2010.
Note: DMOZ is a user-edited directory of sites (which provided a good starting point for this experiment).

Each website is classified according to a categorization scheme described here. We used the description to download and analyze around 2.8 million sites. Each site name was entered in a program which calculated a hash of the site name and looked it up on the Google Safe Browsing List to determine if the website was on the malware list or not.

Interestingly, we did not find any of the sites on the Google Safe Browsing List. This definitely adds a feather of sorts to DMOZ Directory’s proverbial hat. I think they might just be able to claim that they are the “largest and safest human-edited directory on the web”!

Top 50 DMOZ categories (by number of websites).

A graphical representation of the top 50 categories, sorted by those having the most websites is presented, followed by a list of the top 100 categories.

DMOZ categories ranked by the number of associated sites:

Sites	Category

654667	Regional/N
654420	Regional/North_America
569801	Regional/E
265961	Regional/Europe
120813	Regional/A
120813	Regional/A
88487	Society/Religion_and_Spirituality
66790	Arts/Music
66790	Arts/Music
49708	Regional/O
49268	Regional/Oceania
48641	Regional/Asia
39457	Reference/Education
35682	Games/Video_Games
34671	Arts/Movies
31420	Science/B
31403	Science/Biology
29998	Computers/Software
28296	Computers/Internet
24826	Regional/C
23427	Sports/S
22570	Arts/Literature
22570	Arts/Literature
22274	Recreation/Pets
20796	Science/S
20016	Society/Law
19946	Arts/Performing_Arts
19651	Science/Social_Sciences
18652	Business/Industrial_Goods_and_Services
17764	Sports/B
17753	Society/Issues
17493	Business/Textiles_and_Nonwovens
17248	Business/Business_Services
17247	Business/Construction_and_Maintenance
17033	Business/Arts_and_Entertainment
16948	Regional/Africa
16010	Computers/Programming
15703	Business/Consumer_Goods_and_Services
15418	Sports/Soccer
14186	Sports/M
13701	Society/People
13696	Sports/E
13672	Society/Organizations
13348	Science/E
13063	Arts/Visual_Arts
13001	Recreation/Outdoors
12976	Health/Conditions_and_Diseases
12696	Sports/F
11815	Science/T
11653	Home/Cooking
11383	Shopping/Home_and_Garden
11332	Regional/M
11332	Regional/M
11190	Regional/Middle_East
11075	Science/Technology
10589	Science/M
10456	Sports/Equestrian
10236	Society/History
10157	Health/Medicine
10030	Arts/Television
9875	Science/Math
9429	Sports/G
8948	Arts/Animation
8581	Sports/W
8216	Science/A
8117	Business/Financial_Services
8117	Business/Financial_Services
8070	Business/Electronics_and_Electrical
8049	Arts/People
7920	Shopping/Crafts
7920	Shopping/Crafts
7771	Business/Transportation_and_Logistics
7764	Business/Agriculture_and_Forestry
7597	Sports/Golf
7597	Sports/Golf
7511	Sports/Football
7481	Shopping/Clothing
7481	Shopping/Clothing
7466	Shopping/Food
7265	Recreation/Food
7189	Sports/C
7072	Sports/Basketball
7028	Shopping/Health
6606	Science/Environment
6398	Regional/U
6390	Computers/Hardware
6390	Computers/Hardware
6372	Business/Food_and_Related_Products
6271	Sports/H
6222	Recreation/Autos
6100	Science/Earth_Sciences
6100	Science/Earth_Sciences
6003	Arts/Crafts
6001	Sports/Martial_Arts
5938	Regional/Caribbean
5904	Sports/Motorsports
5880	Recreation/Travel
5590	Business/Healthcare
5560	Society/Genealogy
5437	Society/Philosophy