Crawler Master X

Multithreaded Linux-based crawling system including numerous custom crawlers and extensive documentation on setting it up, running crawls, and using results of the crawls to build large Google-indexed sites which receive in excess of 10,000 unique visitors a day without any SEO or PPC.