Open Source Web Crawling is About Ten to Fifteen Years Behind Google

In 1999, it took Google one month to crawl and build an index of about 50 million pages. In 2012, the same task was accomplished in less than one minute. The 2012 capability is about 50,000 times...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]