Google open sources standardized code in bid to become Mr Robots.txt

Forget about past technical decisions made by fiat, this time your thoughts matter

Google on Monday released its robots.txt parsing and matching library as open source in the hope of its now public code will help encourage web developers to agree on a standard way to spell out the proper etiquette for web crawlers."