Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard
Google wants to turn the Robots Exclusion Protocol (REP) into an official internet standard -- and it's making its own parser open source to help.Read More