We're delighted to announce OpenRobotsTxt.org - a project to archive and analyse the world’s robots.txt files, kicking off with analysis of 595 million hostnames courtesy of a huge export of data from Majestic.

So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames! openrobotstxt.org

Cool new project from Majestic 👀

Interested in user-agent data? @majestic has you covered -> Majestic Launches Robots.txt Archive 600M hostnames scanned and 36K user-agents found. "The project has been bootstrapped by a huge data export of robots.txt files collected by the Majestic crawler, MJ12bot. This has