Skip to content
Changelog
new tooldeveloper-web

robots.txt Validator

v1.0

robots.txt Validator breaks a robots.txt file into its user-agent groups, checks it for problems, tests a path against a crawler, and can generate a starting file.

The checker flags unknown or misspelled directives, a leading byte-order mark, rules with no User-agent above them, Allow and Disallow values that don't start with /, a Crawl-delay that isn't a number, and relative Sitemap URLs. It also notes that Googlebot ignores Crawl-delay (Bing, Yahoo and Yandex honor it) and that noindex in robots.txt stopped working on Google in September 2019. Problems appear once you pause typing, so a half-typed directive doesn't flash an error. Each problem has a Jump button that moves the editor to its line.

The URL tester follows Google's documented precedence. Of all the rules whose pattern matches the path, the one with the longest pattern wins (* and $ count as characters), and a tie goes to Allow. The winning rule is marked in the groups list and highlighted in the editor. A crawler uses the group with the longest matching user-agent name, so Googlebot-News falls back to a plain Googlebot group, then to *.

The Sample menu loads four files: a basic one, the worked examples from Google's documentation, an e-commerce file with a crawl delay and per-crawler groups, and a file full of common mistakes. Generate mode builds a file from a hostname (for the Sitemap line), a default of allow or block, three presets, and Allow or Block switches for search and AI crawlers. Generating over existing text offers an Undo for a few seconds.

Your file, test path, user-agent and generator settings are remembered on this device, and Share packs the file and test into the link. Not yet available: a bulk path tester, a diff between two revisions, and a warning for a sitemap on a different host.

All changes