Edit the content above to lint draft changes before deploying. Use "Fetch robots.txt" to load content from a domain.
Validation & Linting
Run a test to see validation and linting insights for the current robots.txt content.
Robots.txt Tester & Validator
Fetch, lint, and validate robots.txt directives for any URL, user agent, or custom draft before you deploy changes.
How it works+
What this tool does
The Robots.txt Tester & Validator fetches, tests, and validates robots.txt files against Google's official Robots Exclusion Protocol (REP) guidelines. It allows you to test specific URLs against robots.txt rules for any user agent, validate file syntax and structure, and identify potential issues before deploying changes.
How it works
- Fetch or Input: Use "Fetch robots.txt" to automatically retrieve a domain's robots.txt file, or paste/edit content directly in the editor to test draft changes.
- Test URL: Enter a specific URL to test and select a user agent (Googlebot, Bingbot, GPTBot, etc.). The tool uses Google's exact matching logic to determine if the URL is allowed or disallowed.
- Rule Matching: The tool identifies the most specific user-agent group, finds the longest matching path rule, and applies Google's precedence rules (longest path wins; if equal length, Allow wins over Disallow).
- Validation & Linting: The tool performs comprehensive validation checks based on Google's official guidelines, flagging errors, warnings, and passed checks with detailed explanations.
- Visual Feedback: The matched rule is highlighted in the editor (green for allowed, red for disallowed) with line numbers, making it easy to see which rule applies to your test URL.
What it audits & lints for
- Syntax & Format: Malformed lines, missing colons, invalid directives, UTF-8 encoding, file size limits (500 KiB)
- Path Validation: Paths must start with "/" (warnings for wildcard-first paths like "*/pattern"), proper wildcard usage (* and $), empty path handling
- User-Agent Grouping: Rules must be associated with User-agent declarations, proper group separation, missing global groups (User-agent: *)
- Sitemap & Crawl-delay: Valid sitemap URLs, appropriate crawl-delay values, proper placement
- Best Practices: Duplicate rules, overly broad wildcards, conflicting Allow/Disallow directives