Robots.txt Tester
Free online robots.txt tester. Parse robots.txt rules and check which user-agents can crawl any URL with Allow/Disallow verdicts, wildcard support, and directive warnings - fully in-browser.
Loading tool...
What is Robots.txt Tester?
A robots.txt tester is a diagnostic tool that parses robots.txt rules and simulates crawler access: paste your file, enter a URL, and it tells you exactly which user-agents would be allowed or blocked from crawling that path, including wildcard handling and Allow/Disallow precedence.
Key Benefits
- Prevent accidental site-wide blocks before they deindex your pages from search engines
- Verify that Googlebot, Bingbot, and AI crawlers can reach the content you want indexed
- Understand exactly which rule wins for any URL instead of guessing at precedence
- Validate wildcard patterns such as /*.pdf$ that are easy to miswrite
- Works fully offline - safe for internal, staging, or unreleased robots.txt rules
Common Use Cases
- •Pre-launch SEO audit of a new robots.txt before deploying it to production
- •Debugging why a page disappeared from Google after a robots.txt change
- •Checking that AI crawlers like GPTBot or ClaudeBot are blocked or allowed as intended
- •Verifying staging or internal environment rules do not leak into production behavior
- •Learning how Allow/Disallow precedence and wildcards resolve for tricky paths
How to Test Robots.txt Rules
- Paste your robots.txt: Copy the robots.txt content from your site (yoursite.com/robots.txt) into the text area, or load the example to explore.
- Enter the URL to test: Type the full URL of the page you want to check. Only the path is used for matching, exactly like real crawlers do.
- List user-agents: Optionally enter comma-separated agents such as Googlebot, Bingbot, or GPTBot. Leave empty to test every group defined in the file.
- Review the verdicts: Click Test to see ALLOWED or DISALLOWED per agent, the winning rule, Crawl-delay, parsed directives, and any warnings. Copy the results as JSON if needed.
Key Features
- Simulate crawl verdicts per user-agent with Allow/Disallow longest-match resolution
- Supports robots.txt wildcards: * (any character sequence) and $ (end-of-URL anchor)
- Parsed directive table covering User-agent, Allow, Disallow, Crawl-delay, and Sitemap lines
- Warnings for unknown directives and rule groups missing a User-agent
- Test multiple user-agents at once and copy the verdict report as JSON