Robots.txt Tester
Free online robots.txt tester.
Loading tool...
What is Robots.txt Tester?
A robots.txt tester is a diagnostic tool that parses robots.txt rules and simulates crawler access: paste your file, enter a URL, and it tells you exactly which user-agents would be allowed or blocked from crawling that path, including wildcard handling and Allow/Disallow precedence.
Key Benefits
- Prevent accidental site-wide blocks before they deindex your pages from search engines
- Verify that Googlebot, Bingbot, and AI crawlers can reach the content you want indexed
- Understand exactly which rule wins for any URL instead of guessing at precedence
- Validate wildcard patterns such as /*.pdf$ that are easy to miswrite
- Works fully offline - safe for internal, staging, or unreleased robots.txt rules
Common Use Cases
- •Pre-launch SEO audit of a new robots.txt before deploying it to production
- •Debugging why a page disappeared from Google after a robots.txt change
- •Checking that AI crawlers like GPTBot or ClaudeBot are blocked or allowed as intended
- •Verifying staging or internal environment rules do not leak into production behavior
- •Learning how Allow/Disallow precedence and wildcards resolve for tricky paths
How to Test Robots.txt Rules
- Paste your robots.txt: Copy the robots.txt content from your site (yoursite.com/robots.txt) into the text area, or load the example to explore.
- Enter the URL to test: Type the full URL of the page you want to check. Only the path is used for matching, exactly like real crawlers do.
- List user-agents: Optionally enter comma-separated agents such as Googlebot, Bingbot, or GPTBot. Leave empty to test every group defined in the file.
- Review the verdicts: Click Test to see ALLOWED or DISALLOWED per agent, the winning rule, Crawl-delay, parsed directives, and any warnings. Copy the results as JSON if needed.
Key Features
- Simulate crawl verdicts per user-agent with Allow/Disallow longest-match resolution
- Supports robots.txt wildcards: * (any character sequence) and $ (end-of-URL anchor)
- Parsed directive table covering User-agent, Allow, Disallow, Crawl-delay, and Sitemap lines
- Warnings for unknown directives and rule groups missing a User-agent
- Test multiple user-agents at once and copy the verdict report as JSON
Frequently Asked Questions
What does the robots.txt tester check?
It parses your pasted robots.txt into User-agent groups and simulates whether each agent is ALLOWED or DISALLOWED from crawling a given URL path, following the robots.txt convention that the longest matching rule wins and Allow takes precedence on equal-length matches.
Do wildcards work in this tester?
Yes. It supports the * wildcard (matches any sequence of characters) and the $ end-of-URL anchor, which are the two wildcards defined in the robots.txt standard used by Google, Bing, and other major crawlers.
Is my robots.txt content uploaded anywhere?
No. All parsing and matching runs entirely in your browser with JavaScript. Nothing is sent to any server, so it is safe to test internal, staging, or unreleased rules.
Why does an empty Disallow show as allowed?
In robots.txt syntax an empty Disallow value means "no restrictions" for that group. If no rule matches the tested path at all, crawling is allowed by default - this is the standard behavior implemented by real crawlers.