AI Crawler Visibility Checker
Audit robots.txt and meta tags for AI crawler rules - GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Bytespider - with per-bot verdicts and
Loading tool...
What is AI Crawler Visibility Checker?
The AI Crawler Visibility Checker audits your robots.txt and page meta tags to reveal whether major AI crawlers - GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, and Bytespider - can access your content. It reports a per-bot verdict and generates copy-paste blocking rules, entirely offline.
Key Benefits
- Instant per-crawler verdicts instead of reading robots.txt by hand
- Mirrors how real crawlers match specific user-agent groups first
- Detects noai and TDM-Reservation page-level declarations
- One-click blocking snippet for all major AI crawlers
- Runs fully offline - nothing is fetched or uploaded
Common Use Cases
- •Publishers deciding an AI training opt-out policy
- •Auditing sites after a CMS migration that may have dropped robots rules
- •Checking whether wildcard rules silently allow AI bots
- •Generating block rules for a multi-site robots.txt rollout
- •Verifying a page head carries noai declarations where wanted
How to Check AI Crawler Access
- Paste your robots.txt: Copy the live robots.txt from yoursite.com/robots.txt into the first box.
- Paste the page head: Optionally add the HTML head of a key page to detect noai and TDM-Reservation declarations.
- Run the analysis: Click Analyze to see a per-crawler verdict table - blocked, allowed by specific rule, or allowed by wildcard default.
- Copy the blocking snippet: If any crawler is allowed, paste the generated robots.txt rules and meta tag into your site.
Key Features
- Six major AI crawlers checked in one pass
- Specific-rule-first robots.txt matching, like real crawlers
- Page head scan for noai, noimageai, and TDM-Reservation tags
- Per-bot allowed or blocked verdicts with wildcard fallback noted
- Copy-paste robots.txt and meta snippets to block AI crawlers
- Fully offline - your files never leave the browser
Frequently Asked Questions
Which AI crawlers does this tool check?
GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), PerplexityBot, Google-Extended, and Bytespider - the crawlers most commonly referenced in AI training opt-out discussions.
Does blocking AI crawlers affect Google search ranking?
Blocking Google-Extended does not remove your site from Google Search - it only limits Gemini training and grounding. Blocking Googlebot itself would affect search, but Googlebot is a separate user agent.
Are AI crawlers required to obey robots.txt?
Polite crawlers like GPTBot, ClaudeBot, and CCBot state they respect robots.txt, but compliance is voluntary - it is not a technical barrier. Server-side rate limiting or firewall rules are stronger enforcement.
What are the noai meta tags?
They are a proposed non-standard extension to the robots meta tag. Some AI services have signaled support, but coverage is inconsistent - robots.txt rules remain the most widely honored signal.