ai-crawler-audit

Know exactly which AI bots your site allows — and which ones you forgot to block.

Install · Usage · What it checks · Why this exists

---

Most sites that "block AI" only block GPTBot. Meanwhile ClaudeBot, PerplexityBot, Google-Extended, CCBot and two dozen others walk straight through an open door. ai-crawler-audit reads your robots.txt, evaluates every known AI/LLM crawler against the real matching rules, and shows you the gap in one command.

No API keys. No sign-up. No dependencies — just Python.

AI-Crawler Policy Audit  example.com  (path: /)
==================================================================

Model training
  ● BLOCKED  GPTBot                OpenAI
  ○ allowed  ClaudeBot             Anthropic      (not named)
  ○ allowed  Google-Extended       Google         (not named)
  ...
------------------------------------------------------------------
  3/27 AI crawlers blocked · 4 explicitly named · 14.8% coverage

  ⚠ Gap: you address some AI bots but not all. Not named in robots.txt:
    ClaudeBot, anthropic-ai, PerplexityBot, Bytespider, Amazonbot …

Install

No install required. Clone and run:

git clone https://github.com/seoprocheck/ai-crawler-audit.git
cd ai-crawler-audit
python3 ai_crawler_audit.py example.com

Requires Python 3.8+. Uses the standard library only — nothing to pip install.

Usage

# Audit a live site
python3 ai_crawler_audit.py example.com

# Test a specific path (rules can differ per directory)
python3 ai_crawler_audit.py example.com --path /blog/

# Machine-readable output for CI / dashboards
python3 ai_crawler_audit.py example.com --json

# Audit a local robots.txt without hitting the network
python3 ai_crawler_audit.py --file ./robots.txt

What it checks

The tool tracks 27 AI/LLM crawlers across three access categories, because "an AI bot" isn't one thing — training a model on your content and citing you in an answer are very different decisions:

Category	What access means	Example bots
Model training	Your content is used to train an LLM	GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider
AI search / citations	Your pages can surface in AI search answers	OAI-SearchBot, PerplexityBot, Claude-SearchBot, YouBot
Assistant live-fetch	Fetched on-demand when a user asks an assistant	ChatGPT-User, Perplexity-User, Claude-User, Amazonbot

It implements the parts of the robots.txt spec that actually decide the outcome — per-agent groups, longest-match precedence, * wildcards, $ end-anchors, and Allow-over-Disallow tie-breaking — so the verdict matches what the crawler itself would compute, not a naive substring check.

The headline number is coverage: what share of known AI crawlers your robots.txt explicitly names. A site can feel protected while addressing 15% of the bots that matter.

Why

Blocking or allowing AI crawlers is now a real editorial and business decision — publishers gate training access, e-commerce sites want to be cited in AI search, and "we blocked AI" is often quietly false. Existing robots.txt testers were built for Googlebot in 2010 and don't know these bots exist. This one is built for the AI era and stays current.

Output for automation

--json emits a stable schema (summary + per-crawler verdict, reason, explicit, purpose, operator) suitable for monitoring, CI gates, or feeding a dashboard.

License

MIT © SEO Pro Check · built by @seoprocheck. Contributions and new bot signatures welcome.