# Free AI crawler checker · Terradium

Can ChatGPT, Claude, Perplexity and Gemini read your site? Enter a URL at https://terradium.io/tools/ai-crawler-checker to check three things. It's free and needs no sign-up.

## What it checks

1. **robots.txt, per AI crawler.** GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Bytespider, evaluated against the page's path with Allow/Disallow precedence and wildcards. A blocked search or browsing crawler fails, because it removes the page from live AI answers. A blocked training crawler only warns.
2. **llms.txt.** Whether it exists, whether it's the HTML app shell served for a missing file, and whether it follows the llmstxt.org format (H1 title, `>` summary, link list). It also checks for llms-full.txt.
3. **Server-rendered content.** The page is fetched as GPTBot and the visible text in the raw HTML is measured. An empty JavaScript app shell fails. It also flags a firewall that blocks AI crawlers, a crawler getting less content than a browser, missing title, description and H1 tags, and missing JSON-LD.

## API

```
POST https://api.terradium.io/api/v1/public/tools/crawler-check
Content-Type: application/json

{"url": "example.com"}
```

No auth. Rate-limited per IP. Results are cached for 24 hours per URL.

## Badge

A live SVG that re-checks daily:

```
<a href="https://terradium.io/tools/ai-crawler-checker?url=example.com"><img src="https://api.terradium.io/api/v1/public/tools/crawler-badge/example.com.svg" alt="AI crawler access, checked by Terradium" height="20"></a>
```

## FAQ

**Should I block GPTBot and the other training crawlers?** It's your call. Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect content for future models. Search and browsing crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages for live answers. Blocking training doesn't stop live answers citing you. Blocking search does.

**What is Google-Extended?** It's a robots.txt token, not a separate crawler. It controls whether Google may use your content for Gemini models. Blocking it doesn't remove you from Google Search or AI Overviews.

**Why does server rendering matter?** Most AI crawlers don't run JavaScript. A client-rendered app shell looks empty to them.