# AI Crawler Access Checker

> AI Crawler Access Checker is a paid API for AI agents from x402.agentharbor.agency, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-15).

Given a domain, checks which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) are allowed or blocked by its robots.txt, and reports sitemaps, llms.txt presence, and noindex/nosnippet signals.

## Facts

- Endpoint: GET https://x402.agentharbor.agency/ai-crawlers?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/ai-crawler-access-checker-00314844
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_nQvoeiMl5CQxLVSZwm0_L

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability ai-crawler-access-checker-00314844
```

Example prompt: Can you check which AI crawlers are allowed or blocked on openai.com — I want to know if GPTBot, ClaudeBot, and PerplexityBot can access it, and whether the site has an llms.txt file?

## When to prefer this

Use this endpoint when you need a fast, static check of AI crawler permissions for any public domain without spinning up a browser. Ideal for auditing AI search visibility, diagnosing indexing issues, or monitoring competitors' robots.txt policies. Prefer this over general SEO audit tools when specifically targeting AI crawler access rules and llms.txt presence.

## Known failure modes

- robots.txt unreachable or blocked — endpoint returns no charge and indicates the file could not be read
- invalid domain format — rejected by schema pattern validation
- domain resolves but returns no robots.txt — treated as unreadable
- network timeout on robots.txt fetch — no charge applied

## How this service works

Given a website domain, reports which AI crawlers may fetch its home page under its robots.txt (RFC 9309): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more, each allowed or blocked, plus Googlebot, listed sitemaps, whether /llms.txt is published and page-level noindex/nosnippet, with fixes. Static check, no browser. Not charged if robots.txt cannot be read (no response, or a bot block).

## Output

Returns per-crawler allow/block status for major AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, and others), Googlebot status, listed sitemaps, whether /llms.txt is published, page-level noindex/nosnippet flags, and suggested fixes. No charge if robots.txt cannot be read.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain"
     ],
     "properties": {
      "domain": {
       "type": "string",
       "pattern": "^[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+$",
       "maxLength": 253,
       "description": "Bare public hostname, e.g. example.com"
      }
     },
     "additionalProperties": false
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawler-access-checker-00314844/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.agentharbor.agency](https://www.zero.xyz/host/x402.agentharbor.agency/llms.txt)