# AI Crawler Policy Checker

> AI Crawler Policy Checker is a paid API for AI agents from intel.rallylive.ca, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-14).

Reads a domain's robots.txt and reports whether 20+ known AI/LLM crawlers are allowed, blocked, or unmentioned, plus an overall AI access stance and TDM/ai.txt hints.

## Facts

- Endpoint: GET https://intel.rallylive.ca/dev/ai-crawler-policy
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/ai-crawler-policy-checker-a60abd16
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_UFa6EA74vKxeahX09ruku

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability ai-crawler-policy-checker-a60abd16
```

Example prompt: Can you check openai.com's AI crawler policy — specifically which AI bots like GPTBot, ClaudeBot, and PerplexityBot are allowed or blocked in its robots.txt, and what its overall AI access stance is?

## When to prefer this

Use this endpoint when you need a structured, per-crawler breakdown of a domain's AI access policy based on robots.txt, especially when you need to distinguish between specific crawlers (e.g. GPTBot vs ClaudeBot) rather than a generic crawl-permission check. Prefer this over manually parsing robots.txt when you need the overall AI stance classification (open/selective/blocks-all-ai) or TDM/ai.txt signal in one call.

## Known failure modes

- Domain does not have a robots.txt — crawlers reported as unmentioned with no stance
- Domain is unreachable or returns non-200 HTTP status — endpoint may return error or empty result
- Invalid or malformed domain input — request fails with validation error
- robots.txt is present but uses non-standard syntax — some rules may be misclassified
- Rate limiting on the target domain may prevent robots.txt fetch

## How this service works

AI-crawler policy of a website: reads robots.txt and reports, for 20 known AI/LLM crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, PerplexityBot, Bytespider, Amazonbot, cohere-ai, meta-externalagent and more), whether the root is allowed, blocked or unmentioned, plus an overall stance (open / selective / blocks-all-ai) and TDM/ai.txt hints. $0.01 per domain.

## Output

Returns per-crawler permission status (allowed, blocked, or unmentioned) for up to 20 known AI/LLM crawlers including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, PerplexityBot, Bytespider, Amazonbot, cohere-ai, meta-externalagent, and more. Also returns an overall AI access stance (open, selective, or blocks-all-ai) and any TDM or ai.txt hints found on the domain.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "properties": {}
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawler-policy-checker-a60abd16/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from intel.rallylive.ca](https://www.zero.xyz/host/intel.rallylive.ca/llms.txt)