# AI Crawler Robots.txt Access Matrix

> AI Crawler Robots.txt Access Matrix is a paid API for AI agents from api.agentstools.dev, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-15).

Audits a website's robots.txt to report whether the 2026 cohort of AI/LLM crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are allowed, blocked, or unlisted, plus sitemaps and a score.

## Facts

- Endpoint: GET https://api.agentstools.dev/aeo/crawlers
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/ai-crawler-robots-txt-access-matrix-52e61983
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_EwR2QWaTsO_wBxo1Q17vg

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability ai-crawler-robots-txt-access-matrix-52e61983
```

Example prompt: Can you check whether openai.com allows or blocks the major AI crawlers like GPTBot, ClaudeBot, and PerplexityBot — and tell me if there's a default block rule or any sitemaps declared?

## When to prefer this

Use this endpoint when you need a structured, machine-readable summary of robots.txt AI crawler permissions for a specific website, especially when you want per-bot granularity across the full 2026 cohort (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) rather than just a raw robots.txt fetch. Ideal for AEO audits, competitive research on crawler access policies, or verifying your own site's AI indexing posture in a single call.

## Known failure modes

- Target URL is unreachable or returns non-200 status — endpoint may return an error or empty result
- Domain has no robots.txt — reported as absent with no bot data
- Malformed URL or hostname input — validation error returned
- Rate limit or payment failure for x402 micropayment — transaction rejected

## How this service works

robots.txt AI-crawler access matrix: for the 2026 cohort of AI and answer-engine crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and more) reports allowed, blocked or unlisted per bot, whether the default rule blocks everything, declared sitemaps and a score. One network operation.

## Output

Returns a per-bot access matrix for the 2026 cohort of AI/LLM crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, and more), each with an allowed/blocked/unlisted status, whether the site's default rule blocks everything, a list of declared sitemaps, and an overall AI-crawlability score — all retrieved in a single network operation.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [],
     "properties": {
      "url": {
       "type": "string",
       "description": "Target URL to audit (e.g. https://example.com)"
      },
      "host": {
       "type": "string",
       "description": "Alternative input: a hostname"
      },
      "domain": {
       "type": "string",
       "description": "Alternative input: a bare domain (e.g. example.com)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawler-robots-txt-access-matrix-52e61983/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.agentstools.dev](https://www.zero.xyz/host/api.agentstools.dev/llms.txt)