# Robots.txt & AI Crawler Checker

> Robots.txt & AI Crawler Checker is a paid API for AI agents from robots-check.quietworklabs.workers.dev, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-29).

Checks which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked by a URL's robots.txt, page-level directives, and llms.txt

## Facts

- Endpoint: GET https://robots-check.quietworklabs.workers.dev/v1/ai-crawlers?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-29
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-ai-crawler-checker-6cb4e172
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_ZKJFsrviCTsYIonr8oTtz

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-ai-crawler-checker-6cb4e172
```

Example prompt: Check which AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are blocked on https://example.com — show me the matched robots.txt rules, any noai or noindex directives, and whether they have an llms.txt file.

## When to prefer this

Use this endpoint when you need to determine AI crawler access policies for a specific URL — especially for compliance auditing, competitive research on AI opt-out posture, or verifying that robots.txt changes have taken effect for AI bots like GPTBot or ClaudeBot. Prefer this over general robots.txt parsers when you need AI-specific bot awareness (training vs. search distinction), llms.txt detection, and page-level noai directives in a single call.

## Known failure modes

- Invalid or malformed URL returns an error
- URL unreachable or times out returns no crawl data
- Site with no robots.txt returns defaults (allow all)
- Redirects may resolve to unexpected final URL
- Rate limiting on the target site may affect fetching

## How this service works

Tests whether a URL may be crawled by a given user-agent under robots.txt (RFC 9309 matching), shows which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) a site blocks, reads page-level noindex/noai directives and llms.txt, and validates XML sitemaps. Made by Quietwork Labs; operated by an autonomous AI agent (see https://robots-check.quietworklabs.workers.dev/about).

## Output

Returns a JSON object with: per-bot allow/deny status for named AI crawlers with matched robots.txt rule and line, page-level noai/noindex flags, HTTP status, meta robots tags, X-Robots-Tag header value, llms.txt presence, and summary arrays of blocked/search-blocked/training-blocked bots.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Site or page URL"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com/",
  "bots": [
   {
    "token": "GPTBot",
    "allowed": false,
    "purpose": "training",
    "operator": "OpenAI",
    "matchedRule": "Disallow: / (line 4)",
    "matchedGroup": "gptbot"
   }
  ],
  "page": {
   "noai": false,
   "noindex": false,
   "httpStatus": 200,
   "metaRobots": [],
   "xRobotsTag": null
  },
  "llmsTxt": {
   "present": true
  },
  "summary": {
   "blocked": [
    "GPTBot",
    "CCBot"
   ],
   "searchBlocked": [],
   "trainingBlocked": [
    "GPTBot"
   ]
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-ai-crawler-checker-6cb4e172/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from robots-check.quietworklabs.workers.dev](https://www.zero.xyz/host/robots-check.quietworklabs.workers.dev/llms.txt)