# SiteCheck AI Crawler Policy Checker

> SiteCheck AI Crawler Policy Checker is a paid API for AI agents from api.sitecheck-api.workers.dev, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-15).

Checks whether a given webpage allows or blocks AI crawlers by inspecting its robots.txt, meta robots tags, X-Robots-Tag headers, and llms.txt file

## Facts

- Endpoint: GET https://api.sitecheck-api.workers.dev/api/ai-crawlers?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sitecheck-ai-crawler-policy-checker-23beac3c
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_aUbLMEs3ssTxAx0OPsT4Z

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sitecheck-ai-crawler-policy-checker-23beac3c
```

Example prompt: Can you check whether https://developers.cloudflare.com/ allows AI crawlers like GPTBot and ChatGPT-User to access it, including what its robots.txt and llms.txt say?

## When to prefer this

Choose this endpoint when you need to programmatically verify whether a specific webpage or domain permits AI crawlers before scraping, indexing, or training on its content. It is the right choice when you need granular per-bot permission data (GPTBot, ChatGPT-User, OAI-SearchBot) and want to check both robots.txt and llms.txt in one call. Prefer it over generic robots.txt parsers when you specifically care about AI crawler opt-out signals (noai, noimageai) and meta/header-level directives.

## Known failure modes

- URL not reachable or returns non-200 — robots.txt may be assumed absent
- Malformed URL input causes validation error
- Site uses JavaScript-rendered robots directives that static HTML parsing misses (no JS execution)
- robots.txt exists but is malformed or uses non-standard syntax — partial parsing may occur
- Rate limits or network timeouts on the target server may cause incomplete results

## How this service works

For eleven AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others): allowed or blocked for the page's path under robots.txt, plus the noai and noimageai opt-outs in meta and X-Robots-Tag, and whether llms.txt exists. Reads the page as static HTML; no JavaScript runs.

## Output

Returns a JSON object containing the page URL, whether llms.txt exists, a list of named AI crawlers (e.g. GPTBot, ChatGPT-User, OAI-SearchBot) with their allowed/blocked status, robots.txt existence, opt-out flags (noai, noimageai, metaRobots, xRobotsTag), the data source (robots.txt + static HTML), and a timestamp of when the check was performed.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Page address, https://…"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://developers.cloudflare.com/",
  "path": "/",
  "optOut": {
   "noai": false,
   "noimageai": false,
   "metaRobots": null,
   "xRobotsTag": null
  },
  "source": "https://developers.cloudflare.com/robots.txt and the page (static HTML, no JavaScript run)",
  "llmsTxt": true,
  "crawlers": [
   {
    "name": "GPTBot",
    "allowed": true
   },
   {
    "name": "ChatGPT-User",
    "allowed": true
   },
   {
    "name": "OAI-SearchBot",
    "allowed": true
   }
  ],
  "checkedAt": "2026-10-14T07:09:42.733Z",
  "robotsTxt": {
   "exists": true
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sitecheck-ai-crawler-policy-checker-23beac3c/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.sitecheck-api.workers.dev](https://www.zero.xyz/host/api.sitecheck-api.workers.dev/llms.txt)