# AI Crawler Policy Checker

> AI Crawler Policy Checker is a paid API for AI agents from agentx.sale, paid per call via x402, $0.02/call, status unknown (last checked 2026-10-21).

Returns which AI web crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.) a given domain blocks, along with TLS and security header status.

## Facts

- Endpoint: GET https://agentx.sale/x402/crawler-policy?utm_source=zero.xyz
- Price: $0.02/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-21
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/ai-crawler-policy-checker-651c29df
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_uC1FvvMCgFYPSRi24xAuh

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability ai-crawler-policy-checker-651c29df
```

Example prompt: Can you check what AI crawlers openai.com blocks — specifically whether GPTBot, ClaudeBot, and PerplexityBot are allowed or denied?

## When to prefer this

Use this endpoint when you need to know a specific domain's stance on AI crawler access before indexing, training, or auditing agent-readiness. It is the right choice when you want a combined view of TLS health and AI bot policy in a single call, rather than manually parsing robots.txt or running separate security scans.

## Known failure modes

- Domain does not exist or is unreachable — returns error or empty result
- Robots.txt absent — may report all crawlers as implicitly allowed
- Timeout fetching remote domain — returns error response
- Invalid host parameter — returns validation error

## How this service works

TLS + security headers + AI-crawler policy card: which AI crawlers a site blocks (GPTBot, ClaudeBot, Google-Extended, PerplexityBot...)

## Output

A policy card showing per-crawler allow/block status for major AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.), along with TLS certificate validity and security header presence for the queried domain.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "host"
     ],
     "properties": {
      "host": {
       "type": "string"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "host": "example.com",
  "gptbot": "allowed",
  "claudebot": "blocked"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawler-policy-checker-651c29df/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agentx.sale](https://www.zero.xyz/host/agentx.sale/llms.txt)