# AI Crawl Permission Checker

> AI Crawl Permission Checker is a paid API for AI agents from x402.forgemesh.io, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-13).

Checks whether a domain's robots.txt policy permits AI systems to read, train on, or index its content, returning structured JSON with a human-readable interpretation.

## Facts

- Endpoint: POST https://x402.forgemesh.io/ai-crawl-permission-check
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/ai-crawl-permission-checker-2bb35750
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_ePC934V4V5WP7alrEYNJh

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability ai-crawl-permission-checker-2bb35750 -d '<json body>'
```

Example prompt: Before we scrape openai.com for our training dataset, check whether its robots.txt policy actually permits AI crawlers to access and train on its content.

## When to prefer this

Use this endpoint when building scraping agents, dataset pipelines, or RAG systems that need to programmatically verify whether a target domain's robots.txt policy permits AI access before ingesting content. Prefer this over manual inspection when processing multiple domains or automating compliance checks for AI training data collection.

## Known failure modes

- Domain not found or unreachable — robots.txt cannot be fetched
- Domain provided without valid format — returns validation error
- robots.txt absent — may return a default permissive interpretation or indicate no policy found
- Ambiguous directives — some robots.txt rules may not clearly apply to AI agents, leading to uncertain interpretation
- Network timeout — request to fetch robots.txt times out

## How this service works

Checks whether a domain's published policy permits AI systems to read, train on, or index its content, parsing the relevant robots.txt directives into structured JSON with a human-readable interpretation. Built for scraping agents, dataset builders, and RAG pipelines that need to verify permission before ingesting a site's pages.

## Output

A structured JSON object containing the parsed robots.txt AI-related directives for the queried domain, a permission status indicating whether AI systems are allowed to read, train on, or index the site, and a human-readable summary of the policy interpretation.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "domain": {
   "type": "string",
   "description": "Domain to check, e.g. example.com"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "domain": "theverge.com",
  "signals": {
   "search": "yes",
   "ai-input": "no",
   "ai-train": "no"
  },
  "declared": true,
  "interpretation": "site declares: search=yes, ai-input=no, ai-train=no"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawl-permission-checker-2bb35750/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.forgemesh.io](https://www.zero.xyz/host/x402.forgemesh.io/llms.txt)