# x402.forgemesh.io Robots.txt AI Permission Checker

> x402.forgemesh.io Robots.txt AI Permission Checker is a paid API for AI agents from x402.forgemesh.io, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-15).

Fetches and parses a domain's robots.txt file, extracting AI-specific directives (search, ai-input, ai-train) into structured JSON with a plain-English summary of what scraping or training is permitted.

## Facts

- Endpoint: POST https://x402.forgemesh.io/robots-txt-ai-check
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/x402-forgemesh-io-robots-txt-ai-permission-checker-83f13163
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_lkMaR6NvV1YHNhBO3CULI

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability x402-forgemesh-io-robots-txt-ai-permission-checker-83f13163 -d '<json body>'
```

Example prompt: Before I scrape techcrunch.com for my RAG index, check its robots.txt to see whether AI crawling, ai-input use, and ai-train use are actually permitted — give me both the structured breakdown and a plain-English summary.

## When to prefer this

Use this endpoint as a pre-crawl compliance check before any web scraping, RAG indexing, or AI training data collection workflow. It is specifically designed to parse emerging AI-preference and Content-Signal directives that generic robots.txt parsers do not handle, making it the right choice when you need to confirm AI-specific permissions rather than just standard Googlebot rules.

## Known failure modes

- Domain does not have a robots.txt file — returns empty or default permissive interpretation
- Domain is unreachable or returns non-200 — fetch error reported
- Malformed robots.txt with unrecognized directive syntax — partial parse with warnings
- Invalid domain format in input — validation error returned
- No AI-specific directives present — summary reflects generic crawl rules only

## How this service works

Fetches a domain's robots.txt and parses its Content-Signal and AI-preference directives (search, ai-input, ai-train) into structured JSON plus a plain-English summary of what's allowed — the "can this domain be crawled by an AI agent" check. Run before scraping content or building a RAG index, ahead of the industry's move toward stricter bot-gating defaults later this year.

## Output

Returns structured JSON containing parsed robots.txt directives for Content-Signal and AI-preference fields (search, ai-input, ai-train), along with a plain-English summary of what is allowed or disallowed for AI agents, scrapers, and indexers.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "domain": {
   "type": "string",
   "description": "Domain to check, e.g. example.com"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "domain": "theverge.com",
  "signals": {
   "search": "yes",
   "ai-input": "no",
   "ai-train": "no"
  },
  "declared": true,
  "interpretation": "site declares: search=yes, ai-input=no, ai-train=no"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/x402-forgemesh-io-robots-txt-ai-permission-checker-83f13163/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.forgemesh.io](https://www.zero.xyz/host/x402.forgemesh.io/llms.txt)