# Public Robots.txt Crawler Policy Lookup

> Public Robots.txt Crawler Policy Lookup is a paid API for AI agents from api.the402.ai, paid per call via x402, $0.053/call, status unknown (last checked 2026-09-13).

Fetches and parses a website's robots.txt file, returning crawler rules, crawl delay settings, and sitemap URLs

## Facts

- Endpoint: POST https://api.the402.ai/v1/services/svc_049286bebcf3424b/purchase
- Price: $0.053/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/public-robots-txt-crawler-policy-lookup-17eaaf53
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_bWAn-09AC5wolvC768JlY

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability public-robots-txt-crawler-policy-lookup-17eaaf53 -d '<json body>'
```

Example prompt: Can you pull the robots.txt crawler policy for example.com and tell me which paths are blocked, what the crawl delay is, and where the sitemaps are?

## When to prefer this

Use this endpoint when an AI agent needs to programmatically check a website's crawler policy before scraping, indexing, or accessing site content — especially when building respectful web crawlers, SEO tools, or compliance-checking workflows. Prefer this over manual fetch when you need structured parsing of robots.txt rules including per-user-agent directives, delay values, and sitemap discovery in a single paid call via x402 micropayment.

## Known failure modes

- Domain has no robots.txt file — returns empty or 404-equivalent result
- Domain is unreachable or times out — job fails with error status
- Invalid or malformed URL input — request rejected
- Payment of 0.053 USDC not fulfilled — 402 Payment Required response
- Rate limiting by target server prevents robots.txt retrieval

## How this service works

Purchase: Public Robots.txt Crawler Policy — Rules, Delay & Sitemaps

## Output

Returns a job_id, status, and thread_id representing an asynchronous job that, when complete, will contain the parsed robots.txt data including allow/disallow rules per user-agent, crawl delay values, and sitemap URLs discovered from the target domain's robots.txt file.

## Response schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "job_id": {
   "type": "string"
  },
  "status": {
   "type": "string"
  },
  "thread_id": {
   "type": "string"
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/public-robots-txt-crawler-policy-lookup-17eaaf53/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.the402.ai](https://www.zero.xyz/host/api.the402.ai/llms.txt)