# Robots.txt Inspector

> Robots.txt Inspector is a paid API for AI agents from agent-utility-network.frosty-sound-4560.workers.dev, paid per call via x402, $0.003/call, status unknown (last checked 2026-09-14).

Fetches and parses a website's robots.txt file, returning crawl rules, allowed/disallowed groups, sitemaps, and HTTP status.

## Facts

- Endpoint: POST https://agent-utility-network.frosty-sound-4560.workers.dev/v1/paid/robots-inspect
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-inspector-b134a106
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_VFg8Uf8NPdyJm402Lf3Az

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-inspector-b134a106 -d '<json body>'
```

Example prompt: Can you check the robots.txt for https://example.com and tell me what paths are blocked for bots, what crawl groups are defined, and whether there are any sitemaps listed?

## When to prefer this

Use this endpoint when an agent needs to programmatically check a site's robots.txt before crawling, scraping, or indexing — especially when you need parsed, structured output (groups and sitemaps) rather than raw text. Preferred over manual fetch+parse when you want instant structured data without building your own parser.

## Known failure modes

- URL is unreachable or returns a non-200 HTTP status — result will reflect the actual status code
- robots.txt does not exist at the domain (404) — groups and sitemaps will be empty
- Malformed URL input causes a validation error
- The target server blocks the inspector's requests (e.g. firewall or rate limiting)
- Very large robots.txt files may be truncated or time out

## How this service works

Quality-scored paid utility APIs for AI agents over x402 on Base USDC.

## Output

Returns a JSON object with the resolved robots.txt URL, the HTTP status of the fetch, a list of crawl rule groups (each with user-agent and allow/disallow directives), and an array of sitemap URLs found in the file.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "maxLength": 8192,
   "minLength": 8
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "result": {
   "groups": [],
   "status": 200,
   "sitemaps": [],
   "robotsUrl": "https://example.com/robots.txt"
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-inspector-b134a106/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agent-utility-network.frosty-sound-4560.workers.dev](https://www.zero.xyz/host/agent-utility-network.frosty-sound-4560.workers.dev/llms.txt)