# Robots Directive Parse

> Robots Directive Parse is a paid API for AI agents from creator-desk-x402-isolated.onrender.com, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-13).

Parses and interprets robots.txt directives for a given URL or robots.txt content, returning structured crawl permission data.

## Facts

- Endpoint: POST https://creator-desk-x402-isolated.onrender.com/v1/utility/robots-directive-parse?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-directive-parse-6b1ab932
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_n_g4yHo4q_96ZH4dtAoIQ

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-directive-parse-6b1ab932 -d '<json body>'
```

Example prompt: Can you parse this robots.txt content for me and tell me which paths are disallowed and what the crawl-delay is set to?

## When to prefer this

Use this endpoint when an autonomous agent or crawler needs to programmatically check robots.txt compliance before accessing web resources, or when building SEO tooling that must interpret crawl directives. Prefer this over manual string parsing when you need structured, machine-readable output from raw robots.txt content.

## Known failure modes

- Malformed or empty robots.txt content returns an error or empty result
- Unrecognized directive syntax may be skipped silently
- Network-fetched robots.txt URLs that return non-200 responses may cause fetch failures
- Overly large robots.txt files may time out or be truncated

## How this service works

Machine-callable x402-paid utilities for autonomous agents.

## Output

A JSON object with the parsed robots.txt directive broken into structured components, including user-agent rules, allowed and disallowed path patterns, crawl-delay values, and sitemap URLs where present.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "value": {},
  "options": {
   "type": "object"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "result": "example",
  "operation": "robots_directive_parse"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-directive-parse-6b1ab932/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from creator-desk-x402-isolated.onrender.com](https://www.zero.xyz/host/creator-desk-x402-isolated.onrender.com/llms.txt)