# Robots.txt Fetcher and Parser

> Robots.txt Fetcher and Parser is a paid API for AI agents from api.x402node.dev, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-14).

Fetches and parses robots.txt for a given domain, returning allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content.

## Facts

- Endpoint: GET https://api.x402node.dev/web/robots
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/api-x402node-dev-394ad214
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_lSLlDm-1ibXkFdO6u2v4H

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability api-x402node-dev-394ad214
```

Example prompt: Can you fetch and parse the robots.txt for example.com and tell me which paths are disallowed for all crawlers, what the crawl delay is, and list any sitemap URLs?

## When to prefer this

Use this endpoint when an AI agent needs to plan ethical web crawling by checking a site's crawl permissions before scraping, or when performing an SEO audit that requires inspecting robots.txt directives. Prefer this over manual fetching when structured parsing of allow/disallow rules, sitemap discovery, and crawl-delay extraction is needed in a single call.

## Known failure modes

- Domain not found or unreachable — network or DNS error returned
- No robots.txt file exists at the domain — empty or 404 response
- Malformed robots.txt that cannot be parsed — partial or empty result
- Invalid domain parameter format — validation error returned
- Timeout fetching the remote file — timeout error returned

## How this service works

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.

## Output

Returns a structured breakdown of the robots.txt file including: allow and disallow rules grouped by user-agent, sitemap URLs, crawl-delay values per agent, and the raw robots.txt text content.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "properties": {
      "domain": {
       "type": "string",
       "description": "Domain name (optional)"
      }
     }
    }
   },
   "additionalProperties": false
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/api-x402node-dev-394ad214/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.x402node.dev](https://www.zero.xyz/host/api.x402node.dev/llms.txt)