# Strale Robots.txt Parser

> Strale Robots.txt Parser is a paid API for AI agents from api.strale.io, paid per call via x402, $0.0216/call, status unknown (last checked 2026-09-15).

Fetches and parses a website's robots.txt file, extracting crawl rules, directives, and allowed/disallowed paths

## Facts

- Endpoint: GET https://api.strale.io/x402/v2/robots-txt-parse
- Price: $0.0216/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/strale-robots-txt-parser-b58d5f44
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_-DD1zRlNQVoFEU5Ho_any

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability strale-robots-txt-parser-b58d5f44
```

Example prompt: Can you fetch and parse the robots.txt file for https://example.com and tell me which paths are disallowed for all bots and whether there are any sitemaps listed?

## When to prefer this

Use this endpoint when you need a structured, auditable parse of a robots.txt file without building your own fetcher and parser. It is especially useful in compliance-conscious pipelines, SEO audits, or pre-crawl authorization checks where you need a trust-layer audit record attached to every result. Prefer it over rolling your own HTTP fetch when you need verifiable, chain-hashed provenance of the parse result.

## Known failure modes

- URL not provided or malformed — request rejected with validation error
- Target site returns non-200 for robots.txt (404, 403, 500) — error returned with HTTP status
- robots.txt is empty or missing standard directives — partial or empty result returned
- Network timeout reaching target domain — timeout error returned
- Payment failure (insufficient USDC balance) — 402 response before execution

## How this service works

The trust layer for AI agents — 250+ independently tested data capabilities across 27 countries. Execute capabilities via REST, MCP, A2A, or x402 micropayments. Every call returns an audit record with cryptographic chain hashing.

## Output

Returns a structured breakdown of the robots.txt file including user-agent rules, disallowed and allowed path lists, crawl-delay values, and any sitemap URLs declared. Each response includes an audit record with cryptographic chain hashing for provenance.

## Request schema (JSON Schema)

```json
{
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "type": "object",
 "properties": {
  "input": {
   "type": "object",
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "type": "string",
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ]
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string"
      }
     }
    }
   },
   "required": [
    "type",
    "method"
   ],
   "additionalProperties": false
  }
 },
 "required": [
  "input"
 ]
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/strale-robots-txt-parser-b58d5f44/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.strale.io](https://www.zero.xyz/host/api.strale.io/llms.txt)