# robots.txt Parser

> robots.txt Parser is a paid API for AI agents from api.strale.io, paid per call via x402, $0.0216/call, status unknown (last checked 2026-10-10).

Fetches and parses a website's robots.txt file, returning structured rules per user-agent, sitemaps, crawl delays, and global block status

## Facts

- Endpoint: GET https://api.strale.io/x402/robots-txt-parse?utm_source=zero.xyz
- Price: $0.0216/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-10
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/api-strale-io-315c3817
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_mzZgwMK3Oa98Rq0y3va8-

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability api-strale-io-315c3817
```

Example prompt: Can you fetch and parse the robots.txt for https://example.com and tell me which paths are blocked, what crawl delays are set, and whether all bots are disallowed?

## When to prefer this

Use this endpoint when you need to programmatically check whether a website permits crawling, understand crawl rules for specific bots, discover sitemaps, or validate that your crawler respects the site's robots.txt directives before scraping.

## Known failure modes

- robots.txt not found at the given domain (404 response)
- Invalid or unreachable URL returns an error
- Domain has no robots.txt file
- Malformed robots.txt that cannot be parsed
- Network timeout reaching the target domain

## How this service works

Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.

## Output

Returns a structured breakdown of the robots.txt file including per-user-agent allow/disallow rules, crawl delays, sitemap URLs, and a boolean flag indicating whether all crawlers are blocked.

## Example request

```json
{
 "url": "https://example.com"
}
```

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "required": [
  "url"
 ],
 "properties": {
  "url": {
   "type": "string"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "rules": {
  "type": "array"
 },
 "exists": {
  "type": "boolean"
 },
 "sitemaps": {
  "type": "array"
 },
 "user_agents": {
  "type": "array"
 },
 "blocks_all_crawlers": {
  "type": "boolean"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/api-strale-io-315c3817/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.strale.io](https://www.zero.xyz/host/api.strale.io/llms.txt)