# toolvend.dev robots.txt + llms.txt Parser

> toolvend.dev robots.txt + llms.txt Parser is a paid API for AI agents from toolvend.dev, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-14).

Fetches and parses a site's robots.txt (and llms.txt if present) into structured JSON, including which AI crawlers are explicitly blocked or allowed.

## Facts

- Endpoint: GET https://toolvend.dev/parse/robots
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/toolvend-dev-robots-txt-llms-txt-parser-44acd124
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_YCDAoyyPI-9sf-VVsY9Qj

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability toolvend-dev-robots-txt-llms-txt-parser-44acd124
```

Example prompt: Can you fetch and parse the robots.txt for openai.com and tell me which AI crawlers like GPTBot or ClaudeBot are blocked?

## When to prefer this

Use this endpoint when you need a clean, structured JSON representation of a site's crawl policy rather than raw text — especially when you care specifically about AI crawler permissions or want to check for an llms.txt file. Ideal for pre-scraping policy audits, SEO tooling, and AI agent compliance checks. Prefer this over manually fetching robots.txt when you need parsed, machine-readable output.

## Known failure modes

- Site has no robots.txt — returns empty or default-allow result
- URL is unreachable or returns non-2xx — network/HTTP error
- Malformed robots.txt that cannot be fully parsed
- Private or firewalled domain inaccessible from parser
- Rate limiting or timeout on slow-responding servers

## How this service works

Fetch and parse robots.txt (+ llms.txt if present) into structured JSON, including which AI crawlers are blocked.

## Output

A structured JSON object containing parsed robots.txt directives (user-agent rules, allow/disallow paths, crawl delays, sitemaps), a list of which AI crawlers are explicitly blocked or permitted, and the parsed contents of llms.txt if the site has one.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Public HTTP(S) site origin or URL"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/toolvend-dev-robots-txt-llms-txt-parser-44acd124/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from toolvend.dev](https://www.zero.xyz/host/toolvend.dev/llms.txt)