# robots.txt / llms.txt Fetcher and Parser

> robots.txt / llms.txt Fetcher and Parser is a paid API for AI agents from mcpfax-utility.bowling-anthony.workers.dev, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-15).

Fetches and parses a website's robots.txt and llms.txt files, returning structured crawl rules and LLM access directives.

## Facts

- Endpoint: GET https://mcpfax-utility.bowling-anthony.workers.dev/v1/robots
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-llms-txt-fetcher-and-parser-4672e643
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_bNE3upcMLcRMVnabVuTLD

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-llms-txt-fetcher-and-parser-4672e643
```

Example prompt: Can you fetch and parse the robots.txt and llms.txt files for openai.com and tell me which paths are disallowed and whether they have any LLM-specific access rules?

## When to prefer this

Use this endpoint when you need to programmatically check a site's crawl permissions or LLM access policy before scraping, indexing, or training on its content. Prefer this over manual fetching when you need structured, parsed output from both robots.txt and llms.txt in a single call.

## Known failure modes

- Site does not have a robots.txt (returns 404 or empty)
- Site does not have an llms.txt file
- Domain is unreachable or times out
- Malformed robots.txt that cannot be parsed
- Invalid or non-existent domain provided

## How this service works

robots.txt / llms.txt — Fetch and parse a site's robots.txt and llms.txt.

## Output

Returns the raw and parsed contents of the site's robots.txt and llms.txt files, including structured crawl directives (allow/disallow rules, user-agent groups, crawl-delay) and any LLM-specific access policies defined in llms.txt.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object"
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-llms-txt-fetcher-and-parser-4672e643/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from mcpfax-utility.bowling-anthony.workers.dev](https://www.zero.xyz/host/mcpfax-utility.bowling-anthony.workers.dev/llms.txt)