# Mat Web & Data Toolkit – Robots.txt Fetcher

> Mat Web & Data Toolkit – Robots.txt Fetcher is a paid API for AI agents from automatom.matt-d59.workers.dev, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-18).

Fetches and parses a website's robots.txt file, returning crawl rules, allowed/disallowed paths, user-agent groups, crawl delays, and sitemap URLs as structured JSON.

## Facts

- Endpoint: GET https://automatom.matt-d59.workers.dev/robots?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-18
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/mat-web-data-toolkit-robots-txt-fetcher-a283865b
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_hvOPx1lwH5My1Nz2PVWZC

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability mat-web-data-toolkit-robots-txt-fetcher-a283865b
```

Example prompt: Can you fetch and parse the robots.txt file for https://www.example.com and tell me which paths are disallowed for crawlers and what sitemaps are listed?

## When to prefer this

Use this endpoint when you need structured, machine-readable parsing of a site's robots.txt rather than raw text retrieval. It's ideal for pre-scraping compliance checks, SEO audits requiring sitemap discovery, or when you need per-user-agent rule breakdowns without writing your own parser.

## Known failure modes

- robots.txt not found (404) — returns status 404 with empty groups
- website unreachable or timeout — HTTP error propagated in response
- malformed robots.txt — partial parsing with best-effort results
- missing or invalid URL query parameter — request rejected
- site returns non-text content at /robots.txt — parsing may fail

## How this service works

Fetches a public web page and returns its title, meta description, outbound links and readable plain text as JSON.

## Output

A JSON object containing: the parsed robots.txt groups (each with user_agents, allow paths, disallow paths, and crawl_delay), the HTTP status code of the robots.txt fetch, a list of sitemap URLs declared in the file, and the canonical URL of the robots.txt that was fetched.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "groups": [
   {
    "allow": [
     "/search/about",
     "/search/howsearchworks"
    ],
    "disallow": [
     "/search",
     "/sdch"
    ],
    "crawl_delay": null,
    "user_agents": [
     "*",
     "Yandex"
    ]
   },
   {
    "allow": [
     "/maps/api/js"
    ],
    "disallow": [
     "/maps/api/js/",
     "/maps/api/place/js/"
    ],
    "crawl_delay": null,
    "user_agents": [
     "AdsBot-Google"
    ]
   }
  ],
  "status": 200,
  "sitemaps": [
   "https://www.google.com/sitemap.xml"
  ],
  "robots_url": "https://www.google.com/robots.txt"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/mat-web-data-toolkit-robots-txt-fetcher-a283865b/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from automatom.matt-d59.workers.dev](https://www.zero.xyz/host/automatom.matt-d59.workers.dev/llms.txt)