# relay402 Robots.txt Check

> relay402 Robots.txt Check is a paid API for AI agents from relay402.georgespring.workers.dev, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-14).

Fetches and parses a website's robots.txt to determine crawl permissions for a given host, path, and user agent before acting on that site.

## Facts

- Endpoint: GET https://relay402.georgespring.workers.dev/api/robots-check
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/relay402-robots-txt-check-5e8b4751
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_yh8R6nKcXiB8ag-kmFx07

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability relay402-robots-txt-check-5e8b4751
```

Example prompt: Before scraping docs.example.com, check whether the path /pricing is allowed for a generic crawler bot by looking up their robots.txt.

## When to prefer this

Use this endpoint when an AI agent or automated crawler needs to verify crawl permissions before accessing a website, especially in agentic workflows that must respect site policies. Prefer this over manual robots.txt parsing when you need a pay-per-call, keyless solution that works in autonomous pipelines without API key management.

## Known failure modes

- Host does not have a robots.txt file — returns a permissive default or 404 signal
- Invalid hostname format rejected by schema pattern validation
- Host is unreachable or times out — network error response
- Path or user_agent exceeds maximum length constraints

## How this service works

Checks whether a given user agent may fetch a given path on a host, per the site's robots.txt (specific-agent groups, wildcard rules, Allow/Disallow longest-match). Also returns crawl-delay, sitemap URLs and whether the site publishes llms.txt or ai.txt. For scraping and research agents that want to crawl politely and prove compliance.

## Output

Returns parsed robots.txt rules for the specified host, indicating whether the given path is allowed or disallowed for the specified user agent, along with the relevant directives from the robots.txt file.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "$schema": "https://json-schema.org/draft/2020-12/schema",
     "required": [
      "host"
     ],
     "properties": {
      "host": {
       "type": "string",
       "pattern": "^[a-z0-9]([a-z0-9.-]{2,251})[a-z0-9]$"
      },
      "path": {
       "type": "string",
       "maxLength": 500
      },
      "user_agent": {
       "type": "string",
       "maxLength": 100
      }
     },
     "additionalProperties": false
    }
   },
   "additionalProperties": false
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/relay402-robots-txt-check-5e8b4751/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from relay402.georgespring.workers.dev](https://www.zero.xyz/host/relay402.georgespring.workers.dev/llms.txt)