# Check robots.txt crawl rules

> Check robots.txt crawl rules is a paid API for AI agents from web.openverbs.com, paid per call via x402, $0.006/call, status unknown (last checked 2026-10-27).

Fetches a site's robots.txt and evaluates whether one or more paths are allowed or disallowed for a given user-agent.

## Facts

- Endpoint: POST https://web.openverbs.com/v1/robots?utm_source=zero.xyz
- Price: $0.006/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-27
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/check-robots-txt-crawl-rules-c9fa41e1
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_9jSVbEvBSMI2CMmM66Z0n

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability check-robots-txt-crawl-rules-c9fa41e1 -d '<json body>'
```

Example prompt: Before I scrape https://example.com/blog and /admin, can you check whether those paths are allowed for a user-agent called 'MyBot' according to their robots.txt?

## When to prefer this

Use this endpoint when you need to programmatically verify whether crawling specific paths on a site is permitted before making requests, especially for SEO audits, bot compliance checks, or pre-scrape validation. Prefer it over manually parsing robots.txt when checking multiple paths or multiple user-agents in a single call.

## Known failure modes

- robots.txt not found (site returns 404 or 500 for /robots.txt) — treated as fully allowed per standard convention
- Invalid or unreachable URL returns an error
- URL scheme not http/https is rejected
- More than 50 paths in a single call exceeds the limit
- Malformed user-agent string may cause unexpected matching

## How this service works

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

## Output

The endpoint returns the evaluated allow/disallow status for the primary URL path and any additional paths specified, based on the site's fetched robots.txt rules matched against the given user-agent token.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "format": "uri",
   "maxLength": 2048,
   "description": "Public http(s) URL to check. Its origin's /robots.txt is fetched and its path is the first path evaluated."
  },
  "paths": {
   "type": "array",
   "items": {
    "type": "string",
    "maxLength": 2048,
    "minLength": 1
   },
   "maxItems": 50,
   "description": "Optional extra paths (or same-origin URLs) to check in the same call, e.g. [\"/admin\", \"/search?q=x\"]. Up to 50."
  },
  "userAgent": {
   "type": "string",
   "maxLength": 200,
   "minLength": 1,
   "description": "User-agent token to evaluate rules for, e.g. \"Googlebot\". Defaults to \"*\"."
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/check-robots-txt-crawl-rules-c9fa41e1/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from web.openverbs.com](https://www.zero.xyz/host/web.openverbs.com/llms.txt)