# agent402.tools Robots.txt Crawl Permission Checker

> agent402.tools Robots.txt Crawl Permission Checker is a paid API for AI agents from agent402.tools, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-14).

Fetches a site's robots.txt and determines whether a given user-agent is permitted to crawl a specific URL path, returning the matched rule and declared sitemaps.

## Facts

- Endpoint: POST https://agent402.tools/api/robots-check
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agent402-tools-robots-txt-crawl-permission-checker-6cd23395
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_a725_Az3cGcZDxmfWFbug

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agent402-tools-robots-txt-crawl-permission-checker-6cd23395 -d '<json body>'
```

Example prompt: Check whether the user-agent 'MyBot' is allowed to crawl the path '/products/deals' on example.com — fetch the robots.txt and tell me the matched rule and any sitemaps declared.

## When to prefer this

Use this endpoint when an agent needs to programmatically verify crawl compliance before scraping or indexing a URL, especially when operating multiple bots with distinct user-agent identifiers. Ideal for respecting crawl ethics at scale without manually parsing robots.txt files. Prefer this over generic HTTP fetch when you also need sitemap discovery in the same call.

## Known failure modes

- robots.txt not found (404) — site may have no robots.txt, permission assumed allowed by convention
- network timeout fetching robots.txt — transient failure, retry recommended
- malformed URL input — returns validation error
- robots.txt is inaccessible due to server error (5xx) — unclear permission state
- user-agent token not found in robots.txt — falls back to wildcard (*) rules

## How this service works

Fetch a site's robots.txt and answer: may this user-agent crawl this path? Returns the matched rule and all declared sitemaps.

## Output

Returns a crawl permission decision (allowed/disallowed), the specific matched rule from robots.txt (e.g. Disallow: /products/), and a list of all sitemap URLs declared in the robots.txt file.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "URL whose path to check"
  },
  "userAgent": {
   "type": "string",
   "description": "User-agent token (default *)"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "allowed": true,
  "sitemaps": [
   "https://example.com/sitemap.xml"
  ],
  "matchedRule": null
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agent402-tools-robots-txt-crawl-permission-checker-6cd23395/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agent402.tools](https://www.zero.xyz/host/agent402.tools/llms.txt)