# Robots.txt Path Permission Checker

> Robots.txt Path Permission Checker is a paid API for AI agents from agentshelf.syntexa.ch, paid per call via x402, $0.003/call, status unknown (last checked 2026-10-01).

Checks whether a given URL path is allowed for a specified user agent according to the site's robots.txt rules, with SSRF protection.

## Facts

- Endpoint: POST https://agentshelf.syntexa.ch/v1/robots-allowed?utm_source=zero.xyz
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-01
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-path-permission-checker-7dff0c7e
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_QEsKv3x3ZAViZ559kFDtZ

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-path-permission-checker-7dff0c7e -d '<json body>'
```

Example prompt: Before scraping the product pages, check if robots.txt on https://example.com allows my bot (user agent 'MyBot/1.0') to access the path /products/.

## When to prefer this

Use this endpoint when an agent needs to check robots.txt compliance before crawling or scraping a URL, especially when SSRF safety is required. Prefer the free sandbox POST /v1/sandbox/robots-allowed for testing; use this paid endpoint for production compliance checks at $0.003 USDC per call.

## Known failure modes

- robots.txt file not found or unreachable — may return an error or default allow
- Invalid or malformed URL input returns validation error
- SSRF-blocked internal/private IP targets return an error
- Path or user_agent exceeding max length limits rejected
- Network timeout reaching the target site's robots.txt

## How this service works

Call when an agent needs a path checked against a site's robots.txt (SSRF-safe). Exact $0.003 USDC. Prefer unpaid POST /v1/sandbox/robots-allowed first.

## Output

Returns whether the specified path is allowed or disallowed for the given user agent based on the site's robots.txt, effectively indicating crawl permission status.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "examples": [
    "https://example.com/docs"
   ],
   "maxLength": 2048,
   "minLength": 8,
   "description": "Public page or origin. robots.txt is fetched from this origin."
  },
  "path": {
   "type": "string",
   "maxLength": 2048,
   "minLength": 1,
   "description": "Path to test, such as /docs. Omit it to use the path from url."
  },
  "user_agent": {
   "type": "string",
   "examples": [
    "*"
   ],
   "maxLength": 200,
   "minLength": 1,
   "description": "User-agent token to match in robots.txt. Default *."
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-path-permission-checker-7dff0c7e/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agentshelf.syntexa.ch](https://www.zero.xyz/host/agentshelf.syntexa.ch/llms.txt)