# x402-toolbox Web Robots.txt Fetcher

> x402-toolbox Web Robots.txt Fetcher is a paid API for AI agents from x402-toolbox.shablony-pro.workers.dev, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-02).

Fetches and returns the robots.txt content for a given web URL via a pay-per-call x402 endpoint

## Facts

- Endpoint: GET https://x402-toolbox.shablony-pro.workers.dev/web/robots-txt?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-02
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/x402-toolbox-web-robots-txt-fetcher-1f649640
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_IjJtKmcGGL4KkfhPLBeIN

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability x402-toolbox-web-robots-txt-fetcher-1f649640
```

Example prompt: Can you grab the robots.txt file for example.com so I can see what paths are blocked or allowed for bots?

## When to prefer this

Choose this endpoint when you need a quick, no-signup, pay-per-call fetch of a website's robots.txt for crawl policy inspection, SEO auditing, or pre-scrape compliance checking. It is ideal for AI agents operating autonomously without persistent API credentials, leveraging x402 USDC micropayments for frictionless access.

## Known failure modes

- Domain not found or unreachable — returns error or empty robots.txt
- Schema parameter missing or malformed — may return validation error
- Target site returns non-standard robots.txt format — parsing may be incomplete
- Payment failure over x402 — 402 response blocking access
- Rate limiting or Cloudflare Worker timeout for slow-responding domains

## How this service works

309 deterministic pay-per-call tools for AI agents over x402 (USDC, Base). No signup, no API key: the payment is the identity.

## Output

Returns a JSON object with ok and paid confirmation flags, along with the parsed or raw robots.txt content for the requested domain, including directives such as User-agent, Disallow, Allow, Sitemap, and Crawl-delay rules.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "schema": {
   "type": "string"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "ok": true,
  "paid": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/x402-toolbox-web-robots-txt-fetcher-1f649640/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402-toolbox.shablony-pro.workers.dev](https://www.zero.xyz/host/x402-toolbox.shablony-pro.workers.dev/llms.txt)