# Swift Bot Labs Toolbox – Web Robots.txt Checker

> Swift Bot Labs Toolbox – Web Robots.txt Checker is a paid API for AI agents from x402-toolbox.swiftbotlab.workers.dev, paid per call via x402, $0.005/call, status unknown (last checked 2026-10-31).

Fetches a site's robots.txt and checks whether a given URL path is allowed or disallowed for a specified user agent

## Facts

- Endpoint: GET https://x402-toolbox.swiftbotlab.workers.dev/v1/web/robots?utm_source=zero.xyz
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-31
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/swift-bot-labs-toolbox-web-robots-txt-checker-b36652ae
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_veXCyK-EoSW7sCtrl568T

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability swift-bot-labs-toolbox-web-robots-txt-checker-b36652ae
```

Example prompt: Can you check whether GPTBot is allowed to crawl https://www.example.com/search?q=test — look up the site's robots.txt and tell me if that path is permitted or blocked, and what rule matched.

## When to prefer this

Use this endpoint when you need a fast, deterministic, pay-per-call check of robots.txt permissions for a specific URL and user agent, with no API key or account required. Prefer it over rolling your own robots.txt parser when you need a verified, structured result including matched rule, sitemaps, and allow/disallow decision in a single call.

## Known failure modes

- URL unreachable or domain does not exist — ok:false with error message
- robots.txt returns non-200 HTTP status — treated as allow-all or error
- malformed URL input — validation error
- No robots.txt present — typically treated as allow-all
- Rate limiting or timeout on the target server — connection error returned

## How this service works

62 small, deterministic utility APIs (DNS and e-mail checks, web extraction, EVM decoding, x402 checks, IP math, JSON/CSV, hashing, conversions). No account or API key: pay per call in USDC with x402.

## Output

A JSON object with a boolean 'allowed' field indicating whether the path is permitted for the given user agent, the matched robots.txt rule string, the URL of the robots.txt file that was read, the user agent string that was matched, and an array of sitemap URLs declared in the file.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD"
     ],
     "type": "string"
    },
    "headers": {
     "type": "object",
     "additionalProperties": {
      "type": "string"
     }
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Page URL (its origin's robots.txt is read)"
      },
      "userAgent": {
       "type": "string"
      }
     },
     "additionalProperties": false
    },
    "discoverable": {
     "type": "boolean"
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "ok": true,
  "tool": "web-robots",
  "result": {
   "path": "/search?q=x",
   "allowed": false,
   "sitemaps": [
    "https://www.google.com/sitemap.xml"
   ],
   "robotsUrl": "https://www.google.com/robots.txt",
   "userAgent": "GPTBot",
   "matchedRule": "Disallow: /search"
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/swift-bot-labs-toolbox-web-robots-txt-checker-b36652ae/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402-toolbox.swiftbotlab.workers.dev](https://www.zero.xyz/host/x402-toolbox.swiftbotlab.workers.dev/llms.txt)