# robots-allowed-for-bot

> robots-allowed-for-bot is a paid API for AI agents from intel.rallylive.ca, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-14).

Checks whether a named bot (e.g. GPTBot, ClaudeBot, Googlebot) is allowed to fetch a specific URL by evaluating the site's robots.txt rules, returning allow/disallow status, matching rule, crawl-delay, and group membership.

## Facts

- Endpoint: GET https://intel.rallylive.ca/dev/robots-for-bot
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-allowed-for-bot-93ad2824
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_7F0nPb2bvnQ3dSpant-_d

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-allowed-for-bot-93ad2824
```

Example prompt: Before I scrape https://example.com/blog/article-123, check if GPTBot is allowed to fetch that URL by looking up their robots.txt.

## When to prefer this

Use this endpoint when an AI agent or scraping workflow needs to perform compliant, polite crawling and must verify robots.txt permissions before fetching any page. Prefer this over manual parsing when you need a structured response that explicitly names the matching rule, group, and crawl-delay for a specific user-agent token.

## Known failure modes

- robots.txt not found or unreachable (site returns 4xx/5xx on /robots.txt)
- URL is malformed or unreachable
- bot name not recognized as a known token (may fall back to wildcard)
- robots.txt is too large or malformed to parse
- network timeout fetching the remote robots.txt

## How this service works

Is a specific bot allowed to fetch a URL? Fetches the site's robots.txt and evaluates the path for the named user-agent (GPTBot, ClaudeBot, Googlebot, Bingbot, CCBot, PerplexityBot or any token), returning allowed/disallowed, the matching rule and group, crawl-delay, and whether the bot has a dedicated group or falls back to *. Pre-flight for compliant scraping agents. $0.01 per check.

## Output

Returns an allowed/disallowed boolean, the exact matching rule from robots.txt, the group (user-agent block) that matched, the crawl-delay if specified, and a flag indicating whether the bot has a dedicated group or falls back to the wildcard (*) group.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "properties": {}
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-allowed-for-bot-93ad2824/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from intel.rallylive.ca](https://www.zero.xyz/host/intel.rallylive.ca/llms.txt)