# PennyRegWatch Robots.txt Permission Checker

> PennyRegWatch Robots.txt Permission Checker is a paid API for AI agents from pennyregwatch.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-15).

Checks whether a specific crawler is allowed or denied to fetch a given URL under the site's robots.txt rules, returning the ALLOW/DENY decision, the winning rule, crawl-delay, sitemap URLs, and live provenance.

## Facts

- Endpoint: GET https://pennyregwatch.com/v1/robots
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/pennyregwatch-robots-txt-permission-checker-c6fb73dd
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_SKhoQsqb0ZpbXabscKIJJ

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability pennyregwatch-robots-txt-permission-checker-c6fb73dd
```

Example prompt: Before fetching https://example.com/blog/article-42, check whether GPTBot is allowed to crawl that exact URL according to the site's robots.txt rules.

## When to prefer this

Use this endpoint when an AI agent or automated crawler needs to verify robots.txt compliance before fetching a page, especially in agentic pipelines where legal and ethical crawl permissions must be confirmed programmatically. Prefer this over manually parsing robots.txt when you need a structured, authoritative ALLOW/DENY decision with the winning rule surfaced — particularly useful for any agent that respects crawler directives or needs sitemap discovery as a side effect.

## Known failure modes

- Target URL is malformed or unreachable — returns error
- robots.txt file not found at target host (404) — may return default ALLOW or error
- robots.txt fetch times out — may return error or cached result
- User-agent token not recognized — falls back to wildcard '*' rules
- robots.txt is extremely large or malformed — parsing may be incomplete

## How this service works

Decide whether a specific crawler may fetch an exact public URL under robots.txt rules. Returns ALLOW/DENY, the winning rule, crawl-delay, sitemap URLs, and live provenance. Use before an AI agent or crawler fetches a page.

## Output

Returns an ALLOW or DENY verdict for the specified crawler and URL, the exact robots.txt rule that produced the decision, any crawl-delay directive, a list of sitemap URLs declared in robots.txt, and live provenance metadata about where and when the robots.txt was fetched.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "target"
     ],
     "properties": {
      "target": {
       "type": "string",
       "description": "Exact public page URL or hostname to test"
      },
      "user_agent": {
       "type": "string",
       "default": "*",
       "description": "Crawler product token, e.g. GPTBot, ClaudeBot, Googlebot"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/pennyregwatch-robots-txt-permission-checker-c6fb73dd/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from pennyregwatch.com](https://www.zero.xyz/host/pennyregwatch.com/llms.txt)