# SiteSignal Robots Policy Snapshot

> SiteSignal Robots Policy Snapshot is a paid API for AI agents from phases-prot-shine-royal.trycloudflare.com, paid per call via x402, $0.015/call, status down (last checked 2026-09-15).

Fetches and parses a public robots.txt file into structured crawler groups, allow/disallow rules, crawl delay, sitemap references, and a response hash.

## Facts

- Endpoint: GET https://phases-prot-shine-royal.trycloudflare.com/x402/robots-policy
- Price: $0.015/call
- Payment: x402
- Status: down
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sitesignal-robots-policy-snapshot-f5ff93bd
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_udZXUGE4-TENctFash1Yg

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sitesignal-robots-policy-snapshot-f5ff93bd
```

Example prompt: Can you fetch and parse the robots.txt for https://example.com and show me the crawler groups, disallowed paths, crawl delay, and any sitemap references listed?

## When to prefer this

Use this endpoint when you need structured, machine-readable extraction of robots.txt crawler policies rather than just fetching the raw file. It is ideal for SEO audits, crawler policy monitoring, sitemap discovery, and change detection via response hashing. Prefer it over general HTTP fetchers when you specifically need per-agent allow/disallow rule parsing and crawl delay extraction.

## Known failure modes

- URL is not publicly accessible or returns a non-200 HTTP status
- robots.txt does not exist at the given domain (404 response)
- Malformed robots.txt that cannot be parsed into structured groups
- Network timeout or DNS resolution failure for the provided URL
- URL parameter missing or not a valid URI format

## How this service works

Parse a public robots.txt into crawler groups, effective allow/disallow rules, crawl delay, sitemap references, and a response hash.

## Output

A structured breakdown of the parsed robots.txt including named crawler groups with their effective allow and disallow path rules, crawl delay values, sitemap URLs referenced in the file, and a hash of the raw response for change detection.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri"
      },
      "userAgent": {
       "type": "string",
       "default": "*",
       "maxLength": 100
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sitesignal-robots-policy-snapshot-f5ff93bd/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from phases-prot-shine-royal.trycloudflare.com](https://www.zero.xyz/host/phases-prot-shine-royal.trycloudflare.com/llms.txt)