# SiteSignal Robots Policy Snapshot

> SiteSignal Robots Policy Snapshot is a paid API for AI agents from trinity-throw-thursday-gravity.trycloudflare.com, paid per call via x402, $0.015/call, status unknown (last checked 2026-09-16).

Fetches and parses a public robots.txt file into structured crawler groups, allow/disallow rules, crawl delays, sitemap references, and a response hash.

## Facts

- Endpoint: GET https://trinity-throw-thursday-gravity.trycloudflare.com/x402/robots-policy
- Price: $0.015/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-16
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sitesignal-robots-policy-snapshot-904888f3
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_BKJr7M8As0vKDMynIzRRJ

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sitesignal-robots-policy-snapshot-904888f3
```

Example prompt: Can you fetch and parse the robots.txt for https://example.com and tell me which crawlers are blocked, what paths are disallowed, any crawl delays set, and what sitemaps are listed?

## When to prefer this

Use this endpoint when you need structured, machine-readable extraction of a public robots.txt file rather than raw text. It is ideal for SEO auditing, crawler compliance checks, competitive research on site access policies, and change detection via response hashing. Prefer this over manual fetching when you need normalized rule sets per user-agent group and sitemap enumeration without writing your own parser.

## Known failure modes

- URL not provided or malformed — returns validation error
- Target site returns non-200 status for robots.txt (e.g. 404, 403) — returns error indicating file unavailable
- robots.txt exists but is empty — returns empty groups and rules
- Target site is unreachable or times out — returns network/timeout error
- Malformed robots.txt that cannot be parsed — returns partial or error response

## How this service works

Parse a public robots.txt into crawler groups, effective allow/disallow rules, crawl delay, sitemap references, and a response hash.

## Output

Returns a structured breakdown of the robots.txt including: all crawler/user-agent groups with their effective allow and disallow rules, any crawl delay directives, a list of sitemap URLs referenced, and a hash of the raw response for change detection purposes.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri"
      },
      "userAgent": {
       "type": "string",
       "default": "*",
       "maxLength": 100
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sitesignal-robots-policy-snapshot-904888f3/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from trinity-throw-thursday-gravity.trycloudflare.com](https://www.zero.xyz/host/trinity-throw-thursday-gravity.trycloudflare.com/llms.txt)