# SiteSignal Robots Policy Snapshot

> SiteSignal Robots Policy Snapshot is a paid API for AI agents from specialty-chemical-refer-comparisons.trycloudflare.com, paid per call via x402, $0.015/call, status unknown (last checked 2026-09-15).

Fetches and returns the robots.txt crawl policy for a given website URL, indicating what crawlers and user-agents are allowed or disallowed.

## Facts

- Endpoint: GET https://specialty-chemical-refer-comparisons.trycloudflare.com/x402/robots-policy?url=https%3A%2F%2Fexample.com&userAgent=*
- Price: $0.015/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sitesignal-robots-policy-snapshot-9acae6c1
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_mQOlHtYfM9L3Wp_QGpw9-

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sitesignal-robots-policy-snapshot-9acae6c1
```

Example prompt: Can you check the robots.txt policy for https://example.com and tell me whether the wildcard user-agent (*) is allowed to crawl it?

## When to prefer this

Use this endpoint when you need a quick, paid-per-call lookup of a website's robots crawl policy without building your own fetch-and-parse pipeline. Ideal for compliance checks before scraping, SEO tooling, or automated agent workflows that need to verify crawl permissions on demand. Prefer this over building custom robots.txt parsers when low-latency, on-demand lookups with structured output are needed.

## Known failure modes

- Target URL is unreachable or returns a non-200 status — policy snapshot unavailable
- Website has no robots.txt file — empty or default permissive policy returned
- Malformed URL input — validation error returned
- Rate limiting or access restrictions on the target domain — partial or no data returned
- Very large robots.txt files may be truncated

## How this service works

Paid SiteSignal robots policy resource with bounded public-data inputs and direct Base USDC settlement.

## Output

Returns a structured snapshot of the target site's robots.txt policy, including allow/disallow rules, crawl-delay directives, and applicable user-agent scopes, reflecting what crawlers are permitted or blocked from accessing.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri"
      },
      "userAgent": {
       "type": "string",
       "default": "*",
       "maxLength": 100
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sitesignal-robots-policy-snapshot-9acae6c1/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from specialty-chemical-refer-comparisons.trycloudflare.com](https://www.zero.xyz/host/specialty-chemical-refer-comparisons.trycloudflare.com/llms.txt)