# SignalHarness Robots Policy Parse

> SignalHarness Robots Policy Parse is a paid API for AI agents from signalharness.ai, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-13).

Fetches and parses a website's robots.txt file to determine whether a given user agent is allowed to crawl a specific URL path

## Facts

- Endpoint: POST https://signalharness.ai/api/agent/services/robots_policy_parse/invoke
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/signalharness-robots-policy-parse-adf843ab
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Cf1r6qGF64N6rj0qGplYV

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability signalharness-robots-policy-parse-adf843ab -d '<json body>'
```

Example prompt: Before I scrape https://example.com/blog/article-123, check the robots.txt to see if a user agent called 'MyBot/1.0' is actually allowed to access that path.

## When to prefer this

Use this endpoint when an AI agent or web crawler needs to programmatically verify robots.txt compliance before accessing any web resource, especially in automated pipelines where legal and ethical scraping compliance must be enforced. Prefer this over manual robots.txt parsing when you need structured, machine-readable output including matched rules, crawl delay, and sitemaps in a single call. Particularly useful for agents that need to self-certify access permissions before taking action on web content.

## Known failure modes

- URL is unreachable or returns a non-200 status — crawl permission may default to allowed or error
- robots.txt is malformed or empty — warnings array will be populated, results may be incomplete
- Invalid URL format not matching ^https:// pattern — request rejected with validation error
- Target site blocks the fetch request — result may reflect inability to retrieve policy
- User agent string too long (>256 chars) or path too long (>2048 chars) — request rejected

## How this service works

Explore 330 pay-per-call x402 API services and 27 agent-native digital products, with Base USDC pricing, secure Polar checkout, and free discovery.

## Output

Returns a JSON object indicating whether crawling is allowed for the specified user agent and path, the matched robots.txt rule and group, any applicable crawl delay, a list of sitemaps declared, any warnings encountered during parsing, and full execution receipt metadata including payment confirmation and result hash.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "pattern": "^https://",
   "maxLength": 2048,
   "minLength": 9
  },
  "text": {
   "type": "string",
   "maxLength": 65536,
   "minLength": 0
  },
  "userAgent": {
   "type": "string",
   "maxLength": 256,
   "minLength": 1
  },
  "targetPath": {
   "type": "string",
   "maxLength": 2048,
   "minLength": 1
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "replay": false,
  "result": {
   "groups": [
    {
     "rules": [
      {
       "path": "example",
       "directive": "allow"
      }
     ],
     "crawlDelay": 0,
     "userAgents": [
      "example"
     ]
    }
   ],
   "sitemaps": [
    "example"
   ],
   "warnings": [
    "Verify the caller-supplied data before relying on this result."
   ],
   "crawlDelay": 0,
   "matchedRule": {},
   "crawlAllowed": false,
   "matchedGroup": [
    null
   ]
  },
  "status": "succeeded",
  "receipt": {
   "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
   "usage": [],
   "status": "succeeded",
   "network": "eip155:8453",
   "artifacts": [],
   "endedAtMs": 0,
   "latencyMs": 0,
   "paymentId": "example-payment",
   "receiptId": "example-receipt",
   "requestId": "example-request",
   "serviceId": "robots_policy_parse",
   "executionId": "example-execution",
   "startedAtMs": 0,
   "amountAtomic": "5000",
   "resultSha256": "9f187257ea9c44cac9c39fc35e42f5352601cd2a27c6b8ee4122b8e15d4a2306",
   "serviceVersion": "1.0.0",
   "settlementReference": "0x0000000000000000000000000000000000000000000000000000000000000000"
  },
  "artifacts": [],
  "requestId": "example-request",
  "executionId": "example-execution"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/signalharness-robots-policy-parse-adf843ab/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from signalharness.ai](https://www.zero.xyz/host/signalharness.ai/llms.txt)