# Search Readiness Lab – Crawler Policy Audit

> Search Readiness Lab – Crawler Policy Audit is a paid API for AI agents from search-readiness.inboxtzdjqv.workers.dev, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-10).

Audits a URL's robots.txt, sitemaps, and crawler access policies for Google, traditional search engines, and AI-agent crawlers.

## Facts

- Endpoint: POST https://search-readiness.inboxtzdjqv.workers.dev/v1/crawler-policy?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-10
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/search-readiness-lab-crawler-policy-audit-c88f0c9d
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_gydEllKQyJueifG7Q8bO-

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability search-readiness-lab-crawler-policy-audit-c88f0c9d -d '<json body>'
```

Example prompt: Can you audit https://www.myshop.com for crawler access? I want to know whether Googlebot, GPTBot, and ClaudeBot are allowed in, and whether there are any sitemaps discoverable.

## When to prefer this

Choose this endpoint when you need a deterministic, structured audit of a specific URL's crawler access policies — especially when you need to verify AI-agent crawler discoverability alongside traditional search engine access. It is ideal for pre-launch SEO checks, competitive analysis, or diagnosing why a site might not be indexed by AI search platforms.

## Known failure modes

- Invalid or unreachable URL returns an error or empty agent array
- robots.txt not found (non-200 status) may result in no policy data
- More than 20 user agents specified will be rejected due to maxItems constraint
- Network timeouts fetching the target URL's robots.txt
- Malformed URL format fails schema validation

## How this service works

Paid deterministic audits for Google, conventional search, and AI-agent crawler discoverability.

## Output

Returns a JSON object with the HTTP status of the robots.txt file, a list of discovered sitemaps, the origin URL, and per-agent access rules indicating which of the requested user agents are allowed or disallowed from crawling the site.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "format": "uri"
  },
  "userAgents": {
   "type": "array",
   "items": {
    "type": "string"
   },
   "maxItems": 20
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "agents": [],
  "origin": "https://example.com",
  "sitemaps": [],
  "robotsStatus": 200
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/search-readiness-lab-crawler-policy-audit-c88f0c9d/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from search-readiness.inboxtzdjqv.workers.dev](https://www.zero.xyz/host/search-readiness.inboxtzdjqv.workers.dev/llms.txt)