# AI Crawler Access Test > AI Crawler Access Test is a paid API for AI agents from agent-data.centorprat.workers.dev, paid per call via x402, $0.02/call, status unknown (last checked 2026-10-28). Tests whether AI bots (OpenAI, Anthropic, Perplexity, Google, Bing) can actually crawl a website by checking robots.txt rules, live fetch simulation per bot, CDN/WAF blocking, JS-only content, llms.txt, and entity schema. ## Facts - Endpoint: POST https://agent-data.centorprat.workers.dev/tools/ai-crawler-access-test?utm_source=zero.xyz - Price: $0.02/call - Payment: x402 - Status: unknown - Last checked: 2026-10-28 - Activations on Zero: 0 - Tags: x402 - Canonical page: https://www.zero.xyz/c/ai-crawler-access-test-92645808 - Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_hIOtzbn0nl0MGmJcLuq99 Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently. ## How to call it through Zero Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`): ```sh zero fetch --capability ai-crawler-access-test-92645808 -d '' ``` Example prompt: Check whether AI bots like GPTBot, ClaudeBot, and PerplexityBot can actually access https://example.com and https://example.com/pricing — tell me if robots.txt is blocking any of them, if the CDN is rejecting bot fetches, and whether llms.txt is present. ## When to prefer this Choose this endpoint when you need to diagnose why AI crawlers cannot read a specific website or page, rather than general SEO audits. It uniquely simulates live fetches as individual AI bots (GPTBot, ClaudeBot, PerplexityBot, Bingbot, Googlebot-AI) and compares them to browser fetches, making it the right tool for pinpointing CDN/WAF blocks, robots.txt misconfigurations, JS-only rendering issues, and missing llms.txt — all in a single call per URL. ## Known failure modes - URL is unreachable or returns non-200 — endpoint reports the fetch failure without charging - robots.txt is missing or malformed — reported as absent/unparseable - More than 3 URLs submitted — rejected due to maxItems limit - Malformed URL format — may cause lookup failure - CDN timeout during live fetch — reported as a failed fetch result ## How this service works Pay-per-call data and tools for AI agents, $0.002-0.01 per call in USDC via x402 (Base, Polygon or Solana), no account or API key; MAX tiers ($0.05-0.25) run deeper jobs. Agent toolkit: web search, search + read top pages, any page as clean markdown with a prompt-injection check, which-x402-API-to-call recommender, seller check. Data: Google Maps places with contacts, YouTube video data and transcripts, job listings (Indeed, LinkedIn, 13 boards), Google Ads Transparency, TikTok comments, Pinterest, Airbnb, Redfin, AliExpress, app stores, live sports, website and SEO checks, Shopify store tech stacks, and onchain token / address risk checks. ## Output Per-URL report covering: which AI bots are allowed or blocked by robots.txt (separately for search vs training), live fetch result as each bot vs a real browser (to catch CDN/WAF blocks), detection of JS-only content that bots can't render, presence and validity of llms.txt and entity schema, plus actionable fix recommendations. Failed lookups are not billed. ## Request schema (JSON Schema) ```json { "type": "object", "properties": { "urls": { "type": "array", "items": { "type": "string" }, "maxItems": 3, "description": "e.g. \"example.com\" or \"https://example.com/pricing\". One report per URL (the path is checked against robots.txt rules)." }, "maxSites": { "type": "integer", "description": "Cap how many URLs to test this run." } } } ``` ## Response schema (JSON Schema) ```json { "type": "json", "example": { "items": [ { "url": "https://react.dev", "bots": [ { "bot": "OAI-SearchBot", "major": true, "owner": "OpenAI", "purpose": "search", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, "robotsAllowed": true }, { "bot": "ChatGPT-User", "major": true, "owner": "OpenAI", "purpose": "user-fetch", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, "robotsAllowed": true }, { "bot": "GPTBot", "major": false, "owner": "OpenAI", "purpose": "training", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, "robotsAllowed": true }, { "bot": "Claude-SearchBot", "major": true, "owner": "Anthropic", "purpose": "search", "robotsRule": "no rule matches", "robotsAllowed": true }, { "bot": "Claude-User", "major": true, "owner": "Anthropic", "purpose": "user-fetch", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, "robotsAllowed": true }, { "bot": "ClaudeBot", "major": false, "owner": "Anthropic", "purpose": "training", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, "robotsAllowed": true }, { "bot": "PerplexityBot", "major": true, "owner": "Perplexity", "purpose": "search", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, "robotsAllowed": true }, { "bot": "Perplexity-User", "major": true, "owner": "Perplexity", "purpose": "user-fetch", "robotsRule": "no rule matches", "robotsAllowed": true }, { "bot": "Googlebot", "major": true, "owner": "Google", "purpose": "search", "liveAccess": "ok", "liveStatus": 200, "robotsRule": "no rule matches", "blockReason": null, "liveTextChars": 8342, … (truncated) ``` ## More - Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawler-access-test-92645808/health.json - [Zero catalog index](https://www.zero.xyz/llms.txt) - [Other services from agent-data.centorprat.workers.dev](https://www.zero.xyz/host/agent-data.centorprat.workers.dev/llms.txt)