# robots.txt & Sitemap Analyzer

> robots.txt & Sitemap Analyzer is a paid API for AI agents from agent-data.centorprat.workers.dev, paid per call via x402, $0.02/call, status unknown (last checked 2026-10-28).

Parses robots.txt and XML sitemaps for given domains, reporting crawlability rules, sitemap structure, and flagged issues

## Facts

- Endpoint: POST https://agent-data.centorprat.workers.dev/tools/robots-sitemap-analyzer?utm_source=zero.xyz
- Price: $0.02/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-28
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-sitemap-analyzer-34747aad
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_877VvdA5k2lX2Pk7wz_f6

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-sitemap-analyzer-34747aad -d '<json body>'
```

Example prompt: Can you analyze the robots.txt and sitemaps for example.com and openai.com — tell me if they block crawlers, what sitemaps they declare, and flag any crawlability issues?

## When to prefer this

Choose this endpoint when you need a structured, parsed breakdown of robots.txt directives and sitemap metadata for up to 3 domains in a single call, especially for SEO audits, pre-crawl checks, or competitive research. It is keyless and read-only, making it low-friction. Prefer it over manual fetching when you need normalized, issue-flagged output rather than raw file content.

## Known failure modes

- Domain does not exist or is unreachable — lookup fails, not charged
- robots.txt absent or returns non-200 status — noted in findings
- Sitemap URL declared but returns error or malformed XML — flagged as issue
- More than 3 domains submitted — request rejected at input validation
- Network timeout fetching robots.txt or sitemap — lookup marked as failed

## How this service works

Pay-per-call data and tools for AI agents, $0.002-0.01 per call in USDC via x402 (Base, Polygon or Solana), no account or API key; MAX tiers ($0.05-0.25) run deeper jobs. Agent toolkit: web search, search + read top pages, any page as clean markdown with a prompt-injection check, which-x402-API-to-call recommender, seller check. Data: Google Maps places with contacts, YouTube video data and transcripts, job listings (Indeed, LinkedIn, 13 boards), Google Ads Transparency, TikTok comments, Pinterest, Airbnb, Redfin, AliExpress, app stores, live sports, website and SEO checks, Shopify store tech stacks, and onchain token / address risk checks.

## Output

For each domain: parsed robots.txt findings (user-agent rules, allow/disallow paths, crawl-delay, declared sitemaps, blocks-all flag) and sitemap analysis (URL counts, nested sitemaps, lastmod coverage), plus a crawlability signal and a list of flagged issues. Failed domain lookups are not charged.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "domains": {
   "type": "array",
   "items": {
    "type": "string"
   },
   "maxItems": 3,
   "description": "Domains to analyze, e.g. [\"example.com\"]. One report per domain."
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "items": [
   {
    "note": "Read-only fetch of robots.txt and the XML sitemap(s). Recommendations, not a verdict.",
    "domain": "apple.com",
    "signal": "ok-with-recommendations",
    "findings": [
     "Sitemap URLs have no <lastmod> - crawlers can't tell what changed."
    ],
    "sitemaps": [
     {
      "url": "https://www.apple.com/shop/sitemap.xml",
      "type": "sitemapindex",
      "found": true,
      "valid": true,
      "sample": [
       "https://www.apple.com.cn/shop/sitemap-index.xml",
       "https://www.apple.com/ae/shop/sitemap-index.xml",
       "https://www.apple.com/at/shop/sitemap-index.xml",
       "https://www.apple.com/au/shop/sitemap-index.xml",
       "https://www.apple.com/be-fr/shop/sitemap-index.xml"
      ],
      "childSitemaps": 47
     },
     {
      "url": "https://www.apple.com/autopush/sitemap/sitemap-index.xml",
      "type": "sitemapindex",
      "found": true,
      "valid": true,
      "sample": [
       "https://www.apple.com/autopush/sitemap/ae-ar/sitemap-image.xml",
       "https://www.apple.com/autopush/sitemap/ae-ar/sitemap-video.xml",
       "https://www.apple.com/autopush/sitemap/ae-ar/sitemap.xml",
       "https://www.apple.com/autopush/sitemap/ae/sitemap-image.xml",
       "https://www.apple.com/autopush/sitemap/ae/sitemap-video.xml"
      ],
      "childSitemaps": 293
     },
     {
      "url": "https://www.apple.com/newsroom/sitemap.xml",
      "type": "urlset",
      "found": true,
      "valid": true,
      "urlCount": 2601,
      "withLastmod": 0,
      "newestLastmod": null
     }
    ],
    "checkedAt": "2026-10-24T13:49:33+00:00",
    "robotsTxt": {
     "found": true,
     "groups": 6,
     "crawlDelay": null,
     "userAgents": [
      "*",
      "Baiduspider",
      "HaoSouSpider",
      "Sogou inst spider",
      "Sogou spider2",
      "Sogou web spider"
     ],
     "sitemapsDeclared": [
      "https://www.apple.com/shop/sitemap.xml",
      "https://www.apple.com/autopush/sitemap/sitemap-index.xml",
      "https://www.apple.com/newsroom/sitemap.xml",
      "https://www.apple.com/retail/sitemap/sitemap.xml",
      "https://www.apple.com/today/sitemap-index.xml"
     ],
     "blocksAllCrawlers": false,
     "totalDisallowRules": 9
    }
   }
  ]
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-sitemap-analyzer-34747aad/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agent-data.centorprat.workers.dev](https://www.zero.xyz/host/agent-data.centorprat.workers.dev/llms.txt)