# ReadFetch Site Mapper

> ReadFetch Site Mapper is a paid API for AI agents from readfetch.com, paid per call via x402, $0.005/call, status unknown (last checked 2026-10-31).

Maps a website's crawlable surface by parsing robots.txt, collecting URLs from sitemaps, finding RSS/Atom/JSON feeds, and counting homepage links — without full crawling.

## Facts

- Endpoint: POST https://readfetch.com/v1/map?utm_source=zero.xyz
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-31
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/readfetch-site-mapper-34501f40
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_ouOxjn1042RtCnNILya0D

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability readfetch-site-mapper-34501f40 -d '<json body>'
```

Example prompt: Map the crawlable surface of https://techcrunch.com — give me up to 500 URLs from its sitemaps, any RSS or Atom feeds you find, and note what robots.txt allows or disallows, using direct egress.

## When to prefer this

Use this endpoint when you need to understand a site's crawlable structure before committing to a full scrape or crawl. It is ideal for planning /v1/scrape or /v1/crawl calls, discovering feeds for aggregation, or auditing robots.txt compliance — much cheaper and faster than crawling all pages to find URLs organically.

## Known failure modes

- Site has no sitemap.xml or robots.txt — returns empty URL lists
- Target blocks datacenter IPs — retry with residential egress
- Timeout exceeded before all sitemaps fetched — partial results returned
- Sitemap or robots.txt document exceeds max_bytes — fetch aborted
- Invalid or unreachable URL — HTTP error response
- Sitemap index references too many child sitemaps to fetch within timeout

## How this service works

Discover a site's crawlable surface without crawling it: parse robots.txt allow/disallow rules, collect URLs from sitemap.xml and sitemap indexes, find RSS, Atom and JSON feeds, and count internal and external links on the homepage. Use the returned URLs to plan readfetch_scrape, readfetch_batch, or readfetch_crawl — not readfetch_map again on each URL. Prefer readfetch_probe on one URL before paying for bodies.

## Output

Returns a structured payload including: URLs discovered from sitemap.xml and sitemap indexes (up to the requested limit), robots.txt allow/disallow rules, discovered RSS/Atom/JSON feed URLs, and counts of internal and external links found on the homepage.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "title": "Url",
   "format": "uri",
   "maxLength": 2083,
   "minLength": 1,
   "description": "Public http(s) URL of the site to map."
  },
  "limit": {
   "type": "integer",
   "title": "Limit",
   "default": 200,
   "maximum": 1000,
   "minimum": 1,
   "description": "Maximum number of URLs to return (1-1000)."
  },
  "max_bytes": {
   "type": "integer",
   "title": "Max Bytes",
   "default": 5000000,
   "maximum": 20000000,
   "minimum": 1024,
   "description": "Abort if any single document exceeds this many bytes."
  },
  "timeout_s": {
   "type": "number",
   "title": "Timeout S",
   "default": 60,
   "maximum": 95,
   "minimum": 1,
   "description": "Total budget for the whole call in seconds; every sitemap fetch and the browser fallback share it. 95 s is the edge ceiling (Cloudflare closes at 100 s)."
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/readfetch-site-mapper-34501f40/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from readfetch.com](https://www.zero.xyz/host/readfetch.com/llms.txt)