# ReadFetch Bounded Async Web Crawl

> ReadFetch Bounded Async Web Crawl is a paid API for AI agents from readfetch.com, paid per call via x402, $0.003/call, status unknown (last checked 2026-10-31).

Initiates a bounded asynchronous crawl of a website up to a declared page cap, returning a job ID and per-page Markdown content for each crawled page.

## Facts

- Endpoint: POST https://readfetch.com/v1/crawl?utm_source=zero.xyz
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-31
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/readfetch-bounded-async-web-crawl-100d7aff
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_5gn92D0QhMrqtNRIIr765

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability readfetch-bounded-async-web-crawl-100d7aff -d '<json body>'
```

Example prompt: Crawl the documentation site at https://docs.example.com asynchronously, cap it at 20 pages, and give me each page's content as Markdown so I can index them.

## When to prefer this

Choose this endpoint when you need to crawl multiple pages of a website asynchronously with a known, bounded page budget — ideal for documentation indexing, content audits, or knowledge base extraction where you want predictable cost and a job ID to track progress. Prefer this over the single-page scrape endpoint when you need many pages at once, and over the batch URL endpoint when you want the crawler to discover links automatically rather than supplying explicit URLs.

## Known failure modes

- Target site blocks crawlers (robots.txt disallowed) returning an error or empty results
- Page cap exceeded — crawl stops at declared limit even if more pages exist
- Job ID not found if polling too early before crawl initializes
- Network timeouts on slow or unresponsive target sites
- Dynamic JavaScript-heavy pages may render as incomplete Markdown without a JS-rendering option
- Invalid starting URL format returns a validation error

## How this service works

Start a bounded async crawl from one seed URL. Returns job_id, status (queued), pages_quoted, and price_usd — then poll readfetch_crawl_status with that id until status is completed or failed. Completed jobs include per-page url and Markdown in the status payload. Cap scope with max_pages (default 20, max 100) and max_depth (default 2, max 3); include/exclude filter paths by prefix (* suffix on include). Prefer readfetch_batch for a fixed URL list; readfetch_map to list URLs 

## Output

Returns a job ID immediately upon submission, then provides per-page Markdown for each crawled URL up to the declared page cap. Results are bounded and predictable — the page limit is quoted upfront so cost is known before crawling begins.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "format": "uri",
   "description": "Seed URL; crawl stays on the same registrable domain."
  },
  "exclude": {
   "type": "array",
   "items": {
    "type": "string"
   },
   "description": "Path prefixes to skip during the crawl."
  },
  "include": {
   "type": "array",
   "items": {
    "type": "string"
   },
   "description": "Path prefixes to allow; * suffix wildcard."
  },
  "max_depth": {
   "type": "number",
   "default": 2,
   "maximum": 3,
   "minimum": 1,
   "description": "Link depth from the seed URL (default 2)."
  },
  "max_pages": {
   "type": "number",
   "default": 20,
   "maximum": 100,
   "minimum": 1,
   "description": "Maximum pages to fetch in this job (quoted up front for payment)."
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/readfetch-bounded-async-web-crawl-100d7aff/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from readfetch.com](https://www.zero.xyz/host/readfetch.com/llms.txt)