# vaaya.ai CRW — Async Multi-Page Web Crawl

> vaaya.ai CRW — Async Multi-Page Web Crawl is a paid API for AI agents from vaaya.ai, paid per call via x402, $0.1/call, status unknown (last checked 2026-10-30).

Starts an asynchronous multi-page crawl from a seed URL, following links across up to 100 pages, and returns a job ID to poll for results

## Facts

- Endpoint: POST https://vaaya.ai/api/run/crw/crawl?utm_source=zero.xyz
- Price: $0.1/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-30
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/vaaya-ai-crw-async-multi-page-web-crawl-80a43ef2
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_4CJZWZpYaA_dauhHzgIjU

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability vaaya-ai-crw-async-multi-page-web-crawl-80a43ef2 -d '<json body>'
```

Example prompt: Crawl the entire docs.example.com site starting from the homepage, following links up to 3 levels deep and scraping up to 50 pages — return everything as clean markdown.

## When to prefer this

Choose this endpoint when you need to extract content from multiple pages of a website in a single job, following internal links automatically. Prefer it over the single-URL scrape endpoint when the target content spans many pages. Use the URL discovery endpoint first if you only need URLs without content. Best suited for documentation sites, blogs, product catalogs, or any multi-page content extraction task where you want to set a page budget (maxPages) and depth limit.

## Known failure modes

- Invalid or unreachable seed URL — crawl fails immediately with an error status
- maxPages or maxDepth out of range (maxPages must be 1-100) — validation error returned
- Site blocks crawlers/robots.txt disallows — crawl returns zero or few pages
- Polling not performed — agent never retrieves results if it doesn't call crawl_status
- Timeout on large sites — crawl may partially complete if site is very large or slow
- renderJs failures on JavaScript-heavy sites with complex SPAs

## How this service works

CRW — Start an ASYNC multi-page crawl from a seed URL, following links. Pass `url`; optional `maxPages` (1-100, default 100 — vendor bills 1 credit/page so keep it tight), `maxDepth` (default 2), `formats`, `onlyMainContent`, `renderJs`. Returns `{ id }` immediately; KEEP POLLING `crw/crawl_status` with that id until status is completed/failed. One price covers the whole job.

## Output

Returns immediately with a JSON object containing a job `id`. The agent must poll the `crw/crawl_status` endpoint with that ID until status is `completed` or `failed`. On completion, the full crawl results include scraped page content in the requested formats (markdown, HTML, JSON) for all pages visited, along with metadata per page.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method",
    "bodyType",
    "body"
   ],
   "properties": {
    "body": {
     "type": "object",
     "$schema": "http://json-schema.org/draft-07/schema#",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri"
      },
      "formats": {
       "type": "array",
       "items": {
        "type": "string"
       }
      },
      "maxDepth": {
       "type": "integer",
       "minimum": 0
      },
      "maxPages": {
       "type": "integer",
       "maximum": 100,
       "minimum": 1
      },
      "renderJs": {
       "type": [
        "boolean",
        "null"
       ]
      },
      "onlyMainContent": {
       "type": "boolean"
      }
     },
     "additionalProperties": true
    },
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "POST"
     ],
     "type": "string"
    },
    "bodyType": {
     "enum": [
      "json",
      "form-data",
      "text"
     ],
     "type": "string"
    },
    "pathParams": {
     "type": "object"
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/vaaya-ai-crw-async-multi-page-web-crawl-80a43ef2/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from vaaya.ai](https://www.zero.xyz/host/vaaya.ai/llms.txt)