# AgentData Web Crawl

> AgentData Web Crawl is a paid API for AI agents from agentdata-api.sander-van-aard.workers.dev, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-15).

Breadth-first crawl of a website starting from a given URL, returning up to 20 pages as clean markdown, paid per page via USDC on Base with no account required.

## Facts

- Endpoint: POST https://agentdata-api.sander-van-aard.workers.dev/web/crawl
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agentdata-web-crawl-aa0247f3
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_FAu6-Ik8d8o4aiSE5rE0C

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agentdata-web-crawl-aa0247f3 -d '<json body>'
```

Example prompt: Crawl the blog at https://example.com/blog and grab up to 10 pages, returning each one as clean markdown so I can index them.

## When to prefer this

Choose this endpoint when you need to extract content from multiple pages of a single website in one paid call, rather than making individual page requests. It is ideal for documentation indexing, RAG pipeline ingestion, site archiving, and competitive content gathering where you know a starting URL and want breadth-first link traversal up to 20 pages. It requires no account or API key and settles payment in USDC on Base via x402, making it suitable for autonomous agent workflows without pre-registration.

## Known failure modes

- Start URL is unreachable or returns unreadable content — results in 400 error, no charge
- Individual pages within the crawl fail to load — silently skipped and listed in the skipped[] array, call still settles
- max_pages outside the accepted range of 2-20 — likely validation error
- Domain does not resolve or TLS error on start URL — 400 error
- Non-same-host links are not followed — content from external domains will not appear in results

## How this service works

Crawl one site in one paid call: start URL + max_pages (2-20); we follow same-host links breadth-first, returning each page as clean markdown (same extraction as /web/contents). Priced at $0.001 per requested page, quoted as max_pages up front — finding fewer pages is still a complete delivery. Failed pages are skipped and listed in skipped[]; the call still settles. Only an unreadable start page is a 400, no charge. USDC on Base, no account, no API key.

## Output

A structured response containing each successfully crawled page as clean markdown text (same extraction quality as the /web/contents endpoint), plus a list of any pages that were skipped due to fetch or parse errors. The caller pays for up to max_pages; fewer found pages is still a complete successful delivery. Only an unreadable start URL triggers a 400 error with no charge.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method",
    "bodyType",
    "body"
   ],
   "properties": {
    "body": {
     "type": "object"
    },
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "POST",
      "PUT",
      "PATCH"
     ],
     "type": "string"
    },
    "headers": {
     "type": "object",
     "additionalProperties": {
      "type": "string"
     }
    },
    "bodyType": {
     "enum": [
      "json",
      "form-data",
      "text"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "additionalProperties": {
      "type": "string"
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agentdata-web-crawl-aa0247f3/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agentdata-api.sander-van-aard.workers.dev](https://www.zero.xyz/host/agentdata-api.sander-van-aard.workers.dev/llms.txt)