# Agent402 Site Crawl

> Agent402 Site Crawl is a paid API for AI agents from agent402.tools, paid per call via x402, $0.02/call, status unknown (last checked 2026-09-14).

Crawls a website breadth-first up to a configurable depth and page limit, returning page content, links, titles, and crawl metadata as structured JSON

## Facts

- Endpoint: POST https://agent402.tools/api/site-crawl
- Price: $0.02/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agent402-site-crawl-9eb353f5
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Nc9M5CqY9u3MZS2IiZAoZ

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agent402-site-crawl-9eb353f5 -d '<json body>'
```

Example prompt: Crawl https://docs.example.com starting from the homepage, go 2 levels deep, fetch up to 15 pages, and return everything in markdown format — stay on the same host and skip any URLs containing /changelog

## When to prefer this

Choose this endpoint when you need to programmatically crawl multiple pages of a website breadth-first without setting up your own scraping infrastructure. It is especially useful when you want robots.txt compliance baked in, need content in clean markdown or plain text, and want structured metadata (titles, links, status codes) alongside the content. Prefer it over single-page fetch tools when you need 2–20 pages from a site in one call.

## Known failure modes

- Invalid or unreachable start URL returns an error or zero crawled pages
- Limit reached before all pages are fetched — skipped.limit counter increments
- robots.txt disallows Agent402Bot on target pages — those pages are skipped and counted under skipped.robots
- Requested depth of 0-2 only — values outside range may be rejected or clamped
- Binary resources (PDFs, images) are skipped and counted under skipped.binary
- Off-site links skipped when sameHost is true (default) — counted under skipped.offsite
- Pattern filters excluding all found links result in minimal crawl output

## How this service works

Crawl a website from a starting URL and return each page as clean markdown: breadth-first over internal links, bounded by page count and depth, honouring robots.txt, with per-page title, status, depth and outbound links. Use it when an agent needs a whole section of a site rather than one known page.

## Output

A JSON object containing an array of crawled pages (each with URL, depth, HTTP status, title, content in markdown or text, character count, and outbound links), plus summary stats: total crawled, fetched, queued, skipped counts broken down by reason (depth, error, limit, binary, robots, unsafe, offsite, pattern), elapsed time in ms, fetch timestamp, robots.txt status, format used, and a truncated flag.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Start URL"
  },
  "limit": {
   "type": "integer",
   "description": "Max pages to fetch, 1-20 (default 10); failed fetches count toward it"
  },
  "format": {
   "enum": [
    "markdown",
    "text"
   ],
   "type": "string",
   "description": "Page content format (default markdown)"
  },
  "maxDepth": {
   "type": "integer",
   "description": "Link depth from the start URL, 0-2 (default 1)"
  },
  "sameHost": {
   "type": "boolean",
   "description": "true (default): stay on the start host (www and bare host count as one); false: also follow subdomains of the start site"
  },
  "excludePatterns": {
   "type": "array",
   "items": {
    "type": "string"
   },
   "description": "Never follow links whose URL contains any of these substrings (max 20)"
  },
  "includePatterns": {
   "type": "array",
   "items": {
    "type": "string"
   },
   "description": "Only follow links whose URL contains at least one of these substrings (max 20)"
  },
  "maxCharsPerPage": {
   "type": "integer",
   "description": "Cap on content characters per page, 200-20000 (default 8000)"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com/",
  "pages": [
   {
    "url": "https://example.com/",
    "depth": 0,
    "links": [],
    "title": "Example Domain",
    "status": 200,
    "content": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
    "contentChars": 166
   }
  ],
  "format": "markdown",
  "queued": 0,
  "source": "live fetch over internal links (breadth-first), robots.txt honoured for Agent402Bot",
  "crawled": 1,
  "fetches": 2,
  "skipped": {
   "depth": 0,
   "error": 0,
   "limit": 0,
   "binary": 0,
   "robots": 0,
   "unsafe": 0,
   "offsite": 1,
   "pattern": 0
  },
  "elapsedMs": 420,
  "fetchedAt": "2026-08-22T00:00:00.000Z",
  "robotsTxt": "not readable",
  "truncated": false,
  "untrustedContent": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agent402-site-crawl-9eb353f5/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agent402.tools](https://www.zero.xyz/host/agent402.tools/llms.txt)