# DexL Agents Site Crawl

> DexL Agents Site Crawl is a paid API for AI agents from agents.dexl.io, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-13).

Crawls a website starting from a given URL, follows internal links, and returns the content of every page reached within a configurable page budget.

## Facts

- Endpoint: POST https://agents.dexl.io/v1/tools/site-crawl
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/dexl-agents-site-crawl-8f889977
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_ULnt92GVEtriNmnuoBtSc

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability dexl-agents-site-crawl-8f889977 -d '<json body>'
```

Example prompt: Can you crawl the docs site at https://docs.example.com starting from the homepage and fetch up to 8 pages, following only links on the same host, and give me all the content?

## When to prefer this

Use this endpoint when you need to read multiple linked pages from a single host in one API call rather than making many individual fetch requests. It is ideal for ingesting documentation sites, product catalogues, changelogs, or small wikis where following internal links is essential. Prefer it over single-page fetch when you need breadth across a site within a controlled page budget.

## Known failure modes

- URL is unreachable or returns non-200 status — crawl may return partial results or an error
- max_pages budget exhausted before all linked pages are visited — only pages within budget are returned
- Target site blocks crawlers via robots.txt or rate limiting — may return fewer pages or empty body
- Non-absolute or malformed URL provided — request rejected with validation error
- Pages behind authentication or paywalls cannot be accessed — content will be missing or gated

## How this service works

Crawl a website: fetch a page, follow the links inside the same host and return every page it reached, within a page budget you set. For reading documentation, a product catalogue or a changelog in one paid call instead of many.

## Output

A JSON object containing the crawl output: the content of every page reached (up to the max_pages budget), along with metadata such as whether the result was cached and how many units were consumed.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Absolute http(s) URL to start from"
  },
  "max_pages": {
   "type": "integer",
   "default": 3,
   "maximum": 10,
   "minimum": 1
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "units": 1,
  "cached": false,
  "output": {}
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/dexl-agents-site-crawl-8f889977/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agents.dexl.io](https://www.zero.xyz/host/agents.dexl.io/llms.txt)