# WebLens Crawl API

> WebLens Crawl API is a paid API for AI agents from api.weblens.dev, paid per call via x402, $0.015/call, status unknown (last checked 2026-10-02).

Crawls an entire website up to a configurable depth and page limit, returning structured markdown content for each discovered page

## Facts

- Endpoint: POST https://api.weblens.dev/crawl?utm_source=zero.xyz
- Price: $0.015/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-02
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/weblens-crawl-api-e5c8cde4
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Sr4GJn4_lbOfT7ex55r10

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability weblens-crawl-api-e5c8cde4 -d '<json body>'
```

Example prompt: Crawl the entire docs.example.com website up to 3 levels deep, fetching up to 50 pages, and give me the content of every page in markdown so I can build a knowledge base from it.

## When to prefer this

Choose this endpoint when you need to retrieve content from multiple pages across an entire site rather than a single URL. It is ideal for building knowledge bases, indexing documentation, auditing site content, or feeding website text into downstream AI pipelines. Prefer it over single-page fetch endpoints whenever you need breadth across a domain. The pay-per-call model (no monthly commitment) makes it cost-effective for sporadic or one-off full-site crawls compared to subscription-based alternatives.

## Known failure modes

- Target site blocks crawlers or returns 403/429 — affected pages will have status 'failed' in the response
- Robots.txt exclusions prevent crawling certain paths when robotsRespected is true
- Very large sites may hit the page limit before full discovery
- Network timeouts on slow or unreliable target servers
- Invalid or unreachable starting URL returns an error response
- JavaScript-rendered pages may not return full content if the site requires a browser runtime

## How this service works

# WebLens - Web Intelligence API, pay per call

Scrape, crawl, map and extract the web. No account, no API key, no monthly
minimum — you pay for the calls you make and nothing else.

## Pricing
Page fetching starts at **$0.002**, whole-site crawling at
**$0.0015/page**, and sitemap discovery at **$0.004**.
Comparable services bill $0.007-0.008 per request, or reach a lower per-page
rate only on a $99/month commitment. WebLens has no commitment to reach.

## Payment Protocol
All paid endpoints use the [x402 protocol](https://x402.org) for HTTP-native
micropayments (USDC on Base).

## Cache Discount
Cached responses are **70% cheaper** than fresh fetches.

## Output

A JSON object containing an array of crawled pages, each with its URL, depth, title, HTTP status, and full markdown content (with a truncation flag). Also includes a summary with counts of crawled, successful, failed, and discovered pages, max depth reached, robots.txt compliance status, the timestamp, and a request ID.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Start URL"
  },
  "limit": {
   "type": "number",
   "maximum": 25,
   "description": "Page budget, 1-25 (default 10)"
  },
  "exclude": {
   "type": "array",
   "description": "Skip URLs whose path+query contains one of these substrings"
  },
  "include": {
   "type": "array",
   "description": "Only crawl URLs whose path+query contains one of these substrings"
  },
  "maxChars": {
   "type": "number",
   "description": "Per-page content character cap (default 8000)"
  },
  "maxDepth": {
   "type": "number",
   "maximum": 3,
   "description": "Link depth from the start URL, 0-3 (default 2)"
  },
  "respectRobots": {
   "type": "boolean",
   "description": "Honour robots.txt (default true)"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com",
  "pages": [
   {
    "url": "https://example.com",
    "depth": 0,
    "title": "Example",
    "status": "success",
    "content": "# Example\\n\\nWelcome...",
    "truncated": false
   },
   {
    "url": "https://example.com/about",
    "depth": 1,
    "title": "About",
    "status": "success",
    "content": "# About us...",
    "truncated": false
   }
  ],
  "summary": {
   "limit": 10,
   "failed": 0,
   "crawled": 2,
   "maxDepth": 2,
   "discovered": 5,
   "successful": 2,
   "robotsRespected": true
  },
  "crawledAt": "2026-07-31T12:00:00.000Z",
  "requestId": "req_crawl123"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/weblens-crawl-api-e5c8cde4/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.weblens.dev](https://www.zero.xyz/host/api.weblens.dev/llms.txt)