# x402.orthogonal.com Web Crawler

> x402.orthogonal.com Web Crawler is a paid API for AI agents from x402.orthogonal.com, paid per call via x402, $0.03/call, status unknown (last checked 2026-10-19).

Crawls a website starting from a given URL, follows internal links up to a configurable depth, and returns all page content as Markdown

## Facts

- Endpoint: POST https://x402.orthogonal.com/context-dev/web/crawl?utm_source=zero.xyz
- Price: $0.03/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-19
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/x402-orthogonal-com-web-crawler-59106162
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_sU7RMBFvjuV3Ddz4bvLWj

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability x402-orthogonal-com-web-crawler-59106162 -d '<json body>'
```

Example prompt: Crawl https://docs.example.com up to 2 levels deep, limit it to 50 pages, only follow URLs matching /docs/, and give me all the content as clean Markdown with links included but without headers, footers, or navigation.

## When to prefer this

Use this endpoint when you need to extract content from multiple pages of a website automatically, following internal links rather than scraping a single page. It is ideal for documentation indexing, competitive research, content audits, and knowledge base construction where you need breadth across an entire domain. Prefer it over a single-page scraper when the target content spans many URLs. Avoid it for real-time lookups where latency matters, since multi-page crawls are inherently batch operations.

## Known failure modes

- Invalid or unreachable starting URL returns an error
- maxPages cap of 500 hard-limits results regardless of site size
- URL regex pattern errors may silently exclude pages
- Sites blocking crawlers (robots.txt or rate-limiting) may yield partial results
- Deep or large sites may time out before reaching maxPages
- base64 images may inflate response size if shortenBase64Images is false

## How this service works

Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages. Only follows links within the same domain as the starting URL. Costs 1 credit per successful page crawled.

## Output

Returns a list of crawled pages, each with its URL and extracted content rendered as Markdown. Optionally includes hyperlinks and image references within the Markdown. Only pages within the same domain (and optionally subdomains) as the starting URL are included. Costs 1 credit per successfully crawled page.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "The starting URL for the crawl (must include http:// or https:// protocol)"
  },
  "maxDepth": {
   "type": "integer",
   "description": "Maximum link depth from the starting URL (0 = only the starting page)"
  },
  "maxPages": {
   "type": "integer",
   "description": "Maximum number of pages to crawl. Hard cap: 500."
  },
  "urlRegex": {
   "type": "string",
   "description": "Regex pattern. Only URLs matching this pattern will be followed and scraped."
  },
  "includeLinks": {
   "type": "boolean",
   "description": "Preserve hyperlinks in the Markdown output"
  },
  "includeImages": {
   "type": "boolean",
   "description": "Include image references in the Markdown output"
  },
  "followSubdomains": {
   "type": "boolean",
   "description": "When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent."
  },
  "useMainContentOnly": {
   "type": "boolean",
   "description": "Extract only the main content, stripping headers, footers, sidebars, and navigation"
  },
  "shortenBase64Images": {
   "type": "boolean",
   "description": "Truncate base64-encoded image data in the Markdown output"
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/x402-orthogonal-com-web-crawler-59106162/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.orthogonal.com](https://www.zero.xyz/host/x402.orthogonal.com/llms.txt)