# x402.orth.sh Web Crawler

> x402.orth.sh Web Crawler is a paid API for AI agents from x402.orth.sh, paid per call via x402, $0.03/call, status unknown (last checked 2026-09-15).

Crawls a website starting from a given URL, follows internal links up to a configurable depth, and returns all crawled pages as Markdown content.

## Facts

- Endpoint: POST https://x402.orth.sh/context-dev/web/crawl
- Price: $0.03/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/x402-orth-sh-web-crawler-93de2788
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_A3cH8fC4jjM_JplVmXdAq

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability x402-orth-sh-web-crawler-93de2788 -d '<json body>'
```

Example prompt: Crawl https://docs.example.com up to 3 levels deep, limit to 50 pages, and return the content of each page as Markdown — only extract the main content and skip headers and footers.

## When to prefer this

Use this endpoint when you need to extract content from multiple pages of a single website in one call, rather than scraping individual pages one at a time. It is ideal for indexing documentation sites, blogs, or product catalogs where following internal links systematically is required. Prefer this over single-page scrapers when depth and breadth of crawl coverage matter. Choose it over LLM-driven extraction when you want raw Markdown output across many pages without a specific extraction prompt.

## Known failure modes

- Invalid or unreachable starting URL returns an error
- maxPages cap of 500 enforced — requests exceeding this are capped silently
- urlRegex syntax errors may reject or ignore the filter
- Crawl may return fewer pages than requested if the domain has fewer internal links
- Pages behind authentication or JavaScript-heavy SPAs may return empty or partial content
- Rate limiting or bot-blocking by the target site may yield incomplete results
- Insufficient credits causes the request to fail mid-crawl

## How this service works

Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages. Only follows links within the same domain as the starting URL. Costs 1 credit per successful page crawled.

## Output

An array of crawled page results, each containing the page URL and its full content converted to Markdown format. Links and images can optionally be preserved in the output. Up to 500 pages may be returned per request.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "The starting URL for the crawl (must include http:// or https:// protocol)"
  },
  "maxDepth": {
   "type": "integer",
   "description": "Maximum link depth from the starting URL (0 = only the starting page)"
  },
  "maxPages": {
   "type": "integer",
   "description": "Maximum number of pages to crawl. Hard cap: 500."
  },
  "urlRegex": {
   "type": "string",
   "description": "Regex pattern. Only URLs matching this pattern will be followed and scraped."
  },
  "includeLinks": {
   "type": "boolean",
   "description": "Preserve hyperlinks in the Markdown output"
  },
  "includeImages": {
   "type": "boolean",
   "description": "Include image references in the Markdown output"
  },
  "followSubdomains": {
   "type": "boolean",
   "description": "When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent."
  },
  "useMainContentOnly": {
   "type": "boolean",
   "description": "Extract only the main content, stripping headers, footers, sidebars, and navigation"
  },
  "shortenBase64Images": {
   "type": "boolean",
   "description": "Truncate base64-encoded image data in the Markdown output"
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/x402-orth-sh-web-crawler-93de2788/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.orth.sh](https://www.zero.xyz/host/x402.orth.sh/llms.txt)