# ScrapeGraphAI Website Crawler

> ScrapeGraphAI Website Crawler is a paid API for AI agents from x402.orthogonal.com, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-14).

Crawls a website starting from a seed URL with configurable depth, page limits, URL filtering, and output format options.

## Facts

- Endpoint: POST https://x402.orthogonal.com/scrapegraphai/api/crawl
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/scrapegraphai-website-crawler-f838ac88
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_L-H3AEBFJmaGZ4pUEcZZ1

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability scrapegraphai-website-crawler-f838ac88 -d '<json body>'
```

Example prompt: Crawl the website at https://docs.example.com starting from the homepage, going up to 3 levels deep, and visiting no more than 100 pages — return the content in markdown format and skip any URLs containing '/blog/'.

## When to prefer this

Use this endpoint when you need to recursively collect content from multiple pages of a website rather than scraping a single URL. It is best suited for documentation indexing, site auditing, competitive intelligence gathering, or building training datasets from web content. Choose it over single-URL scrapers when you need breadth across a domain with configurable depth and filtering.

## Known failure modes

- Seed URL is unreachable or returns non-200 status — crawl returns empty or error
- maxPages or maxDepth set to values outside allowed range — validation error
- Target site blocks crawlers (robots.txt or rate limiting) — partial or empty results
- Malformed URL in input — request rejected with validation error
- External link following disabled by default — pages outside seed domain silently skipped

## How this service works

Start a website crawl from a seed URL with configurable depth and page limits.

## Output

Returns structured data from all crawled pages, including page content in the requested output formats (e.g. markdown, HTML), URLs visited, and metadata. The crawl respects depth and page count limits and optionally filters URLs using include/exclude patterns.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Seed URL to crawl"
  },
  "formats": {
   "type": "array",
   "description": "Output formats"
  },
  "maxDepth": {
   "type": "number",
   "description": "Max crawl depth (default 2)"
  },
  "maxPages": {
   "type": "number",
   "description": "Max pages (1-1000, default 50)"
  },
  "fetchConfig": {
   "type": "object",
   "description": "Fetch options"
  },
  "contentTypes": {
   "type": "array",
   "description": "Allowed content types"
  },
  "allowExternal": {
   "type": "boolean",
   "description": "Follow external links (default false)"
  },
  "excludePatterns": {
   "type": "array",
   "description": "URL patterns to exclude"
  },
  "includePatterns": {
   "type": "array",
   "description": "URL patterns to include"
  },
  "maxLinksPerPage": {
   "type": "number",
   "description": "Max links per page (default 10)"
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/scrapegraphai-website-crawler-f838ac88/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.orthogonal.com](https://www.zero.xyz/host/x402.orthogonal.com/llms.txt)