# agentsvc.io Site Crawl

> agentsvc.io Site Crawl is a paid API for AI agents from agentsvc.io, paid per call via x402, $0.005/call, status unknown (last checked 2026-10-10).

Crawls a website starting from a given URL, following internal links and returning page content as markdown with metadata for up to 20 pages.

## Facts

- Endpoint: POST https://agentsvc.io/api/v1/proxy/site-crawl?utm_source=zero.xyz
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-10
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agentsvc-io-site-crawl-a7472a33
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_kUJGpK9OqZvXxXuJb4KGW

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agentsvc-io-site-crawl-a7472a33 -d '<json body>'
```

Example prompt: Can you crawl https://docs.example.com for me, following up to 10 pages but only under the /api path, and give me the full text content of each page?

## When to prefer this

Use this endpoint when you need to extract content from multiple pages of a site in a single call, especially when you want to follow internal links automatically under a specific path prefix. Prefer it over a single-page scraper when the target content spans multiple pages (e.g. documentation, blogs, product catalogs). It is ideal for indexing, summarization, or competitive research tasks where breadth of coverage matters and you can specify a path scope to stay focused.

## Known failure modes

- Target site blocks crawlers or returns 403/429 — affected pages marked ok:false with status code
- URL is unreachable or DNS fails — page entry with ok:false returned
- max_pages or max_chars limits truncate content before full site is covered
- path_prefix is too restrictive and no matching links are found — returns fewer pages than expected
- Site uses JavaScript rendering and content is not in raw HTML — markdown may be incomplete or empty
- Payment fails or USDC balance insufficient — HTTP 402 returned before crawl starts

## How this service works

86 APIs for AI agents: web scraping and crawling to Markdown, AI summaries, data extraction and cited answers, screenshots and PDF, PDF, OCR and document conversion, news, paper and developer search and more, paid per call with USDC on Base, Solana, Polygon or Arbitrum via x402. No account, no API key, free sample on every service.

## Output

Returns a JSON object containing a list of crawled pages, each with its URL, HTTP status, title, meta description, markdown content, and word count. Also includes overall stats: number of pages crawled, discovered URLs, skipped non-HTML resources, total time taken, and the crawl timestamp.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string"
  },
  "max_chars": {
   "type": "integer",
   "default": 4000,
   "maximum": 20000,
   "minimum": 500
  },
  "max_pages": {
   "type": "integer",
   "default": 5,
   "maximum": 20,
   "minimum": 1
  },
  "path_prefix": {
   "type": "string",
   "description": "Only follow links whose path starts with this, e.g. /docs"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "data": {
   "site": "https://agentsvc.io",
   "pages": [
    {
     "ok": true,
     "url": "https://agentsvc.io/docs",
     "title": "Docs: how agents pay per call with x402 · agentsvc.io",
     "status": 200,
     "markdown": "Documentation\n\n# Pay per call. Nothing else to set up.\n\nEvery service is a JS...",
     "word_count": 104,
     "description": "Quickstart, payment protocol (x402 v1 and v2), free trial, error codes and th..."
    }
   ],
   "took_ms": 125,
   "ok_count": 2,
   "crawled_at": "2026-10-06T14:17:50.682Z",
   "pages_crawled": 2,
   "discovered_urls": 37,
   "skipped_non_html": 2
  },
  "success": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agentsvc-io-site-crawl-a7472a33/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agentsvc.io](https://www.zero.xyz/host/agentsvc.io/llms.txt)