# Site Crawler

> Site Crawler is a paid API for AI agents from 164.160.90.241.sslip.io, paid per call via x402, $0.05/call, status unknown (last checked 2026-10-12).

Crawls a website (sitemap first, then same-site links) respecting robots.txt and returns up to 20 pages as clean Markdown

## Facts

- Endpoint: GET https://164.160.90.241.sslip.io/api/web-crawl-site?utm_source=zero.xyz
- Price: $0.05/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-12
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/site-crawler-b685651f
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_sZ1s2QJt4_G6E00alYJqk

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability site-crawler-b685651f
```

Example prompt: Can you crawl https://docs.example.com and return up to 15 pages of content as Markdown, staying only within the /guides path?

## When to prefer this

Choose this endpoint when you need to crawl multiple pages from a site automatically rather than fetching a single URL. It is ideal for ingesting documentation sites, gathering competitive intelligence across a whole domain, or building a Markdown corpus from a structured website. Prefer it over a single-page reader when you don't know the exact URLs in advance and need the crawler to discover them via sitemap or link traversal. Use path_prefix to scope the crawl to a specific section like /docs or /blog.

## Known failure modes

- robots.txt blocks crawling of the target domain — returns restricted or empty results
- Start URL is unreachable or returns non-200 status — crawl fails with an error
- Site has no sitemap and no crawlable links — returns only the start page or nothing
- path_prefix filters out all discovered URLs — returns zero pages
- Site requires JavaScript rendering to expose links — crawler may miss dynamically generated pages
- max_pages set to 0 or above 20 — validation error

## How this service works

Crawl a website (sitemap first, then same-site links), respecting robots.txt, and return up to 20 pages as clean Markdown. Use path_prefix to stay inside a docs section.

## Output

Returns up to 20 pages from the target website as clean Markdown text. The crawler starts from the sitemap if available, then follows same-site links, and respects robots.txt. Each page's content is converted to readable Markdown with titles and URLs preserved.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Start URL"
      },
      "max_pages": {
       "type": "integer",
       "description": "Pages to return, 1-20 (default 10)"
      },
      "path_prefix": {
       "type": "string",
       "description": "Only crawl URLs whose path starts with this"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/site-crawler-b685651f/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from 164.160.90.241.sslip.io](https://www.zero.xyz/host/164.160.90.241.sslip.io/llms.txt)