# Grey Ridge Web Crawl API (x402)

> Grey Ridge Web Crawl API (x402) is a paid API for AI agents from assetfactory-jobs-x402.sigrunner.workers.dev, paid per call via x402, $0.004/call, status unknown (last checked 2026-11-18).

Crawls a seed URL breadth-first, following same-domain links up to a configurable depth, and returns clean markdown, text, metadata, and links for every page reached — including JS-rendered/SPA content.

## Facts

- Endpoint: HEAD https://assetfactory-jobs-x402.sigrunner.workers.dev/web/crawl?utm_source=zero.xyz
- Price: $0.004/call
- Payment: x402
- Status: unknown
- Last checked: 2026-11-18
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/grey-ridge-web-crawl-api-x402-f005427d
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_DUy96zXO0atnfTmuQ4AvL

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability grey-ridge-web-crawl-api-x402-f005427d -d '<json body>'
```

Example prompt: Crawl https://docs.example.com starting from the homepage, follow links up to 2 levels deep, limit to 20 pages, only include URLs containing /docs, and give me clean markdown for every page you find.

## When to prefer this

Choose this endpoint when you need to retrieve content from multiple pages of a site in a single paid call, especially when link-following, depth control, or JS/SPA rendering is required. It is superior to single-page fetch endpoints when you don't know the full URL list in advance and want breadth-first discovery. The per-call pricing (not per-page) makes it cost-effective for bulk extraction up to 50 pages. Prefer it over manual URL enumeration when the site structure is unknown or when sitemap-seeding is needed for SPAs.

## Known failure modes

- Seed URL unreachable or returns non-2xx — page entry marked ok:false with status code
- JS rendering timeout on heavily dynamic pages — recovered_from field may indicate fallback
- Crawl hits page limit before exhausting frontier — frontier_truncated:true in stats
- include/exclude filters too restrictive — fewer pages returned than limit
- Site blocks crawlers via robots.txt or rate limiting — individual pages fail silently with error status
- Invalid or malformed seed URL — request rejected before crawl begins

## How this service works

A keyless web-retrieval and data API for AI agents. Retrieve live web pages (clean markdown/text, metadata, links, JSON-LD; recovers JS-rendered/SPA content), detect when a page changes, enumerate a site’s URLs, enrich a company from its domain, verify email deliverability, and read a maintained normalized jobs feed across 10 applicant-tracking systems (Greenhouse, Lever, Ashby, SmartRecruiters, Workday, Recruitee, Workable, BambooHR, Breezy, Personio) with company-domain auto-detection. Every endpoint reads live at request time. Pay per call in USDC on Base via x402 (HTTP 402) — no account, no API key, no subscription. Sources are drift-monitored and maintained so the data does not rot.

## Output

A JSON object containing an array of page results (each with URL, title, clean text, markdown, HTTP status, word/char counts, depth, JS-recovery flag, injection suspicion flag) plus aggregate crawl stats (pages crawled, discovered, failed, depth reached, frontier truncation flag) and a generated_at timestamp.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Seed page URL. The crawl starts here, follows same-domain links breadth-first and returns clean content for every page it reaches. The buyer does not need to know the page list in advance."
      },
      "limit": {
       "type": "integer",
       "default": 15,
       "maximum": 50,
       "description": "Hard ceiling on pages crawled (and billed as one call). Bounded downside."
      },
      "exclude": {
       "type": "string",
       "description": "Skip URLs whose path/query contains this substring (e.g. /author/)."
      },
      "include": {
       "type": "string",
       "description": "Only crawl URLs whose path/query contains this case-insensitive substring (e.g. /docs)."
      },
      "maxChars": {
       "type": "integer",
       "default": 20000,
       "maximum": 200000,
       "description": "Per-page ceiling on returned text / markdown length."
      },
      "maxDepth": {
       "type": "integer",
       "default": 2,
       "maximum": 4,
       "description": "Link-follow depth from the seed (0 = seed only)."
      },
      "sameHost": {
       "type": "boolean",
       "default": false,
       "description": "true restricts the crawl to the seed's exact host; default follows the whole registrable domain (e.g. jobs.stripe.com from stripe.com)."
      },
      "useSitemap": {
       "type": "boolean",
       "default": false,
       "description": "Also seed the frontier from the site's sitemap so a link-sparse SPA still reaches real pages."
      },
      "includeLinks": {
       "type": "boolean",
       "default": false,
       "description": "Include each page's outbound-link graph in the output (always used internally for discovery)."
      },
      "includeMarkdown": {
       "type": "boolean",
       "default": true,
       "description": "Return a light Markdown rendering per page."
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [

… (truncated)
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "ok": true,
  "seed": "https://blog.cloudflare.com/",
  "pages": [
   {
    "ok": true,
    "url": "https://blog.cloudflare.com/",
    "kind": "html",
    "text": "clean main-content prose ...",
    "depth": 0,
    "title": "The Cloudflare Blog",
    "status": 200,
    "markdown": "## ...",
    "char_count": 6248,
    "word_count": 858,
    "recovered_from": null,
    "injection_suspected": false
   },
   {
    "ok": true,
    "url": "https://blog.cloudflare.com/dns-targeted-zone-builds/",
    "kind": "html",
    "text": "...",
    "depth": 1,
    "title": "How we halved DNS build delay",
    "status": 200,
    "char_count": 14000,
    "word_count": 2100
   }
  ],
  "stats": {
   "ok": 15,
   "limit": 15,
   "scope": "same_domain",
   "failed": 0,
   "crawled": 15,
   "max_depth": 2,
   "seed_host": "blog.cloudflare.com",
   "discovered": 55,
   "max_depth_reached": 2,
   "frontier_truncated": true
  },
  "generated_at": "2026-11-04T00:00:00.000Z"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/grey-ridge-web-crawl-api-x402-f005427d/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from assetfactory-jobs-x402.sigrunner.workers.dev](https://www.zero.xyz/host/assetfactory-jobs-x402.sigrunner.workers.dev/llms.txt)
