# Web Domain Crawler with Keyword Scanning

> Web Domain Crawler with Keyword Scanning is a paid API for AI agents from store.agentexchange.work, paid per call via x402, $0.09/call, status unknown (last checked 2026-09-15).

Crawls internal links of a domain starting from a given URL, discovers site structure, and optionally scans pages for a target keyword or signal.

## Facts

- Endpoint: GET https://store.agentexchange.work/web/crawl
- Price: $0.09/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/web-domain-crawler-with-keyword-scanning-0395b2c6
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_IbzWUP9xR80jfdWFAq3UL

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability web-domain-crawler-with-keyword-scanning-0395b2c6
```

Example prompt: Crawl https://example.com and follow up to 5 internal links to map the site structure, and flag any pages that mention the keyword 'pricing'.

## When to prefer this

Use this endpoint when you need to programmatically discover the internal link structure or page inventory of a website, or when you want to scan multiple pages of a site for a specific keyword or signal. It is ideal for lightweight site audits, competitive research, content discovery, or verifying whether a topic is covered anywhere on a domain — all within a single pay-per-call API with no authentication setup required.

## Known failure modes

- URL is unreachable or returns non-200 status — crawl returns empty or partial results
- Domain blocks crawlers via robots.txt or rate limiting — fewer pages returned than requested
- limit parameter out of range (must be 1–5) — may return error or default behavior
- No internal links found — returns empty link array with only the starting page
- Keyword not found on any page — returns results with empty keyword match list

## How this service works

Crawl internal links of a domain, discover site structure, and scan for target keywords or signals.

## Output

Returns the starting URL, a list of crawled page URLs, the total count of pages crawled, all discovered internal links across those pages, and — if a keyword was provided — which pages contained that keyword.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Starting URL to crawl"
      },
      "limit": {
       "type": "number",
       "description": "Max pages to crawl, 1-5 (default 3)"
      },
      "keyword": {
       "type": "string",
       "description": "Target keyword to scan for (optional)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "properties": {
      "keyword": {
       "type": "string"
      },
      "starting_url": {
       "type": "string"
      },
      "crawled_pages": {
       "type": "array"
      },
      "pages_crawled_count": {
       "type": "number"
      },
      "all_discovered_internal_links": {
       "type": "array"
      }
     }
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/web-domain-crawler-with-keyword-scanning-0395b2c6/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from store.agentexchange.work](https://www.zero.xyz/host/store.agentexchange.work/llms.txt)