# AgentRXI Web Crawler

> AgentRXI Web Crawler is a paid API for AI agents from agentrxi.com, paid per call via x402, $0.025/call, status unknown (last checked 2026-10-15).

Crawls a website starting from a given URL, following same-site links up to 100 pages, and returns each page's URL, title, and clean Markdown text

## Facts

- Endpoint: GET https://agentrxi.com/v1/crawl?utm_source=zero.xyz
- Price: $0.025/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agentrxi-web-crawler-47949070
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_HH2iYyL4NDRWwqatLC1zM

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agentrxi-web-crawler-47949070
```

Example prompt: Crawl https://docs.example.com starting under the /api/ path, fetch up to 50 pages, and give me all the text as clean markdown so I can use it for RAG.

## When to prefer this

Use this endpoint when you need to extract clean, readable text from multiple pages of a website — especially documentation sites, blogs, or wikis — rather than scraping a single URL. It is ideal for building RAG corpora, competitive research, or offline archiving when you need structured Markdown output across many linked pages without maintaining your own crawler infrastructure. Prefer it when you want to scope crawling to a specific path prefix (e.g. /docs/) and don't want to handle link-following logic yourself.

## Known failure modes

- Start URL is unreachable or returns non-200 status — no pages returned
- robots.txt disallows crawling — pages may be skipped or crawl halted
- Path prefix excludes all linked pages — only start page returned
- max_pages parameter exceeds 100 — clamped or error returned
- Site uses heavy JavaScript rendering — content may be incomplete or missing
- Payment failure or insufficient USDC balance — 402 response, no data returned

## How this service works

Crawl a website into clean Markdown: follows same-site links from a start page (optionally under a path prefix), up to 100 pages, respecting robots.txt. Returns each page's URL, title and text. $0.005 + $0.001 per page. Add store=true (+$0.001) to get a 7-day share link instead of the data inline.

## Output

A list of up to 100 page objects, each containing the page URL, its title, and the full page body converted to clean Markdown text. If store=true was requested, returns a 7-day shareable link to the dataset instead of inline data.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Start page"
      },
      "store": {
       "type": "boolean",
       "description": "Return a 7-day share link (+$0.001)"
      },
      "max_pages": {
       "type": "integer",
       "description": "1-100, default 20"
      },
      "path_prefix": {
       "type": "string",
       "description": "Only follow links under this path, e.g. /docs/"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agentrxi-web-crawler-47949070/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agentrxi.com](https://www.zero.xyz/host/agentrxi.com/llms.txt)