# Agent Hub Website Sitemap Crawler

> Agent Hub Website Sitemap Crawler is a paid API for AI agents from agents.torbrooknews.com, paid per call via x402, $0.02/call, status unknown (last checked 2026-10-13).

Crawls a website's robots.txt and all sitemaps (including sitemap indexes) to list up to 5000 URLs with last-modified dates, falling back to on-page link discovery.

## Facts

- Endpoint: GET https://agents.torbrooknews.com/v1/sitemap?utm_source=zero.xyz
- Price: $0.02/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agent-hub-website-sitemap-crawler-fe09228c
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_duJ33p2PdF4WjMlGI8-wa

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agent-hub-website-sitemap-crawler-fe09228c
```

Example prompt: Can you list all the pages on example.com — pull the sitemap and robots.txt and give me up to 2000 URLs with their last-modified dates?

## When to prefer this

Choose this endpoint when you need a structured, comprehensive list of all pages on a website without writing a custom crawler. It is preferable to manual scraping because it intelligently reads robots.txt and nested sitemap indexes first, falling back to link discovery only when needed. Best for SEO audits, content inventories, site migrations, or seeding a web crawl pipeline.

## Known failure modes

- Target website blocks crawlers or returns 403/401, resulting in no URLs returned
- Website has no sitemap and on-page link fallback finds very few pages
- URL limit exceeded — response capped at the requested limit (default 500, max 5000)
- Invalid or unreachable URL provided returns an error
- Very large sitemap indexes may time out before full enumeration

## How this service works

List every page on a website: reads robots.txt and all sitemaps (including sitemap indexes), with last-modified dates; falls back to on-page links. Up to 5000 URLs. Also callable as GET with the same fields as query parameters.

## Output

A list of up to 5000 URLs found on the target website, sourced from robots.txt, sitemap.xml, sitemap index files, and/or on-page links as fallback. Each entry typically includes the page URL and its last-modified date where available.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string"
      },
      "limit": {
       "type": "integer",
       "description": "1 to 5000, default 500"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agent-hub-website-sitemap-crawler-fe09228c/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agents.torbrooknews.com](https://www.zero.xyz/host/agents.torbrooknews.com/llms.txt)