# Agent402.Tools Site Map Discovery

> Agent402.Tools Site Map Discovery is a paid API for AI agents from agent402.tools, paid per call via x402, $0.005/call, status unknown (last checked 2026-09-14).

Crawls a website and returns a structured list of discovered URLs from its sitemap, robots.txt, and page links

## Facts

- Endpoint: POST https://agent402.tools/api/site-map
- Price: $0.005/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agent402-tools-site-map-discovery-5f6ceba0
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_LAzzrU-0cHtylJHpCbwQR

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agent402-tools-site-map-discovery-5f6ceba0 -d '<json body>'
```

Example prompt: Can you crawl https://www.example.com and give me a list of up to 200 of its pages, but only ones that include the word 'blog' in the URL?

## When to prefer this

Use this endpoint when you need to discover the full set of URLs for a given website quickly without running a full recursive spider — it synthesizes robots.txt, sitemap XML, and page links in a single call. Prefer it over manual sitemap parsing when you want a normalized, deduplicated list with optional substring filtering. It's ideal for site auditing, content indexing, competitor research, and feeding downstream scraping pipelines.

## Known failure modes

- Site has no sitemap or robots.txt, resulting in fewer discovered URLs or empty list
- Start URL is unreachable or returns non-200 status, causing fetch failure
- Limit parameter outside 1-500 range may be rejected or clamped
- Search filter returns zero matches if substring doesn't appear in any URL
- Subdomain URLs missed if includeSubdomains is false (default)
- Large sites may be truncated when total URLs exceed the requested limit

## How this service works

Discover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host, normalized, deduplicated list (up to 500) with an optional substring filter. Hard budgets: at most 6 fetches, 15 seconds, 5 MB. Use it to pick which pages to crawl or extract next.

## Output

A JSON object containing the start URL, host, an array of discovered URLs (from robots.txt, sitemaps, and page links), total URL count, optional search filter applied, data sources used, number of HTTP fetches made, per-source URL counts, any warnings, fetch timestamp, and whether the result was truncated due to the limit.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Start URL (the site's homepage or any page on it)"
  },
  "limit": {
   "type": "integer",
   "description": "Max URLs to return, 1-500 (default 100)"
  },
  "search": {
   "type": "string",
   "description": "Optional case-insensitive substring filter applied to the discovered URLs"
  },
  "includeSubdomains": {
   "type": "boolean",
   "description": "Also keep URLs on subdomains of the start site (default false; www and bare host always count as one site)"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://www.iana.org/",
  "host": "www.iana.org",
  "urls": [
   "https://www.iana.org/",
   "https://www.iana.org/domains",
   "https://www.iana.org/numbers",
   "https://www.iana.org/protocols"
  ],
  "total": 120,
  "search": null,
  "source": "robots.txt, sitemap(s) and start-page links, fetched live",
  "fetches": 3,
  "sources": {
   "links": 24,
   "sitemap": 96
  },
  "warnings": [],
  "fetchedAt": "2026-08-22T00:00:00.000Z",
  "truncated": true,
  "sitemapsRead": 1
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agent402-tools-site-map-discovery-5f6ceba0/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agent402.tools](https://www.zero.xyz/host/agent402.tools/llms.txt)