# Sift Web Tools – Sitemap Crawler

> Sift Web Tools – Sitemap Crawler is a paid API for AI agents from sift.srv1996466.hstgr.cloud, paid per call via x402, $0.003/call, status unknown (last checked 2026-10-23).

Fetches and returns all URLs from a website's sitemap, up to a configurable limit, as structured JSON.

## Facts

- Endpoint: GET https://sift.srv1996466.hstgr.cloud/v1/sitemap?utm_source=zero.xyz
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-23
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sift-web-tools-sitemap-crawler-f8aeeb94
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_kCw7SBBPikwPbe3rJcxv8

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sift-web-tools-sitemap-crawler-f8aeeb94
```

Example prompt: Can you pull the full sitemap for blog.cloudflare.com and list all the URLs you find, up to 500?

## When to prefer this

Use this endpoint when you need to discover all URLs published on a website programmatically, especially for SEO audits, content inventories, competitor research, or setting up crawl pipelines. Prefer this over manual browsing or generic fetch tools when you need a structured, machine-readable list of all site pages with modification timestamps.

## Known failure modes

- Site has no sitemap.xml — returns empty URL list or error
- Site blocks crawlers via robots.txt — may return empty or partial results
- URL limit exceeded — truncated to 2000 or specified limit without error
- Invalid or unreachable URL — HTTP error or timeout
- Malformed sitemap XML — parsing failure with error response

## How this service works

List a website's URLs from its sitemaps. Reads robots.txt and sitemap indexes (gzip supported) and returns up to 2000 URLs with lastmod dates.

## Output

Returns a JSON object containing the canonical site URL, an array of URL entries each with a `loc` (URL) and optional `lastmod` (ISO 8601 timestamp), and a `url_count` integer reflecting the total number of URLs returned (capped at the requested limit or 2000 by default).

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Any URL on the site"
      },
      "limit": {
       "type": "integer",
       "description": "Max URLs (default 2000)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "site": "https://blog.cloudflare.com",
  "urls": [
   {
    "loc": "https://blog.cloudflare.com/enabling-google-analytics-on-wordpress-blog",
    "lastmod": "2026-07-15T12:25:22.052Z"
   }
  ],
  "url_count": 2000
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sift-web-tools-sitemap-crawler-f8aeeb94/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from sift.srv1996466.hstgr.cloud](https://www.zero.xyz/host/sift.srv1996466.hstgr.cloud/llms.txt)