# Robots.txt & Sitemap Checker

> Robots.txt & Sitemap Checker is a paid API for AI agents from bess-x402.onrender.com, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-30).

Fetches a domain's robots.txt crawl rules (including AI crawler blocking status) and sitemap URLs in a single call.

## Facts

- Endpoint: GET https://bess-x402.onrender.com/v1/web/robots?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-30
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-sitemap-checker-a7f01d34
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_rQf2nmWfsx9L736kPm77o

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-sitemap-checker-a7f01d34
```

Example prompt: Before I crawl techcrunch.com, check its robots.txt to see if GPTBot and ClaudeBot are blocked, and also pull up to 50 URLs from its sitemap.

## When to prefer this

Use this endpoint when you need to check crawl permissions before scraping a site, perform an SEO audit of robots.txt directives, or discover sitemap URLs without manually fetching and parsing these files yourself. It is especially valuable when you need quick answers about whether specific AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended) are blocked on a target domain.

## Known failure modes

- Domain not found or unreachable — returns error indicating the host could not be resolved
- No robots.txt present — may return empty rules or a not-found indicator
- Sitemap not declared in robots.txt — sitemap URLs list may be empty
- Invalid domain format — returns validation error
- Limit parameter out of range (not 1-1000) — returns validation error

## How this service works

A site's robots.txt rules per user agent, whether AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) are blocked, and the URLs listed in its sitemap. Use before crawling a site or for SEO audits.

## Output

Returns the parsed robots.txt rules per user agent, a clear flag for whether each major AI crawler (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) is blocked or allowed, and a list of URLs from the domain's sitemap (up to the requested limit).

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain"
     ],
     "properties": {
      "limit": {
       "type": "integer",
       "description": "Sitemap URLs to return (1-1000)"
      },
      "domain": {
       "type": "string",
       "description": "Domain name"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-sitemap-checker-a7f01d34/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from bess-x402.onrender.com](https://www.zero.xyz/host/bess-x402.onrender.com/llms.txt)