# FastX402 Robots.txt & Sitemap Checker

> FastX402 Robots.txt & Sitemap Checker is a paid API for AI agents from fastx402.tools, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-27).

Fetches and parses a website's robots.txt to determine if a given path is allowed for a given user-agent, and retrieves its sitemap URLs.

## Facts

- Endpoint: GET https://fastx402.tools/robots-sitemap?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-27
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/fastx402-robots-txt-sitemap-checker-52df0c4e
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_2KEtDdizl8-kyj030DL-z

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability fastx402-robots-txt-sitemap-checker-52df0c4e
```

Example prompt: Can you check whether a bot called 'MyBot' is allowed to crawl the path /wiki/Special:Search on wikipedia.org, and also grab any sitemap URLs listed there?

## When to prefer this

Choose this endpoint when you need to programmatically check crawl permissions before scraping a site, discover sitemap URLs without manual inspection, or audit robots.txt compliance for a specific bot and path — all without needing an API key, using micro-payments via x402 on Base or Solana.

## Known failure modes

- robots.txt not found (HTTP 404) — found:false returned
- sitemap URL returns non-XML or HTTP error — error field populated in sitemap object
- domain unreachable or invalid URL — error response
- rate limiting or timeout on target domain fetch
- sitemap HTTP 403 — noted in sitemap error field

## How this service works

200 pay-per-call tools for AI agents, from $0.001 per request in USDC on Base or Solana with x402. No account, no API key.

## Output

Returns a JSON object with: whether the path is allowed for the given user-agent, the matched rule and group, the full robots.txt parse result (groups, rules count, crawl delays), all discovered sitemap URLs with their status, the robots.txt URL fetched, and the origin domain.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Site URL or bare domain"
      },
      "path": {
       "type": "string",
       "description": "Path to check, starting with /, default /"
      },
      "agent": {
       "type": "string",
       "description": "User-agent name to check, default *"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "properties": {
      "query": {
       "type": "object",
       "properties": {
        "path": {
         "type": "string"
        },
        "agent": {
         "type": "string"
        }
       }
      },
      "origin": {
       "type": "string"
      },
      "robots": {
       "type": "object",
       "properties": {
        "found": {
         "type": "boolean"
        },
        "groups": {
         "type": "array",
         "items": {
          "type": "object",
          "properties": {
           "rules": {
            "type": "integer"
           },
           "agents": {
            "type": "array",
            "items": {
             "type": "string"
            }
           },
           "crawlDelay": {
            "type": "null"
           }
          }
         }
        },
        "status": {
         "type": "integer"
        }
       }
      },
      "allowed": {
       "type": "boolean"
      },
      "sitemap": {
       "type": "object",
       "properties": {
        "url": {
         "type": "string"
        },
        "error": {
         "type": "string"
        }
       }
      },
      "sitemaps": {
       "type": "array",
       "items": {
        "type": "string"
       }
      },
      "robotsUrl": {
       "type": "string"
      },
      "crawlDelay": {
       "type": "null"
      },
      "matchedRule": {
       "type": "string"
      },
      "matchedGroup": {
       "type": "array",
       "items": {
        
… (truncated)
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "query": {
   "path": "/wiki/Special:Search",
   "agent": "MyBot"
  },
  "origin": "https://wikipedia.org",
  "robots": {
   "found": true,
   "groups": [
    {
     "rules": 1,
     "agents": [
      "mj12bot"
     ],
     "crawlDelay": null
    },
    {
     "rules": 0,
     "agents": [
      "semrushbot"
     ],
     "crawlDelay": 5
    },
    {
     "rules": 432,
     "agents": [
      "*"
     ],
     "crawlDelay": null
    }
   ],
   "status": 200
  },
  "allowed": false,
  "sitemap": {
   "url": "https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0",
   "error": "no sitemap XML (HTTP 403)"
  },
  "sitemaps": [
   "https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0"
  ],
  "robotsUrl": "https://wikipedia.org/robots.txt",
  "crawlDelay": null,
  "matchedRule": "Disallow: /wiki/Special:",
  "matchedGroup": [
   "*"
  ]
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/fastx402-robots-txt-sitemap-checker-52df0c4e/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from fastx402.tools](https://www.zero.xyz/host/fastx402.tools/llms.txt)