# Foundry robots.txt AI Crawler Checker

> Foundry robots.txt AI Crawler Checker is a paid API for AI agents from minty-sprout.onrender.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-11-01).

Checks whether GPTBot, ClaudeBot, CCBot and other AI crawlers are allowed to crawl a site, plus extracts sitemaps, crawl delay, and robots.txt details.

## Facts

- Endpoint: POST https://minty-sprout.onrender.com/v1/robots-txt?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-11-01
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/foundry-robots-txt-ai-crawler-checker-0ed004eb
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_siI4Z2dot7a1HEiyqfwH6

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability foundry-robots-txt-ai-crawler-checker-0ed004eb -d '<json body>'
```

Example prompt: Can you check whether GPTBot, ClaudeBot, and CCBot are allowed to crawl nytimes.com, and also grab any sitemaps listed in its robots.txt?

## When to prefer this

Use this endpoint when you need to quickly check AI crawler permissions for any domain or URL without fetching and parsing the robots.txt yourself. It is the right choice when you want structured, per-bot allow/deny flags for all major LLM crawlers (GPTBot, ClaudeBot, CCBot, etc.) plus sitemaps in a single call. Prefer it over generic HTTP fetch tools when AI crawler compliance, content licensing research, or sitemap discovery is the goal.

## Known failure modes

- Domain does not exist or is unreachable — returns found:false or HTTP error status
- robots.txt not present — returns found:false with status 404
- Malformed URL input — may return validation error
- Timeout on slow or unresponsive servers
- Non-standard robots.txt format may cause incomplete parsing

## How this service works

robots.txt checker: whether GPTBot, ClaudeBot, CCBot and other AI crawlers may crawl a site, plus sitemaps. Input: a domain or URL.

## Output

Returns a JSON object with: the resolved host, whether a robots.txt was found, HTTP status, total line count, a list of sitemap URLs, per-bot allow/deny flags for major AI crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Claude-User, ChatGPT-User, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent), any crawl delay, the path checked, and whether Googlebot is allowed.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "input": {
   "type": "string",
   "maxLength": 8000,
   "description": "A domain or URL"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "host": "nytimes.com",
  "found": true,
  "lines": 364,
  "status": 200,
  "sitemaps": [
   "https://www.nytimes.com/sitemaps/new/news.xml.gz",
   "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz",
   "https://www.nytimes.com/sitemaps/new/collections.xml.gz"
  ],
  "ai_crawlers": {
   "CCBot": false,
   "GPTBot": false,
   "ClaudeBot": false,
   "Bytespider": false,
   "Claude-User": false,
   "ChatGPT-User": false,
   "PerplexityBot": false,
   "Google-Extended": false,
   "Applebot-Extended": false,
   "meta-externalagent": false
  },
  "crawl_delay": null,
  "path_checked": "/",
  "googlebot_allowed": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/foundry-robots-txt-ai-crawler-checker-0ed004eb/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from minty-sprout.onrender.com](https://www.zero.xyz/host/minty-sprout.onrender.com/llms.txt)