# MilliAPI AI Robots.txt Checker

> MilliAPI AI Robots.txt Checker is a paid API for AI agents from milliapi.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-14).

Checks a website's robots.txt to determine whether major AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended) are allowed or blocked on its homepage.

## Facts

- Endpoint: GET https://milliapi.com/api/ai-robots-check
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/milliapi-ai-robots-txt-checker-a34380d2
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_JGeJ0HTmWoaga5Yyz_iv8

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability milliapi-ai-robots-txt-checker-a34380d2
```

Example prompt: Can you check whether the major AI crawlers like GPTBot and ClaudeBot are allowed to crawl https://example.com based on its robots.txt?

## When to prefer this

Use this endpoint when you need to quickly determine whether a specific website's robots.txt permits or blocks major AI crawlers without manually fetching and parsing the file. It is ideal for competitive analysis, SEO audits, compliance checks, and agent workflows that need to assess AI accessibility of external sites. Prefer this over writing custom robots.txt parsers or using general web scraping when you need structured, per-bot permission data across the most relevant AI crawlers.

## Known failure modes

- Target URL is not a valid public HTTPS URL — returns validation error
- robots.txt returns 404 or is absent — reflected in response with present:false
- Target site is unreachable or times out — may return network error
- Malformed robots.txt on the target site — parsing may yield incomplete results
- Rate limiting or access restrictions on the target site may affect result accuracy

## How this service works

Tiny machine-readable APIs that AI agents can discover, pay for with x402 USDC on Base, and use instantly. Services start at $0.001 per call.

## Output

Returns a JSON object with the origin URL, the HTTP status and presence of the robots.txt file, and per-bot access results for each major AI crawler (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended), including whether each is allowed and the reason derived from the robots.txt rules.

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "origin": "https://example.com",
  "robotsTxt": {
   "status": 404,
   "present": false
  },
  "aiCrawlerHomepageAccess": {
   "GPTBot": {
    "reason": "No matching robots.txt group",
    "allowed": true
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/milliapi-ai-robots-txt-checker-a34380d2/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from milliapi.com](https://www.zero.xyz/host/milliapi.com/llms.txt)