# Setu Anantasutra: AI Crawler robots.txt Audit

> Setu Anantasutra: AI Crawler robots.txt Audit is a paid API for AI agents from setu.xyz23110928.workers.dev, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-23).

Fetches and parses a domain's robots.txt to produce a structured matrix showing which AI training and search crawlers are allowed or blocked, with per-bot decisions and summary counts.

## Facts

- Endpoint: GET https://setu.xyz23110928.workers.dev/v1/geo/ai-crawlers?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-23
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/setu-anantasutra-ai-crawler-robots-txt-audit-6830914b
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_K0WI4am0THJn7N3PrAnGs

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability setu-anantasutra-ai-crawler-robots-txt-audit-6830914b
```

Example prompt: Check which AI crawlers and training bots are allowed or blocked on openai.com — I want to see the full breakdown by bot name, operator, and whether they can access the root path.

## When to prefer this

Use this endpoint when you need a structured, machine-readable audit of which AI and search crawlers a domain's robots.txt permits or denies, broken down by bot and operator — especially useful for GEO (generative engine optimization) workflows, content licensing research, or compliance checks. Prefer it over manual robots.txt parsing when you need per-bot categorization (training vs search vs user) and summary statistics in a single call.

## Known failure modes

- Domain not found or DNS failure — robots.txt state will reflect fetch error
- robots.txt returns non-200 status — reported in robotsTxt.httpStatus field
- Domain missing robots.txt — state may be 'missing' with all crawlers treated as allowed
- Malformed domain input — validation error returned
- Rate limit or payment failure via x402 — 402 Payment Required response

## How this service works

Pay-per-call JSON tools for AI agents: Indian identifier validation (GSTIN, PAN, TAN, CIN, LLPIN, IFSC, PIN code, UPI ID, vehicle registration), AI-visibility (GEO) audits, AI crawler robots.txt matrices and llms.txt checks. x402 (USDC on Base) or RapidAPI.

## Output

A JSON object with: the canonical origin URL, a summary object with counts of allowed vs blocked training and search crawlers, an array of per-bot objects (bot name, category, operator, allowedRoot boolean, decidingRule), robots.txt HTTP status and state, and a list of paths checked.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain"
     ],
     "properties": {
      "paths": {
       "type": "string",
       "maxLength": 600,
       "description": "Optional comma-separated extra paths to test, e.g. /blog/,/docs/"
      },
      "domain": {
       "type": "string",
       "maxLength": 253,
       "minLength": 3,
       "description": "Domain or URL, e.g. example.com"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "required": [
      "origin",
      "summary",
      "crawlers"
     ],
     "properties": {
      "origin": {
       "type": "string"
      },
      "summary": {
       "type": "object"
      },
      "crawlers": {
       "type": "array",
       "items": {
        "type": "object",
        "properties": {
         "bot": {
          "type": "string"
         },
         "category": {
          "enum": [
           "training",
           "search",
           "user",
           "other"
          ],
          "type": "string"
         },
         "operator": {
          "type": "string"
         },
         "allowedRoot": {
          "type": "boolean"
         },
         "decidingRule": {
          "type": [
           "string",
           "null"
          ]
         }
        }
       }
      }
     }
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "origin": "https://example.com",
  "summary": {
   "blocked": [
    "GPTBot",
    "CCBot"
   ],
   "searchCrawlersAllowed": {
    "total": 9,
    "allowed": 9
   },
   "trainingCrawlersAllowed": {
    "total": 12,
    "allowed": 0
   }
  },
  "crawlers": [
   {
    "bot": "GPTBot",
    "category": "training",
    "operator": "OpenAI",
    "allowedRoot": false,
    "decidingRule": "Disallow: /"
   }
  ],
  "robotsTxt": {
   "state": "ok",
   "httpStatus": 200
  },
  "pathsChecked": [
   "/",
   "/blog/"
  ]
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/setu-anantasutra-ai-crawler-robots-txt-audit-6830914b/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from setu.xyz23110928.workers.dev](https://www.zero.xyz/host/setu.xyz23110928.workers.dev/llms.txt)