# Vextorium Robots.txt Analyzer

> Vextorium Robots.txt Analyzer is a paid API for AI agents from api.vextorium.com, paid per call via x402, $0.003/call, status unknown (last checked 2026-10-11).

Fetches and parses a website's robots.txt to determine AI bot permissions, allowed/blocked routes, and crawl rules for specific user-agents

## Facts

- Endpoint: POST https://api.vextorium.com/v1/robots-txt?utm_source=zero.xyz
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-11
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/vextorium-robots-txt-analyzer-5a537ae5
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_oS5Y9Kx6yiZcQudV2pkJ2

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability vextorium-robots-txt-analyzer-5a537ae5 -d '<json body>'
```

Example prompt: Check if GPTBot is allowed to crawl https://www.nytimes.com/section/world — I want to know the full robots.txt AI bot permissions including which bots are blocked or allowed.

## When to prefer this

Use this endpoint when you need to programmatically check whether an AI crawler or agent is permitted to access a specific website or route before scraping, indexing, or crawling. It is especially useful for compliance checks before web data collection, building ethical scraping pipelines, or auditing a site's AI bot policy across multiple known AI user-agents simultaneously. Prefer this over raw HTTP fetches of robots.txt when you need structured, parsed output with per-bot status rather than raw text.

## Known failure modes

- Invalid or unreachable URL returns an error or null robots.txt fields
- Site has no robots.txt — robots_txt field returns false with no rules
- User-agent not found in robots.txt — allowed status may default to true (permissive)
- Malformed robots.txt on target site may cause partial parse results
- Network timeout fetching remote robots.txt

## How this service works

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

## Output

Returns a structured JSON with: the parsed robots.txt existence flag, total number of rule groups, a map of well-known AI bots (GPTBot, ClaudeBot, CCBot, PerplexityBot, etc.) with their allowed/blocked status, a list of sitemaps, and — if a specific agent and route are queried — whether that agent is allowed on that path and which rule group applies.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method",
    "bodyType",
    "body"
   ],
   "properties": {
    "body": {
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string"
      },
      "agent": {
       "type": "string",
       "description": "User-agent to test (optional)"
      },
      "route": {
       "type": "string",
       "description": "Path to test (default /)"
      }
     }
    },
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "POST"
     ],
     "type": "string"
    },
    "bodyType": {
     "enum": [
      "json",
      "form-data",
      "text"
     ],
     "type": "string"
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": [
      "object",
      "null"
     ],
     "properties": {
      "site": {
       "type": [
        "string",
        "null"
       ]
      },
      "query": {
       "type": [
        "object",
        "null"
       ],
       "properties": {
        "agent": {
         "type": [
          "string",
          "null"
         ]
        },
        "route": {
         "type": [
          "string",
          "null"
         ]
        },
        "allowed": {
         "type": [
          "boolean",
          "null"
         ]
        },
        "applied_group": {
         "type": [
          "array",
          "null"
         ],
         "items": {
          "type": [
           "null",
           "string"
          ]
         }
        }
       }
      },
      "groups": {
       "type": [
        "number",
        "null"
       ]
      },
      "ai_bots": {
       "type": [
        "object",
        "null"
       ],
       "properties": {
        "CCBot": {
         "type": [
          "string",
          "null"
         ]
        },
        "GPTBot": {
         "type": [
          "string",
          "null"
         ]
        },
        "YouBot": {
         "type": [
          "string",
          "null"
         ]
        },
        "Diffbot": {
         "type": [
          "string",
          "null"
         ]
        },
        "Amazonbot": {
         "type": [
          "string",
          "null"
         ]
      
… (truncated)
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "site": "https://www.nytimes.com",
  "query": {
   "agent": "GPTBot",
   "route": "/section/world",
   "allowed": false,
   "applied_group": [
    "gptbot"
   ]
  },
  "groups": 63,
  "ai_bots": {
   "CCBot": "bloqueado",
   "GPTBot": "bloqueado",
   "YouBot": "bloqueado",
   "Diffbot": "bloqueado",
   "Amazonbot": "permitido",
   "ClaudeBot": "bloqueado",
   "cohere-ai": "bloqueado",
   "Bytespider": "bloqueado",
   "Claude-User": "bloqueado",
   "ChatGPT-User": "bloqueado",
   "anthropic-ai": "bloqueado",
   "DuckAssistBot": "bloqueado",
   "OAI-SearchBot": "bloqueado",
   "PerplexityBot": "bloqueado",
   "MistralAI-User": "permitido",
   "Google-Extended": "bloqueado",
   "Perplexity-User": "bloqueado",
   "Claude-SearchBot": "bloqueado",
   "Applebot-Extended": "bloqueado",
   "meta-externalagent": "bloqueado"
  },
  "sitemaps": [
   "https://www.nytimes.com/sitemaps/new/news.xml.gz",
   "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz"
  ],
  "robots_txt": true,
  "blocked_ai_bots": 18
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/vextorium-robots-txt-analyzer-5a537ae5/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.vextorium.com](https://www.zero.xyz/host/api.vextorium.com/llms.txt)