# Vextorium Robots.txt Parser

> Vextorium Robots.txt Parser is a paid API for AI agents from api.vextorium.com, paid per call via x402, $0.003/call, status unknown (last checked 2026-10-16).

Fetches and parses a website's robots.txt, reporting which AI crawlers are blocked or allowed, whether a specific path is crawlable by a given user-agent, and declared sitemaps.

## Facts

- Endpoint: POST https://api.vextorium.com/robots-txt-pago?utm_source=zero.xyz
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-16
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/vextorium-robots-txt-parser-f281873e
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_MBsavFQj6pY80TJyzTGWT

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability vextorium-robots-txt-parser-f281873e -d '<json body>'
```

Example prompt: Before scraping nytimes.com, check its robots.txt and tell me whether GPTBot is allowed, whether the path /section/world is accessible to it, and list any declared sitemaps.

## When to prefer this

Use this endpoint when an AI agent or scraper needs to check crawl permissions before accessing a site, audit how a site treats AI bots specifically, or discover sitemaps for content indexing. It is purpose-built for AI-bot-awareness (covering ~20 named AI crawlers) rather than being a generic SEO robots.txt tool.

## Known failure modes

- Site has no robots.txt — empty or null result returned
- Domain does not resolve or is unreachable — fetch error
- Malformed robots.txt — parsing may produce incomplete group data
- Invalid URL input — API returns validation error
- Rate limiting or network timeout on the target site

## How this service works

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

## Output

A JSON object containing: the site URL, number of rule groups, a per-AI-bot status map (e.g. GPTBot: blocked/allowed) covering ~20 named bots, a query result showing whether a specific path is permitted for a given user-agent and which rule group applied, a list of sitemap URLs, and the raw robots.txt text.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string"
  },
  "ruta": {
   "type": "string",
   "description": "Path to test (default /)"
  },
  "agente": {
   "type": "string",
   "description": "User-agent to test (optional)"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "sitio": "https://www.nytimes.com",
  "grupos": 63,
  "bots_ia": {
   "CCBot": "bloqueado",
   "GPTBot": "bloqueado",
   "YouBot": "bloqueado",
   "Diffbot": "bloqueado",
   "Amazonbot": "permitido",
   "ClaudeBot": "bloqueado",
   "cohere-ai": "bloqueado",
   "Bytespider": "bloqueado",
   "Claude-User": "bloqueado",
   "ChatGPT-User": "bloqueado",
   "anthropic-ai": "bloqueado",
   "DuckAssistBot": "bloqueado",
   "OAI-SearchBot": "bloqueado",
   "PerplexityBot": "bloqueado",
   "MistralAI-User": "permitido",
   "Google-Extended": "bloqueado",
   "Perplexity-User": "bloqueado",
   "Claude-SearchBot": "bloqueado",
   "Applebot-Extended": "bloqueado",
   "meta-externalagent": "bloqueado"
  },
  "consulta": {
   "ruta": "/section/world",
   "agente": "GPTBot",
   "permitido": false,
   "grupo_aplicado": [
    "gptbot"
   ]
  },
  "sitemaps": [
   "https://www.nytimes.com/sitemaps/new/news.xml.gz",
   "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz",
   "https://www.nytimes.com/sitemaps/new/collections.xml.gz",
   "https://www.nytimes.com/sitemaps/new/video.xml.gz",
   "https://www.nytimes.com/sitemaps/new/cooking.xml.gz",
   "https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz",
   "https://www.nytimes.com/sitemaps/new/regions.xml",
   "https://www.nytimes.com/sitemaps/new/best-sellers.xml",
   "https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml",
   "https://www.nytimes.com/sitemaps/new/weather.xml.gz",
   "https://www.nytimes.com/sitemaps/new/espanol.xml.gz",
   "https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz",
   "https://www.nytimes.com/wirecutter/sitemapindex.xml",
   "https://www.nytimes.com/athletic/sitemap-videos-index.xml",
   "https://www.nytimes.com/athletic/sitemap-authors.xml",
   "https://www.nytimes.com/athletic/sitemap-verticals.xml",
   "https://www.nytimes.com/athletic/sitemap-teams.xml",
   "https://www.nytimes.com/athletic/sitemap-cities.xml",
   "https://www.nytimes.com/athletic/sitemap-players.xml",
   "https://www.nytimes.com/athletic/sitemap-tags.xml",
   "https://www.nytimes.com/athletic/sitemap-stats.xml",
   "https://www.nytimes.com/athletic/sitemap-schedule.xml",
   "https://www.nytimes.com/athletic/sitemap-roster.xml",
   "https://www.nytimes.com/athletic/sitemap.xml",
   "https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml"
  ],
  "robots_txt": true,
  "bots_ia_bloqueados": 18
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/vextorium-robots-txt-parser-f281873e/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.vextorium.com](https://www.zero.xyz/host/api.vextorium.com/llms.txt)