# Robots.txt Fetcher with AI Crawler Detection

> Robots.txt Fetcher with AI Crawler Detection is a paid API for AI agents from remarkable-creation-production-8797.up.railway.app, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-15).

Fetches and returns the robots.txt file for a given public website, and reports which common AI crawlers (e.g. GPTBot, ClaudeBot) are mentioned in it.

## Facts

- Endpoint: GET https://remarkable-creation-production-8797.up.railway.app/v1/robots
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-fetcher-with-ai-crawler-detection-ab3f3da8
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_PzrdZt06oKGQImX-Y2DpO

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-fetcher-with-ai-crawler-detection-ab3f3da8
```

Example prompt: Can you fetch the robots.txt for openai.com and tell me which AI crawlers are mentioned and whether any of them are blocked?

## When to prefer this

Use this endpoint when you need to programmatically check a site's robots.txt content and specifically want to see how AI crawlers are treated — it saves you from manually fetching and parsing robots.txt yourself. Prefer it over a generic HTTP fetch when you want structured AI-crawler mention extraction alongside the raw content. It pairs well with the sibling URL inspection and header-checking endpoints for a complete site audit.

## Known failure modes

- Target URL is not publicly accessible or returns a non-200 status — robots.txt may not exist
- Malformed or missing 'target' query parameter returns a validation error
- Site returns a robots.txt that is very large or non-standard, which may produce incomplete parsing
- Private or localhost URLs may be rejected or fail to resolve
- Some sites serve robots.txt with incorrect content types, which may affect parsing accuracy

## How this service works

Fetch robots.txt for a public site and report common AI crawler mentions plus the robots content.

## Output

Returns the raw robots.txt content for the given domain, along with a structured report of which common AI crawlers (such as GPTBot, ClaudeBot, PerplexityBot, etc.) are explicitly mentioned, whether they are allowed or disallowed, and any relevant path-level directives.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET",
      "HEAD",
      "DELETE"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "target"
     ],
     "properties": {
      "target": {
       "type": "string",
       "description": "Public HTTP(S) URL or hostname, e.g. example.com"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-fetcher-with-ai-crawler-detection-ab3f3da8/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from remarkable-creation-production-8797.up.railway.app](https://www.zero.xyz/host/remarkable-creation-production-8797.up.railway.app/llms.txt)