# TrustSource Robots.txt Intelligence API

> TrustSource Robots.txt Intelligence API is a paid API for AI agents from api.trustsource.cc, paid per call via x402, $0.002/call, status unknown (last checked 2026-09-14).

Parses a domain's robots.txt file and returns AI bot crawl policies, access tiers, per-bot allow/disallow rules, and sitemap URLs

## Facts

- Endpoint: GET https://api.trustsource.cc/robots
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/api-trustsource-cc-5efec1b6
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_LVJ6ZbMaRShuwFlfK_Dik

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability api-trustsource-cc-5efec1b6
```

Example prompt: Can you check whether openai.com allows GPTBot and ClaudeBot to crawl it — I want to know the full robots.txt tier and any per-bot allow or disallow rules before I start crawling.

## When to prefer this

Use this endpoint when an AI agent or crawler needs to programmatically determine whether it is permitted to crawl a given domain, especially for checking AI-specific bot policies (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Prefer over manual robots.txt parsing when you need structured tier classification and per-bot analysis in a single call.

## Known failure modes

- Domain does not exist or is unreachable — may return NO_ROBOTS_TXT tier or error
- robots.txt file is malformed or non-standard — partial parsing results
- Domain missing 'domain' query parameter — 400 bad request
- Upstream fetch timeout if target server is slow
- Payment not attached — 402 Payment Required response

## How this service works

robots.txt and AI crawler policy check. Tells an agent whether a website permits crawling and whether it blocks AI bots such as GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, before scraping, RAG ingestion, training data collection or archiving. Parses robots.txt and returns a crawl-policy tier with per-bot allow and disallow rules and sitemap URLs.

## Output

Returns a tier label (OPEN, SELECTIVE, BLOCKED_AI, BLOCKED_ALL, or NO_ROBOTS_TXT), per-bot allow/disallow analysis for known AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.), and a list of sitemap URLs found in robots.txt

## Example request

```json
{
 "input": {
  "type": "http",
  "method": "GET",
  "queryParams": {
   "domain": "example.com"
  }
 }
}
```

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain"
     ],
     "properties": {
      "domain": {
       "type": "string",
       "description": "Domain whose robots.txt to fetch and parse (e.g. example.com); https is tried first, then http."
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/api-trustsource-cc-5efec1b6/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.trustsource.cc](https://www.zero.xyz/host/api.trustsource.cc/llms.txt)