# CrawlWitness AI Usage Permission Check

> CrawlWitness AI Usage Permission Check is a paid API for AI agents from api.crawlwitness.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-11-01).

Returns a yes/no verdict on whether a specific AI bot may crawl, train on, or RAG-ingest a given domain, by checking robots.txt, IETF AIPREF, TDMRep, noai, and RSL signals, with a signed receipt.

## Facts

- Endpoint: GET https://api.crawlwitness.com/v1/can-use?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-11-01
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/crawlwitness-ai-usage-permission-check-4968f969
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_diLnsypQ_ch6JVWCEFAh8

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability crawlwitness-ai-usage-permission-check-4968f969
```

Example prompt: Before we ingest articles from wsj.com into our RAG pipeline using ClaudeBot, check if the site actually permits that — I need a signed receipt confirming the rag use is allowed.

## When to prefer this

Choose this endpoint when you need a fast, authoritative, multi-signal permission check before crawling, training on, or ingesting a domain into a RAG pipeline, and you need a signed receipt for compliance auditing. It aggregates robots.txt, IETF AIPREF, TDMRep, noai, and RSL signals in a single call, saving you from implementing each standard yourself. Prefer it over rolling your own robots.txt parser when AI-specific policy signals (AIPREF, noai) matter and when legal/compliance auditability requires a signed artifact.

## Known failure modes

- 400 error for invalid or malformed domain (not charged)
- 400 error for unknown or unsupported bot token (not charged)
- Domain has no explicit AI policy signals — verdict may default to permissive or restrictive depending on interpretation
- Network or DNS resolution failure for the target domain
- robots.txt fetch timeout causing delayed or failed verdict

## How this service works

What a website allows for AI (robots.txt per AI bot, RSL, IETF AIPREF, TDMRep, noai, llms.txt, C2PA), with signed EAS receipts anchored on Base and an attested history of every signal. Paid calls use x402 (USDC on Base or Solana): no account, no API key.

## Output

A yes/no permission verdict indicating whether the specified AI bot may perform the requested use (train, rag, or search) on the given domain, along with the policy sources consulted (robots.txt, IETF AIPREF, TDMRep, noai, RSL) and a cryptographically signed receipt that can be stored as a compliance record.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain",
      "bot"
     ],
     "properties": {
      "bot": {
       "enum": [
        "GPTBot",
        "ClaudeBot",
        "anthropic-ai",
        "Google-Extended",
        "GoogleOther",
        "CloudVertexBot",
        "Applebot-Extended",
        "meta-externalagent",
        "Amazonbot",
        "amazon-kendra-",
        "Bytespider",
        "CCBot",
        "cohere-training-data-crawler",
        "AI2Bot",
        "Diffbot",
        "Timpibot",
        "ImagesiftBot",
        "omgili",
        "PetalBot",
        "KimiBot",
        "ICC-Crawler",
        "Cotoyogi",
        "Reflectionbot",
        "CloudflareBrowserRenderingCrawler",
        "LinerBot",
        "FishBot",
        "Novellum",
        "bigsur.ai",
        "NavuBot",
        "QualifiedBot",
        "atlassian-bot",
        "make.com",
        "magpie-crawler",
        "Awario",
        "w4mwnpbXf3MFAbxOkJRw",
        "SemrushBot-OCOB",
        "SemrushBot-SWA",
        "BorderxBot",
        "SelectikaScraper",
        "CitibotSiteCrawler",
        "AdpResearchBot",
        "WARDBot",
        "ygs-scraper-bot",
        "OAI-SearchBot",
        "Claude-SearchBot",
        "PerplexityBot",
        "Applebot",
        "Amzn-SearchBot",
        "Bravebot",
        "Cloudflare-AI-Search",
        "Cloudflare-AI-Search-External",
        "DuckAssistBot",
        "YouBot",
        "KernelSearchBot",
        "ShapBot",
        "alphalens-bot",
        "Anomura",
        "Element451Bot",
        "ChatGPT-User",
        "Claude-User",
        "Perplexity-User",
        "meta-externalfetcher",
        "meta-externaltest",
        "muse-",
        "MistralAI-User",
        "cohere-ai",
        "Google-Agent",
        "GrokAgent",
        "Devin",
        "ApifyWebsiteContentCrawler",
        "Instapaper",
        "Retool",
        "QATechBot",
        "TwinAgent",
        "CledaraBot",
        "HarkBot",
        "HIFIBot",
        "ChathiveCrawler",
        "EasyScan",
        "Nava",
        "ZipchatBot"
       ],
       "type": "string",
       "descript
… (truncated)
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "bot": "GPTBot",
  "use": "train",
  "pago": {
   "recibo": {
    "firma": "0x…"
   },
   "transaccion": "0x…"
  },
  "domain": "nytimes.com",
  "method": "can-use/1",
  "allowed": false,
  "reasons": [
   "robots.txt disallows GPTBot at the site root"
  ],
  "bot_role": "training",
  "decision": "not_allowed",
  "operator": "OpenAI"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/crawlwitness-ai-usage-permission-check-4968f969/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.crawlwitness.com](https://www.zero.xyz/host/api.crawlwitness.com/llms.txt)