# Lordevo robots.txt Can-I-Fetch

> Lordevo robots.txt Can-I-Fetch is a paid API for AI agents from shop.lordevo.com, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-31).

Checks whether a given URL is allowed to be fetched by generic web agents and 16 named AI crawlers according to its robots.txt rules.

## Facts

- Endpoint: GET https://shop.lordevo.com/api/quick/can-fetch?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-31
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/lordevo-robots-txt-can-i-fetch-33f1b081
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_KwCPsvdTVk35XzurPq943

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability lordevo-robots-txt-can-i-fetch-33f1b081
```

Example prompt: Before you scrape https://techcrunch.com/2024/01/ai-trends, check if that URL is allowed for AI crawlers — specifically whether Claude-User and GPTBot are permitted by the site's robots.txt.

## When to prefer this

Choose this endpoint when your agent or scraper needs to check robots.txt compliance before fetching a URL, especially when you need per-named-AI-crawler verdicts (ChatGPT-User, Claude-User, GPTBot, PerplexityBot, etc.) rather than just a generic allow/disallow. It is the right choice for pre-fetch gating in agentic pipelines, AI training data sourcing workflows, or any scenario requiring RFC 9309-compliant robots.txt interpretation plus noai/content-signal awareness. Prefer it over raw robots.txt parsing when you need structured, normalized output with named AI agent breakdowns in a single API call.

## Known failure modes

- URL is unreachable or returns 5xx — treated as disallowed per RFC 9309
- Missing or malformed URL query parameter returns validation error
- robots.txt itself returns 5xx — interpreted as disallow for all agents
- Private or non-HTTP URLs rejected
- Rate limiting or timeout on the target domain may delay response

## How this service works

SEO audits, website security checks, SEO content plans, articles and site fix plans as API endpoints for AI agents. Pay per request in USDC on Base with…

## Output

Returns a structured JSON object with: a verdict enum (allowed, disallowed, allowed_but_no_ai_use, allowed_with_crawl_delay), a boolean allowed flag, per-named-agent boolean permissions (for GPTBot, Claude-User, ChatGPT-User, PerplexityBot, CCBot, ClaudeBot, and others), the robots_txt status (present/missing/unreachable), the HTTP status of the robots.txt response, crawl delay in seconds if specified, a noai boolean, a content_signal object (search/ai-input/ai-train), discovered sitemap URLs, and a human-readable summary string.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Public http(s) URL or bare domain, e.g. https://example.com"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "required": [
      "tool",
      "url",
      "verdict",
      "allowed",
      "agents",
      "robots_txt"
     ],
     "properties": {
      "url": {
       "type": "string"
      },
      "noai": {
       "type": "boolean"
      },
      "page": {
       "type": [
        "object",
        "null"
       ]
      },
      "path": {
       "type": "string"
      },
      "tool": {
       "type": "string"
      },
      "agents": {
       "type": "object",
       "additionalProperties": {
        "type": "boolean"
       }
      },
      "origin": {
       "type": "string"
      },
      "allowed": {
       "type": "boolean"
      },
      "summary": {
       "type": "string"
      },
      "verdict": {
       "enum": [
        "allowed",
        "disallowed",
        "allowed_but_no_ai_use",
        "allowed_with_crawl_delay"
       ],
       "type": "string"
      },
      "sitemaps": {
       "type": "array",
       "items": {
        "type": "string"
       }
      },
      "robots_txt": {
       "enum": [
        "present",
        "missing",
        "unreachable"
       ],
       "type": "string"
      },
      "content_signal": {
       "type": [
        "object",
        "null"
       ]
      },
      "robots_http_status": {
       "type": [
        "integer",
        "null"
       ]
      },
      "crawl_delay_seconds": {
       "type": [
        "number",
        "null"
       ]
      }
     }
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com/blog/post",
  "noai": true,
  "page": {
   "final_url": "https://example.com/blog/post",
   "http_status": 200,
   "content_type": "text/html; charset=utf-8",
   "x_robots_tag": []
  },
  "path": "/blog/post",
  "tool": "can-fetch",
  "agents": {
   "*": true,
   "CCBot": false,
   "GPTBot": false,
   "ClaudeBot": true,
   "Claude-User": true,
   "ChatGPT-User": true,
   "PerplexityBot": true
  },
  "origin": "https://example.com",
  "allowed": true,
  "summary": "Allowed: example.com/blog/post for generic crawlers/agents. Blocked for: GPTBot, CCBot. The site signals no AI use (noai / Content-Signal).",
  "verdict": "allowed_but_no_ai_use",
  "sitemaps": [
   "https://example.com/sitemap.xml"
  ],
  "robots_txt": "present",
  "robots_url": "https://example.com/robots.txt",
  "content_signal": {
   "search": true,
   "ai-input": false,
   "ai-train": false
  },
  "robots_http_status": 200,
  "crawl_delay_seconds": null
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/lordevo-robots-txt-can-i-fetch-33f1b081/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from shop.lordevo.com](https://www.zero.xyz/host/shop.lordevo.com/llms.txt)