# AI Crawl Preflight

> AI Crawl Preflight is a paid API for AI agents from x402.botsmith.dev, paid per call via x402, $0.003/call, status unknown (last checked 2026-09-14).

Checks a URL's robots.txt, llms.txt, HTTP 402 pay-per-crawl gating, and markdown variant availability before an agent wastes a fetch call, returning a plain-language recommendation.

## Facts

- Endpoint: GET https://x402.botsmith.dev/preflight
- Price: $0.003/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/ai-crawl-preflight-eda0d69a
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_viuHazkfLpXdurvPY2hsN

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability ai-crawl-preflight-eda0d69a
```

Example prompt: Before you fetch https://techcrunch.com/2025/01/15/openai-news/, check whether that URL is reachable, whether AI crawlers are allowed, if there's a pay-per-crawl gate or a markdown variant, and tell me whether it's safe to proceed.

## When to prefer this

Use this before any agent web fetch to avoid wasting credits on blocked, paywalled, or AI-restricted URLs. Ideal when building autonomous crawling pipelines that need to respect robots.txt, detect x402 pay-per-crawl gates, or find markdown-optimized content variants. Pairs with any web reader endpoint to form a preflight-then-fetch pattern.

## Known failure modes

- URL is malformed or exceeds 2048 characters — validation error returned
- Target URL is completely unreachable (network timeout, DNS failure) — reachable: false with null status
- robots.txt fetch fails — present: false, genericAllowed: null
- Service itself is unavailable — HTTP error from preflight endpoint
- Payment not attached or insufficient — HTTP 402 from the preflight endpoint itself

## How this service works

AI Crawl Preflight ($0.003/call): Preflight a URL before spending a fetch: reads the site's robots.txt (generic and per-AI-crawler verdict, Cloudflare Content-Signal), checks for llms.txt, detects pay-per-crawl (HTTP 402) gating, and whether a markdown variant is served, then returns a plain-language recommendation. Keyless. Pairs with the web reader.

## Output

A JSON object containing: whether the URL is reachable, the HTTP status code, robots.txt presence and whether generic agents and named AI crawlers (e.g. GPTBot, ClaudeBot) are allowed, a Cloudflare Content-Signal directive if present, a boolean for llms.txt presence, a boolean for HTTP 402 pay-per-crawl gating, a boolean for markdown variant availability, and a plain-language recommendation string advising the agent on next steps.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri",
       "maxLength": 2048,
       "description": "The URL an agent is considering fetching"
      }
     },
     "additionalProperties": false
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "required": [
      "url",
      "reachable",
      "status",
      "robots",
      "llmsTxt",
      "payPerCrawl",
      "markdownVariant",
      "recommendation"
     ],
     "properties": {
      "url": {
       "type": "string"
      },
      "robots": {
       "type": "object",
       "required": [
        "present",
        "genericAllowed",
        "aiBotsBlocked",
        "contentSignal"
       ],
       "properties": {
        "present": {
         "type": "boolean"
        },
        "aiBotsBlocked": {
         "type": "array",
         "items": {
          "type": "string"
         },
         "description": "Named AI crawlers disallowed for this path"
        },
        "contentSignal": {
         "anyOf": [
          {
           "type": "string"
          },
          {
           "type": "null"
          }
         ],
         "description": "Cloudflare Content-Signal directive, if present"
        },
        "genericAllowed": {
         "anyOf": [
          {
           "type": "boolean"
          },
          {
           "type": "null"
          }
         ],
         "description": "Is the path allowed for a generic (*) agent"
        }
       },
       "additionalProperties": false
      },
      "status": {
       "anyOf": [
        {
         "type": "integer",
         "maximum": 9007199254740991,
         "minimum": -9007199254740991
        },
        {
         "type": "null"
        }
       ]
      },
      "llmsTxt": {
       "type": "boolean",
       "description": "Site publishes an llms.txt"
      },
      "reachable": {
       "type
… (truncated)
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/ai-crawl-preflight-eda0d69a/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from x402.botsmith.dev](https://www.zero.xyz/host/x402.botsmith.dev/llms.txt)