# Crawl Preflight Check

> Crawl Preflight Check is a paid API for AI agents from crawl-preflight.postnov01.workers.dev, paid per call via x402, $0.02/call, status unknown (last checked 2026-09-14).

Checks a domain before crawling to determine if AI bots are blocked by robots.txt, if content is paywalled, how many pages are crawlable, and whether to proceed or skip.

## Facts

- Endpoint: GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
- Price: $0.02/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/crawl-preflight-check-e7520d11
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_YiEDopu8MGiEztSN2V3US

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability crawl-preflight-check-e7520d11
```

Example prompt: Before you start crawling techcrunch.com, run a preflight check to see if AI bots are allowed, whether there's a paywall, and how many pages are actually crawlable.

## When to prefer this

Use this endpoint when your agent is about to crawl an unknown or untrusted domain and needs to avoid wasting requests on blocked, paywalled, or content-sparse sites. It is especially valuable before bulk crawls, when building web indexing pipelines, or when you need per-bot robots.txt verdicts for specific AI crawlers rather than a generic accessibility check.

## Known failure modes

- Domain does not exist or is unreachable — may return error or empty result
- robots.txt absent — verdicts default to allowed
- Sitemap missing or malformed — crawlable page count may be zero or unreliable
- Rate limiting on the target domain during preflight — partial data returned
- Invalid domain format — request rejected

## How this service works

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

## Output

Returns per-bot robots.txt verdicts for 16 AI crawlers (e.g. GPTBot, CCBot), a crawlable page count derived from sitemaps, detected CDN provider, paid-access signals (x402, TollBit), and a clear fetch or skip recommendation.

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/crawl-preflight-check-e7520d11/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from crawl-preflight.postnov01.workers.dev](https://www.zero.xyz/host/crawl-preflight.postnov01.workers.dev/llms.txt)