# Minifetch URL Robots.txt Preflight Check

> Minifetch URL Robots.txt Preflight Check is a paid API for AI agents from minifetch.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-15).

Checks whether a given URL is allowed to be fetched according to the target website's robots.txt file, before making a more expensive metadata extraction call.

## Facts

- Endpoint: GET https://minifetch.com/api/v1/x402/preflight/url-check
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-15
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/minifetch-url-robots-txt-preflight-check-d9be526e
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_psDfvUYO3E3ZhC2YrtqkR

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability minifetch-url-robots-txt-preflight-check-d9be526e
```

Example prompt: Before extracting metadata from https://example.com/blog/my-post, quickly check if that URL is allowed to be fetched according to the site's robots.txt so we don't waste money on a blocked page.

## When to prefer this

Use this endpoint as a cheap ($0.001) preflight gate before calling any URL metadata extraction endpoint ($0.01+). Ideal when processing large batches of URLs where some may be blocked by robots.txt, to avoid paying for extraction attempts that will fail or return no data. Prefer this over directly attempting extraction whenever respecting crawl rules or cost efficiency matters.

## Known failure modes

- robots.txt file is unreachable or malformed — success may be false with an explanatory message
- Invalid or malformed URL input — likely returns an error with success: false
- robots.txt does not exist for the domain — typically defaults to allowed: true
- Network timeout reaching the target domain's robots.txt

## How this service works

Cheap robots.txt preflight before you pay for other Minifetch endpoints: returns whether the Minifetch user agent may fetch a target URL (allow/deny) plus its crawl-delay. Blocked or setting a crawl delay? https://minifetch.com/tutorials/unblock-minifetch . Checks the URL as given — does not follow redirects. Supports GET & POST

## Output

Returns a JSON object with: `allowed` (boolean indicating if the URL can be fetched), `reason` (explanation of why it is or isn't allowed), `message` (human-readable summary), `success` (whether the check itself succeeded), and `crawlDelay` (seconds to wait between fetches as specified by robots.txt).

## Example request

```json
{
 "url": "https://example.com/blog/article"
}
```

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "required": [
  "url"
 ],
 "properties": {
  "url": {
   "type": "string",
   "description": "The URL to check against robots.txt"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string"
  },
  "reason": {
   "type": "string"
  },
  "allowed": {
   "type": "boolean"
  },
  "message": {
   "type": "string"
  },
  "success": {
   "type": "boolean"
  },
  "crawlDelay": {
   "type": "number",
   "description": "How often the URL can be fetched, in seconds."
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/minifetch-url-robots-txt-preflight-check-d9be526e/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from minifetch.com](https://www.zero.xyz/host/minifetch.com/llms.txt)