# FreeTier Agent API — Robots.txt Crawl Permission Checker

> FreeTier Agent API — Robots.txt Crawl Permission Checker is a paid API for AI agents from freetier-agent-api.pages.dev, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-18).

Checks whether a given URL is allowed or disallowed for crawling according to the site's robots.txt rules, returning crawl policy, matched rule, sitemap locations, and crawl delay.

## Facts

- Endpoint: GET https://freetier-agent-api.pages.dev/api/robots-check?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-18
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/freetier-agent-api-robots-txt-crawl-permission-checker-5f1b597d
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_LgPtyJIGjol-2oiDMhx95

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability freetier-agent-api-robots-txt-crawl-permission-checker-5f1b597d
```

Example prompt: Can you check whether https://example.com/some/page is allowed to be crawled according to its robots.txt rules, using the default wildcard user-agent?

## When to prefer this

Use this endpoint when an AI agent needs to verify crawl permission before accessing or scraping a URL, especially for compliance with web crawling etiquette. Ideal for pre-flight checks in scraping workflows, SEO audits, or any agent that fetches web content and needs to respect robots.txt policies. Prefer this over manually parsing robots.txt when you need a fast, structured JSON answer with matched rule and sitemap discovery.

## Known failure modes

- Invalid or non-absolute URL returns an error
- Site has no robots.txt — may return allowed=true with no rules
- Network timeout fetching the remote robots.txt
- Malformed robots.txt on target site may produce unexpected results
- URL scheme other than http/https not supported

## How this service works

Pay-per-call data API for AI agents: AI model prices and cost calculator, 4,259 MCP servers, email domain checks, text-to-JSON extraction, country/holiday/currency lookup, page and link checks, page-to-markdown, 2,230 verified APIs. x402, USDC on Base, no signup. Sources cited in every response.

## Output

A JSON object containing: the checked URL, a boolean 'allowed' indicating crawl permission, a list of sitemaps found in robots.txt, the matched user-agent rule, the matched rule string, and any crawl delay specified for that user-agent.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "ua": {
       "type": "string",
       "description": "user-agent token, default *"
      },
      "url": {
       "type": "string",
       "description": "absolute http(s) URL"
      }
     }
    }
   }
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com/some/page",
  "allowed": true,
  "sitemaps": [],
  "user_agent": "*",
  "crawl_delay": null,
  "matched_rule": null
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/freetier-agent-api-robots-txt-crawl-permission-checker-5f1b597d/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from freetier-agent-api.pages.dev](https://www.zero.xyz/host/freetier-agent-api.pages.dev/llms.txt)