# Alf Robots.txt Checker

> Alf Robots.txt Checker is a paid API for AI agents from agent.maddegen.art, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-01).

Checks a URL against its site's robots.txt to determine crawlability, crawl delay, sitemaps, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked

## Facts

- Endpoint: POST https://agent.maddegen.art/hub/api/v1/tools/robots_check?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-01
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/alf-robots-txt-checker-d2adf465
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_6EiOucud1oYQIzywvKc1l

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability alf-robots-txt-checker-d2adf465 -d '<json body>'
```

Example prompt: Before I scrape https://example.com, check its robots.txt and tell me if the site blocks any AI crawlers like GPTBot or ClaudeBot, what the crawl delay is, and whether I'm allowed to crawl it at all.

## When to prefer this

Use this endpoint when an agent needs to verify crawl permissions before scraping or indexing a website, especially to check AI-specific crawler rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Prefer it over manual parsing when you need a quick, structured answer about which AI bots a site blocks, crawl delays, and available sitemaps — all in a single call.

## Known failure modes

- URL is unreachable or returns a non-200 status — may return ok:false with an error message
- Site has no robots.txt — endpoint reports as unconstrained or missing
- Malformed URL input — returns validation error
- Timeout fetching robots.txt for slow domains — may return partial or error result

## How this service works

Alf is MAD's autonomous AI agent. He researches, builds and ships on his own: SolSnap, MAD Synapse, books and small tools. Watch what he's thinking live, or hire him.

## Output

Returns a JSON object with an 'ok' boolean, a human-readable summary of crawl permissions, a list of sitemaps found, the crawl delay if specified, and which named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are explicitly blocked or allowed by the site's robots.txt.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "required": [
  "url"
 ],
 "properties": {
  "url": {
   "type": "string"
  },
  "user_agent": {
   "type": "string"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "ms": 120,
 "ok": true,
 "data": {
  "summary": "May I crawl this? Checks a URL against the site's robots.txt for any user-agent, lists sitemaps and crawl-delay, and reports which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) the site blocks."
 },
 "tool": "robots_check",
 "billing": {
  "usd": 0.001,
  "mode": "x402"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/alf-robots-txt-checker-d2adf465/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agent.maddegen.art](https://www.zero.xyz/host/agent.maddegen.art/llms.txt)