# Robots.txt & LLMs.txt Checker

> Robots.txt & LLMs.txt Checker is a paid API for AI agents from tools.adamhaleworks.workers.dev, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-13).

Checks robots.txt crawl permissions and llms.txt presence for up to 5 URLs across configurable AI and web crawler user-agents.

## Facts

- Endpoint: GET https://tools.adamhaleworks.workers.dev/x402/v1/robots-check?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-llms-txt-checker-9206682b
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_0p51F3VH-Fjg3cQvBBkYb

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-llms-txt-checker-9206682b
```

Example prompt: Can you check whether github.com and openai.com allow GPTBot, ClaudeBot, and PerplexityBot to crawl them, and also tell me if either site has an llms.txt file?

## When to prefer this

Choose this endpoint when you need to programmatically verify how a website's robots.txt treats AI or web crawlers, especially when checking for llms.txt presence simultaneously. It supports batch checking of up to 5 URLs and a configurable set of up to 15 user-agent tokens, making it ideal for auditing multiple domains at once. Prefer it over manual robots.txt parsing or generic HTTP fetchers when you need structured, per-agent crawl permission data in a single call.

## Known failure modes

- Private/internal hostnames or IP addresses are rejected
- More than 5 URLs in a single request causes an error
- Invalid URL format returns a failed result for that entry
- Site unreachable or DNS failure marks the URL as failed
- Ports other than 80 or 443 are rejected

## How this service works

Guides and free-to-run Apify actors for bulk tech stack lookup, sitemap URL export to CSV, TED and UK Find a Tender search, and RSS-to-JSON feed monitoring.

## Output

Returns a JSON object with a count of results, success/failure counts, and per-URL details including: crawl permission (allowed/denied) and crawl delay per user-agent, robots.txt fetch status and group count, any sitemaps listed in robots.txt, and llms.txt presence with its URL, HTTP status, and first line of content.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "urls"
     ],
     "properties": {
      "urls": {
       "type": "string",
       "description": "Up to 5 URLs, comma-separated. A bare domain means its https:// homepage. Public hosts only (no IPs, no internal names), ports 80/443."
      },
      "agents": {
       "type": "string",
       "description": "User-agent product tokens to check, comma-separated, up to 15 (default: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Applebot-Extended, Bytespider, *)."
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "count": 1,
  "failed": 0,
  "results": [
   {
    "ok": true,
    "input": "https://github.com/about",
    "result": {
     "url": "https://github.com/about",
     "agents": {
      "*": {
       "allowed": true,
       "crawlDelay": null
      },
      "GPTBot": {
       "allowed": true,
       "crawlDelay": 1
      },
      "ClaudeBot": {
       "allowed": true,
       "crawlDelay": 1
      }
     },
     "llmsTxt": {
      "url": "https://github.com/llms.txt",
      "status": 200,
      "present": true,
      "firstLine": "# GitHub"
     },
     "robotsTxt": {
      "url": "https://github.com/robots.txt",
      "state": "rules",
      "groups": 8,
      "status": 200,
      "sitemaps": []
     }
    }
   }
  ],
  "succeeded": 1
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-llms-txt-checker-9206682b/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from tools.adamhaleworks.workers.dev](https://www.zero.xyz/host/tools.adamhaleworks.workers.dev/llms.txt)