# robots.txt + llms.txt Checker

> robots.txt + llms.txt Checker is a paid API for AI agents from api.webbersites.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-10).

Checks a website's robots.txt and llms.txt files, returning crawler access verdicts for major search and AI bots, declared sitemaps, syntax warnings, and llms.txt structure summary.

## Facts

- Endpoint: GET https://api.webbersites.com/api/seo/robots-check?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-10
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/robots-txt-llms-txt-checker-19b1c8ab
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_5A4IawEKlRuSYIXPG7TYB

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability robots-txt-llms-txt-checker-19b1c8ab
```

Example prompt: Check the robots.txt and llms.txt files for openai.com — I want to know if GPTBot, ClaudeBot, Googlebot, and PerplexityBot are allowed or blocked, any declared sitemaps, syntax warnings, and whether they have an llms.txt.

## When to prefer this

Use this endpoint when you need to audit a website's crawler access policies in one call — specifically when you need verdicts for both traditional search bots AND AI training/inference bots simultaneously, and also want to check llms.txt presence. Prefer this over a generic HTTP fetch when you need structured, interpreted bot-by-bot verdicts rather than raw file content.

## Known failure modes

- URL parameter missing — returns 400 error
- Unreachable domain or invalid URL — returns error or empty robots result
- robots.txt returns non-200 status — noted in response
- llms.txt absent — field returns null or absent indicator
- Malformed robots.txt — syntax warnings returned but parsing continues

## How this service works

WebberSites – Branding, Web Design, Photography, Marketing Strategy - Youngstown Ohio

## Output

Returns crawler access verdicts (allow/disallow) for major search bots (Googlebot, Bingbot) and AI bots (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and more), a list of sitemaps declared in robots.txt, any syntax warnings found, and whether llms.txt and llms-full.txt are present along with a structural summary of their contents.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Domain or any page on it"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "properties": {
      "origin": {
       "type": "string"
      },
      "robots": {
       "type": "object"
      },
      "llms_txt": {
       "type": "object"
      }
     }
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "origin": "https://example.com",
  "robots": {
   "bots": {
    "GPTBot": {
     "kind": "ai",
     "root_blocked": true
    },
    "Googlebot": {
     "kind": "search",
     "root_blocked": false
    }
   },
   "found": true,
   "sitemaps_declared": [
    "https://example.com/sitemap.xml"
   ]
  },
  "llms_txt": {
   "found": false
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/robots-txt-llms-txt-checker-19b1c8ab/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.webbersites.com](https://www.zero.xyz/host/api.webbersites.com/llms.txt)