# SiteCheck Robots.txt Checker

> SiteCheck Robots.txt Checker is a paid API for AI agents from api.sitecheck-api.workers.dev, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-20).

Fetches and parses a site's robots.txt file to determine which crawlers are allowed or blocked from accessing a given URL path

## Facts

- Endpoint: GET https://api.sitecheck-api.workers.dev/api/robots?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-20
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sitecheck-robots-txt-checker-740aed61
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Z-KbbJ1ndZiWyaHa-t38A

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sitecheck-robots-txt-checker-740aed61
```

Example prompt: Can you check whether Googlebot is allowed to crawl https://www.example.com/products according to that site's robots.txt?

## When to prefer this

Use this endpoint when you need to programmatically verify crawl permissions for a specific URL and crawler combination, especially during SEO audits, before building web scrapers, or when checking competitor crawl policies. It parses robots.txt with RFC 9309-compliant longest-match rules, making it more accurate than naive text parsing.

## Known failure modes

- robots.txt not found (404) — site may not have one, returns 200 with empty rules
- URL is malformed — validation error returned
- Site is unreachable or times out — network error
- Specified user agent doesn't match any group — falls back to wildcard rules

## How this service works

Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged.

## Output

Returns the parsed robots.txt evaluation for the given URL, including: whether each crawler (default: common crawlers like Googlebot, Bingbot, Yandex, etc.) is allowed or disallowed, the specific matching rule and line number, the user-agent group it falls under, any crawl delay, discovered sitemap URLs, and a timestamp of when the check was performed.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "A page URL on the site whose robots.txt is checked"
      },
      "userAgent": {
       "type": "string",
       "description": "One crawler name to check, e.g. Googlebot (optional; default lists the common crawlers)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://www.google.com/search?q=x",
  "note": null,
  "path": "/search?q=x",
  "origin": "https://www.google.com",
  "source": "https://www.google.com/robots.txt, RFC 9309 matching (longest rule wins, allow wins a tie)",
  "status": 200,
  "results": [
   {
    "rule": {
     "line": 3,
     "type": "disallow",
     "pattern": "/search"
    },
    "group": {
     "lines": [
      "User-agent: *",
      "User-agent: Yandex",
      "Disallow: /search"
     ],
     "userAgent": "*"
    },
    "allowed": false,
    "userAgent": "*",
    "crawlDelay": null
   },
   {
    "rule": {
     "line": 3,
     "type": "disallow",
     "pattern": "/search"
    },
    "group": {
     "lines": [
      "User-agent: *",
      "User-agent: Yandex",
      "Disallow: /search"
     ],
     "userAgent": "*"
    },
    "allowed": false,
    "userAgent": "Googlebot",
    "crawlDelay": null
   },
   {
    "rule": {
     "line": 3,
     "type": "disallow",
     "pattern": "/search"
    },
    "group": {
     "lines": [
      "User-agent: *",
      "User-agent: Yandex",
      "Disallow: /search"
     ],
     "userAgent": "*"
    },
    "allowed": false,
    "userAgent": "Bingbot",
    "crawlDelay": null
   }
  ],
  "sitemaps": [
   "https://www.google.com/sitemap.xml"
  ],
  "checkedAt": "2026-10-15T14:07:52.191Z"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sitecheck-robots-txt-checker-740aed61/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.sitecheck-api.workers.dev](https://www.zero.xyz/host/api.sitecheck-api.workers.dev/llms.txt)