# 402utils Robots.txt Compliance Checker

> 402utils Robots.txt Compliance Checker is a paid API for AI agents from 402utils.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-12).

Checks whether a given URL is allowed or disallowed for a specific crawler bot by fetching and parsing the host's robots.txt per RFC 9309

## Facts

- Endpoint: GET https://402utils.com/v1/robots-check?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-12
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/402utils-robots-txt-compliance-checker-af66e5d5
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_HX2rRDgwDHmV6WB6gEhBG

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability 402utils-robots-txt-compliance-checker-af66e5d5
```

Example prompt: Before I scrape https://example.com/blog/article-123, can you check if GPTBot is allowed to fetch that URL according to the site's robots.txt?

## When to prefer this

Use this endpoint when an AI agent or crawler needs to verify RFC 9309-compliant robots.txt access rules before fetching a URL, especially when you need wildcard/anchor matching, longest-match resolution, crawl-delay extraction, and sitemap discovery all in one call. Prefer over manual robots.txt parsing when you need full RFC 9309 compliance including merged groups, $ anchors, and proper 4xx/5xx fallback behavior. Cached per host for 1 hour, making it efficient for batches of URLs on the same domain.

## Known failure modes

- Host unreachable — robotsStatus is 0, allowed is false per RFC 9309
- robots.txt returns 5xx — allowed is false per RFC 9309
- robots.txt returns 4xx — allowed is true (permissive fallback)
- Invalid or malformed URL input — likely 4xx API error
- Robots.txt cache miss causes slight latency increase on first fetch per host

## How this service works

Robots.txt compliance check for polite crawlers and AI agents (GPTBot, ClaudeBot, ...). Pass a page URL and optionally your crawler's product token; returns whether the URL may be fetched, the winning Allow/Disallow rule, crawl-delay and sitemap URLs. Full RFC 9309 matching: wildcards, $ anchors, longest-match, merged groups. A 4xx robots.txt allows all; 5xx or unreachable hosts report disallowed. Cached per host for 1 hour.

## Output

Returns a JSON object with: allowed (boolean), matchedRule (the winning Allow/Disallow directive as written), crawlDelay (seconds from the matched group), sitemaps (array of sitemap URLs declared in robots.txt), robotsUrl (URL of the fetched robots.txt), robotsStatus (HTTP status of the robots.txt response, 0 if unreachable). A 4xx robots.txt status means all crawling is allowed; 5xx or unreachable means disallowed per RFC 9309.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri",
       "description": "Public http(s) page URL to test (required)."
      },
      "userAgent": {
       "type": "string",
       "default": "*",
       "description": "Crawler product token, e.g. \"GPTBot\". Defaults to \"*\"."
      }
     },
     "description": "Query string only."
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "required": [
      "url",
      "userAgent",
      "allowed",
      "robotsUrl",
      "robotsStatus",
      "sitemaps"
     ],
     "properties": {
      "url": {
       "type": "string"
      },
      "allowed": {
       "type": "boolean"
      },
      "sitemaps": {
       "type": "array",
       "items": {
        "type": "string"
       }
      },
      "robotsUrl": {
       "type": "string"
      },
      "userAgent": {
       "type": "string"
      },
      "crawlDelay": {
       "type": "number",
       "description": "Crawl-delay of the matched group, in seconds."
      },
      "matchedRule": {
       "type": "string",
       "description": "Winning rule as written, e.g. \"Disallow: /search\". Absent when no rule matched."
      },
      "robotsStatus": {
       "type": "integer",
       "description": "HTTP status of robots.txt; 0 when it was unreachable (network failure ⇒ full disallow per RFC 9309)."
      }
     }
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com/search?q=demo",
  "allowed": false,
  "sitemaps": [
   "https://example.com/sitemap.xml"
  ],
  "robotsUrl": "https://example.com/robots.txt",
  "userAgent": "GPTBot",
  "matchedRule": "Disallow: /search",
  "robotsStatus": 200
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/402utils-robots-txt-compliance-checker-af66e5d5/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from 402utils.com](https://www.zero.xyz/host/402utils.com/llms.txt)