# Creator Desk Crawl Config (robots.txt Inspector)

> Creator Desk Crawl Config (robots.txt Inspector) is a paid API for AI agents from creator-desk-x402-isolated.onrender.com, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-13).

Fetches and parses a website's robots.txt file, returning declared crawl directives, disallow rules, crawl delay, and sitemap URLs for the wildcard user-agent group.

## Facts

- Endpoint: POST https://creator-desk-x402-isolated.onrender.com/v1/site/crawl-config?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/creator-desk-crawl-config-robots-txt-inspector-5a2e3d45
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_I-KIIAnaLN-I2-jFeYCYl

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability creator-desk-crawl-config-robots-txt-inspector-5a2e3d45 -d '<json body>'
```

Example prompt: Before I start scraping, can you check what https://www.techcrunch.com/articles/ai-news has in its robots.txt — specifically which paths are disallowed and whether there's a sitemap listed?

## When to prefer this

Use this endpoint when an AI agent needs to programmatically understand a site's crawl policy before fetching or indexing its pages. It is the right choice when you need parsed, structured robots.txt data (disallow paths, crawl delay, sitemaps) rather than raw file content, and when you want a fingerprint to detect future changes. Prefer this over manually fetching /robots.txt when you need reliable parsing of wildcard directives and sitemap extraction in a single call.

## Known failure modes

- Target site is unreachable or returns non-200 for robots.txt — HTTP status and latency still reported but rules may be empty
- robots.txt does not exist (404) — wildcard_group.found will be false
- Site blocks the inspector's IP — connection timeout or unexpected status code returned
- Rules list truncated if robots.txt is extremely large (rules_truncated: true flag set)
- Invalid or non-HTTPS URL provided — request rejected before fetch
- Sitemaps list truncated if many sitemaps declared (sitemaps_truncated: true flag set)

## How this service works

Machine-callable x402-paid utilities for autonomous agents.

## Output

Returns a JSON object containing: the resolved robots.txt URL and its HTTP status/latency, parsed wildcard user-agent rules (allowed/disallowed paths, crawl delay), a list of declared sitemap URLs, a SHA-256 fingerprint of the robots.txt body, the inspection timestamp, and scope caveats noting that the result reflects declared directives only and does not certify legal crawl permission.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "maxLength": 2048,
   "description": "Any public HTTPS URL on the target site. The tool inspects that origin's /robots.txt."
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "http": {
   "status": 200,
   "latency_ms": 90,
   "status_text": "OK",
   "content_type": "text/plain"
  },
  "scope": "Parses declared robots.txt directives for User-agent: * and Sitemap lines. It does not certify permission, legal compliance, or how every crawler interprets robots rules.",
  "origin": "https://example.com",
  "version": "1.0.0",
  "sitemaps": [
   "https://example.com/sitemap.xml"
  ],
  "product_id": "creator_desk_crawl_config",
  "robots_url": "https://example.com/robots.txt",
  "inspected_at": "2026-09-25T00:00:00.000Z",
  "requested_url": "https://example.com/some/page",
  "wildcard_group": {
   "found": true,
   "rules": [
    {
     "path": "/private/",
     "directive": "disallow"
    }
   ],
   "crawl_delay": null,
   "rules_truncated": false
  },
  "body_fingerprint": {
   "sha256": "0000000000000000000000000000000000000000000000000000000000000000",
   "bytes_read": 128
  },
  "sitemaps_truncated": false
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/creator-desk-crawl-config-robots-txt-inspector-5a2e3d45/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from creator-desk-x402-isolated.onrender.com](https://www.zero.xyz/host/creator-desk-x402-isolated.onrender.com/llms.txt)