# Crawlspur robots.txt Crawl-Delay Checker

> Crawlspur robots.txt Crawl-Delay Checker is a paid API for AI agents from crawlspur.halowerk.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-14).

Checks a robots.txt file to detect and extract the declared Crawl-Delay directive for a given URL or domain.

## Facts

- Endpoint: POST https://crawlspur.halowerk.com/v1/pruef/crawlspur/crawl-delay/robots-datei
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/crawlspur-robots-txt-crawl-delay-checker-4709fb64
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_gba0FrY7ZpLuyUHstzJ73

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability crawlspur-robots-txt-crawl-delay-checker-4709fb64 -d '<json body>'
```

Example prompt: Check the robots.txt file at https://example.com and tell me what crawl-delay value is declared there.

## When to prefer this

Use this endpoint when you need to programmatically determine the crawl-delay value declared in a website's robots.txt file, particularly for SEO auditing, compliance checking before web scraping, or monitoring changes in crawl policy. Prefer this over generic HTTP fetchers when you specifically need structured extraction of the crawl-delay directive with validation logic built in.

## Known failure modes

- Target URL unreachable or returns non-200 status — check fails with connection or HTTP error
- robots.txt file not found at target domain — no crawl-delay can be extracted
- Invalid or malformed URL in 'ziel' field — validation error returned
- Target is a private/non-public IP or domain — rejected as invalid target
- No crawl-delay directive present in robots.txt — result indicates absence rather than error

## How this service works

Prüft am Objekt robots-datei den Befund deklarierte Crawl-Verzögerung.

## Output

Returns the declared crawl-delay value found in the robots.txt file for the specified target URL or domain, including the finding status (present/absent), the numeric delay value if declared, and relevant metadata about the robots.txt check.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "ziel": {
   "oneOf": [
    {
     "type": "string",
     "maxLength": 2048,
     "minLength": 1,
     "description": "Öffentliche HTTP(S)-Adresse; bei DNS-Befunden Domain oder öffentliche IP."
    },
    {
     "type": "object",
     "required": [
      "adresse"
     ],
     "properties": {
      "adresse": {
       "type": "string",
       "maxLength": 2048,
       "minLength": 1
      },
      "selektor": {
       "type": "string",
       "maxLength": 240,
       "minLength": 1,
       "description": "CSS-Selektor: genau ein Objekt, sonst erster Treffer des Objektfilters."
      },
      "vergleich": {
       "type": "string",
       "maxLength": 2048,
       "description": "Öffentliche Vergleichsadresse für Link-, Sitemap- oder Sprachbefunde."
      },
      "user_agent": {
       "type": "string",
       "pattern": "^[A-Za-z0-9_-]{1,80}$"
      },
      "dkim_selektor": {
       "type": "string",
       "pattern": "^[A-Za-z0-9_-]{1,63}$"
      }
     },
     "additionalProperties": false
    }
   ]
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/crawlspur-robots-txt-crawl-delay-checker-4709fb64/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from crawlspur.halowerk.com](https://www.zero.xyz/host/crawlspur.halowerk.com/llms.txt)