# Crawlspur Meta-Robots URL-Regel Checker

> Crawlspur Meta-Robots URL-Regel Checker is a paid API for AI agents from crawlspur.halowerk.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-16).

Checks the meta-robots tag value for a given URL rule object, returning the crawl/indexing directive found at that URL.

## Facts

- Endpoint: POST https://crawlspur.halowerk.com/v1/pruef/crawlspur/meta-robots/url-regel
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-16
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/crawlspur-meta-robots-url-regel-checker-b0deec55
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_KZayDg60i5OhU1Kw58Pr8

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability crawlspur-meta-robots-url-regel-checker-b0deec55 -d '<json body>'
```

Example prompt: Can you check what the meta-robots tag value is for https://example.com/product-page — I need to know if it's set to noindex or nofollow.

## When to prefer this

Use this endpoint when you need to programmatically verify the meta-robots HTML tag value for a specific URL as part of a technical SEO audit, indexability check, or automated crawl rule validation. Prefer this over robots.txt checks when the directive is embedded in page HTML rather than the robots file, and over general page scrapers when you specifically need the structured crawl directive finding.

## Known failure modes

- URL is not publicly accessible or returns a non-200 HTTP status
- No meta-robots tag present on the page (empty or null result)
- URL exceeds maximum length of 2048 characters
- CSS selector matches no element or multiple elements (first match used)
- Network timeout or DNS resolution failure for the target URL
- Invalid URL format provided

## How this service works

Prüft am Objekt url-regel den Befund Wert der Meta-Robots-Angabe.

## Output

Returns the value of the meta-robots tag found on the inspected URL, indicating the crawl/indexing directive such as 'noindex', 'nofollow', 'index,follow', or similar robot instructions declared in the page's HTML head.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "ziel": {
   "oneOf": [
    {
     "type": "string",
     "maxLength": 2048,
     "minLength": 1,
     "description": "Öffentliche HTTP(S)-Adresse; bei DNS-Befunden Domain oder öffentliche IP."
    },
    {
     "type": "object",
     "required": [
      "adresse"
     ],
     "properties": {
      "adresse": {
       "type": "string",
       "maxLength": 2048,
       "minLength": 1
      },
      "selektor": {
       "type": "string",
       "maxLength": 240,
       "minLength": 1,
       "description": "CSS-Selektor: genau ein Objekt, sonst erster Treffer des Objektfilters."
      },
      "vergleich": {
       "type": "string",
       "maxLength": 2048,
       "description": "Öffentliche Vergleichsadresse für Link-, Sitemap- oder Sprachbefunde."
      },
      "user_agent": {
       "type": "string",
       "pattern": "^[A-Za-z0-9_-]{1,80}$"
      },
      "dkim_selektor": {
       "type": "string",
       "pattern": "^[A-Za-z0-9_-]{1,63}$"
      }
     },
     "additionalProperties": false
    }
   ]
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/crawlspur-meta-robots-url-regel-checker-b0deec55/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from crawlspur.halowerk.com](https://www.zero.xyz/host/crawlspur.halowerk.com/llms.txt)