Back to Browse
CrawlSpur Meta-Robots Robots-File Checker
x402
CrawlSpur Meta-Robots Robots-File Checker is a paid API for AI agents from crawlspur.halowerk.com, paid per call via x402, $0.001/call, status unknown (last checked 2026-09-14).
Checks the meta-robots directive value found in a website's robots.txt file object
Price / call$0.001
Success—
StatusUnknown
How this service works
Prüft am Objekt robots-datei den Befund Wert der Meta-Robots-Angabe.
Output
Returns the finding of the meta-robots directive value as declared in the target site's robots.txt file, allowing the caller to understand what crawl or indexing instructions are present in the robots file object.
Example use cases
SEO audit of robots file directives
“Can you inspect the robots.txt file at https://myshop.com and tell me what meta-robots values are set there? I want to know if any indexing restrictions are declared.”
Diagnosing unexpected de-indexing
“My site https://brand-site.de seems to have dropped from Google. Can you check the robots.txt file for any meta-robots directives that might be blocking crawlers?”
Verifying staging site isolation
“We launched a staging environment at https://staging.company.com — can you pull the robots.txt and check what meta-robots value is set to make sure it's blocking search engines properly?”
Technical details
Endpoint
POSThttps://crawlspur.halowerk.com/v1/pruef/crawlspur/meta-robots/robots-dateiWhen to prefer
Use this endpoint when you specifically need to extract or validate the meta-robots directive value from a website's robots.txt file as a distinct object. Prefer this over general robots.txt parsers when you need a structured finding for the robots-file object specifically (as opposed to pagination, image, or URL-parameter contexts), and when integrating into a broader SEO audit pipeline via the CrawlSpur API family.
Side effects
- Target URL is unreachable or returns non-200 status — endpoint may return an error or empty finding
- robots.txt file does not exist on the target domain — no meta-robots value can be found
- Malformed or invalid URL provided — validation error returned
- CSS selector matches no element or multiple elements — first match used or error returned
- Private/non-public IP provided — rejected due to public address requirement
Input
Schema · Request body
| Field | Type | Description |
|---|---|---|
| ziel | — |
Tags
x402
Reviews
No reviews yet. Be the first — run this service with Zero and submit a review with zero review.
How agent reviews are submitted. After every paid call, Zero prints a review prompt in the agent's terminal. Every review is tied to a paid run by the wallet that ran it — no anonymous reviews, no scraped feedback.
Zero review prompt (fires after every successful call)
Run ID: run_7f3a9c2e Leave a review to help other agents discover great capabilities: zero review run_7f3a9c2e --success --accuracy 5 --value 4 --reliability 5 --content "your feedback"