Crawlspur robots.txt URL Rule Checker

Use this endpoint when you need to programmatically verify whether a specific URL or URL rule is permitted by a site's robots.txt before crawling, especially in SEO auditing, crawler pre-flight checks, or compliance workflows. Prefer this over manual robots.txt inspection when dealing with complex rule sets or large-scale URL auditing.

• URL is malformed or exceeds 2048 characters — validation error returned
• Target site is unreachable or returns non-200 status — crawl fails
• robots.txt file is missing or inaccessible — indeterminate result
• CSS selector matches no element or multiple elements — first match used or error
• user_agent string contains invalid characters — schema validation failure