Crawlspur Meta-Robots Pagination Checker

Use this endpoint specifically when you need to audit the meta-robots directive on paginated pages (e.g. ?page=2, /page/3) for SEO purposes. Prefer this over generic HTTP inspection tools when you want a structured SEO finding focused on pagination crawlability, and when you need to identify whether paginated URLs are inadvertently set to noindex or nofollow.

• URL is not publicly accessible or returns non-200 status
• No pagination object found at the target URL
• Meta-robots tag absent from the page (finding returns empty/null value)
• CSS selector matches zero or multiple elements
• Invalid URL format or URL exceeds length limit
• Target page blocks crawl bots via robots.txt or firewall