3 Services from robots-check.quietworklabs.workers.dev

Checks whether a URL is crawlable by a given user-agent under robots.txt rules, identifies which AI crawlers (GPTBot, ClaudeBot, etc.) are blocked, and reads noindex/noai directives and sitemaps

Checks which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked by a URL's robots.txt, page-level directives, and llms.txt

Validates an XML sitemap for a given URL, checking structure, duplicate entries, HTTP status, and loc count