Automated /llms.txt & Agent Documentation Context Compiler

Use this endpoint when you need to make a website or documentation portal discoverable and consumable by AI agents via the /llms.txt standard. Ideal when onboarding a new SaaS product, API, or open-source library into an AI agent ecosystem, or when you need a one-shot automated compilation rather than manually authoring an llms.txt file. Prefer this over manual authoring or generic scrapers when you need both an /llms.txt manifest and a JSON-LD schema bundle in a single call.

• URL is unreachable or returns non-2xx HTTP status — crawl fails with error
• Website blocks crawlers via robots.txt or rate limiting — partial or empty manifest returned
• URL points to non-documentation content (e.g. login wall, empty page) — low-quality or empty manifest
• Malformed URL input — validation error returned
• Site content is dynamically rendered (heavy JS SPA) — crawler may miss content