Automated /llms.txt & Agent Documentation Context Compiler

Choose this endpoint when you need to programmatically generate an /llms.txt manifest for any documentation or SaaS site without manually authoring it. It is ideal for bootstrapping agent-readable context for third-party sites, preparing MCP-compatible documentation bundles, or automating llms.txt generation at scale. Prefer it over manual authoring or generic scrapers when you specifically need standards-compliant /llms.txt and JSON-LD output.

• Target URL is unreachable or returns non-200 status — crawler cannot access the site
• Site blocks crawlers via robots.txt or rate limiting — partial or empty manifest returned
• URL points to a non-documentation resource (e.g. login wall, paywall) — insufficient content to compile meaningful manifest
• Malformed or missing URL parameter — validation error returned
• Crawl times out on very large or slow-loading documentation sites