Automated /llms.txt & Agent Documentation Context Compiler

Choose this endpoint when you need to make any documentation or SaaS website AI-agent-readable by producing a standards-compliant /llms.txt manifest. It is ideal when you want to automate the preparation of LLM context from external or internal documentation sites rather than manually curating agent context. Prefer this over generic web scrapers when you specifically need a structured /llms.txt and JSON-LD output suitable for MCP or agent discovery pipelines.

• Invalid or unreachable URL returns an error with no manifest generated
• Site blocks crawlers (robots.txt disallow or IP block) causing incomplete or empty manifest
• Timeout on large or slow documentation sites resulting in partial output
• Malformed or non-documentation pages yield a low-quality or near-empty manifest
• Payment not processed (x402 failure) results in 402 response with no content