LLMs.txt & Agent Documentation Context Compiler

Choose this endpoint when you need to make an existing website or documentation portal immediately discoverable and consumable by AI agents via the emerging /llms.txt standard. It is especially useful when you don't want to manually author agent context files, when the target site lacks an existing /llms.txt, or when you need a JSON-LD schema bundle alongside the manifest. Prefer this over generic scraping tools when the goal is specifically agent-friendly structured output rather than raw HTML or markdown extraction.

• Invalid or unreachable URL returns an error with no manifest generated
• Site blocks crawlers (robots.txt or CAPTCHA) resulting in partial or empty output
• Very large documentation sites may produce truncated manifests
• Non-documentation sites (e.g. e-commerce storefronts) may yield low-quality or sparse output
• Payment failure via x402 blocks the request before crawling begins