Automated /llms.txt & Agent Documentation Context Compiler

Choose this endpoint when you need to programmatically generate a machine-readable /llms.txt manifest and JSON-LD schema bundle from any live website or documentation portal — especially when the target site does not already publish an /llms.txt file. It is ideal for onboarding third-party SaaS documentation into an agent's tool registry, automating context compilation for LLM pipelines, or preparing your own documentation for AI agent discovery without manual authoring.

• Invalid or unreachable URL returns an error with HTTP 4xx and a description of the crawl failure
• Paywalled or login-gated documentation sites may return incomplete or empty manifests
• Sites with aggressive bot-blocking (CAPTCHAs, Cloudflare) may fail to crawl successfully
• Very large documentation sites may return partial results or time out
• Malformed URL input triggers a validation error before crawling begins
• Payment failure via x402 results in HTTP 402 before any crawl is attempted