Automated /llms.txt & Agent Documentation Context Compiler

Choose this endpoint when you need to make any external documentation or SaaS site consumable by AI agents in a standards-compliant format (/llms.txt). It is especially useful when you want a fully compiled manifest plus JSON-LD schema bundle in a single call, rather than manually writing or maintaining an /llms.txt file. Prefer it over generic web scrapers when the output needs to conform to the emerging /llms.txt standard for LLM context delivery.

• URL is unreachable or returns non-200 status — crawl fails with an error response
• Target site blocks crawlers via robots.txt or Cloudflare — partial or empty manifest returned
• URL points to a login-gated page — only publicly accessible content is extracted
• Malformed or missing URL parameter — returns validation error
• Very large documentation sites may produce truncated manifests due to crawl depth limits