Automated /llms.txt & Agent Documentation Context Compiler

Choose this endpoint when you need to automatically generate a machine-readable /llms.txt manifest from any existing documentation or SaaS website without manually authoring one. Especially useful when onboarding a new tool or service into an AI agent workflow and needing structured context fast. Prefer this over manual llms.txt authoring or generic web scrapers when you need standards-compliant output (llms.txt spec + JSON-LD) ready for agent consumption.

• URL is unreachable or returns non-200 status — crawl fails with error
• Website blocks crawlers via robots.txt or rate limiting — partial or empty manifest returned
• URL points to a non-documentation or non-SaaS page with insufficient structured content — low-quality or empty output
• Invalid or malformed URL input — validation error returned
• Very large sites may only partially crawl within a single call