Web Content Distillation with BM25 Budget Control

Choose this endpoint when you need to reduce token consumption from web content before passing it to an LLM, especially when you have a strict context window budget or want query-focused excerpts via BM25 ranking. It is superior to raw web scraping for agent workflows because it returns pre-compressed, LLM-ready content with quantified token savings. Prefer over generic scrapers when you need structured output modes (markdown, JSON tables, skeleton) or when working with large documentation pages like Stripe or OpenAPI specs. The BM25 budget mode is uniquely suited for RAG pipelines where only query-relevant passages are needed.

• URL is unreachable or returns non-200 — fetch failure with error status
• HTML payload is empty or malformed — distillation returns minimal or empty data
• max_tokens set too low for meaningful content — may return truncated or nearly empty excerpt
• BM25 query provided without mode=budget — query may be ignored
• Payment not included or insufficient — HTTP 402 response blocking execution
• Rate limiting or upstream timeout on slow target URLs