Web Distillation & BM25 Budget-Controlled Token Optimizer

Choose this endpoint when an AI agent needs to process web content within a strict token budget, reduce LLM API costs by compressing pages before sending to a model, or extract only query-relevant sections of a large webpage. Particularly valuable when processing documentation sites, news articles, or any HTML-heavy page where raw content would be too large for a context window. Prefer over generic scrapers when you need built-in BM25 relevance filtering and ROI metrics showing exactly how many tokens and dollars were saved.

• URL is unreachable or returns a non-200 status — endpoint may return an error status
• HTML payload is empty or malformed — distillation may produce minimal output
• max_tokens set too low for meaningful content extraction — result may be truncated
• BM25 query in budget mode finds no relevant sections — returns lowest-scoring content
• Payment not completed via x402 — request blocked before processing
• Rate limit or timeout fetching a slow target URL