RAG Token Diet – URL & HTML Distiller with BM25 Budget Control

Choose this endpoint when you need to reduce the token footprint of web content before feeding it into an LLM, especially when working with large documentation pages, HTML dumps, or any URL where you want to control the output size precisely. Prefer it over generic scrapers when you need BM25-ranked relevance filtering (budget mode) or structured JSON extraction (structured mode) rather than a raw content dump. It is purpose-built for RAG pipelines and agentic workflows where LLM context and cost are constraints.

• URL is unreachable or returns a non-200 status — endpoint may return an error status
• HTML payload is empty or malformed — distillation may return minimal or empty data
• Token budget (max_tokens) set too low — may produce a very short or truncated excerpt
• BM25 query missing when mode=budget — query field required for meaningful budget excerpts
• Payment failure via x402 — call rejected if USDC payment cannot be processed
• Rate limiting or idempotency key collision — retry with a fresh Idempotency-Key UUID