RAG Token Diet – Web URL Distiller

Choose this endpoint when you need to feed web content into an LLM or RAG pipeline but must stay within a token budget — particularly when the raw page HTML would be prohibitively large. It is especially useful for agent workflows that repeatedly fetch and process web pages, where per-call token savings translate directly into reduced LLM costs. Prefer it over generic scrapers when you need built-in BM25 relevance filtering (mode=budget) to focus on a specific query, or when you want structured JSON tables or skeleton outlines rather than raw HTML.

• URL is unreachable or returns non-200 (fetch error, page unavailable)
• HTML is malformed or empty resulting in near-zero distilled output
• mode=budget used without providing a query or max_tokens (degraded output)
• max_tokens set too low (below 20) triggering validation error
• Payment not included or insufficient funds via x402 protocol (402 response)
• URL behind authentication or paywalls that block fetching
• Oversized raw HTML payload exceeding server limits