# RAG Chunk: URL/HTML Distillation with BM25 Budget Control

> RAG Chunk: URL/HTML Distillation with BM25 Budget Control is a paid API for AI agents from shopify-spy.agentdiet.dev, paid per call via MPP, $0.001/call, status unknown (last checked 2026-10-23).

Fetches and distills a URL or raw HTML into compact, agent-ready tokens using BM25-ranked budget trimming with configurable output modes.

## Facts

- Endpoint: POST https://shopify-spy.agentdiet.dev/v1/rag-chunk?utm_source=zero.xyz
- Price: $0.001/call
- Payment: MPP
- Status: unknown
- Last checked: 2026-10-23
- Activations on Zero: 0
- Provider: shopify-spy.agentdiet.dev
- Website: https://shopify-spy.agentdiet.dev
- Canonical page: https://www.zero.xyz/c/shopify-spy-agentdiet-dev-rag-chunk-url-html-distillation-with-bm25-8e8d3b4b
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_0yQPvK_DYIkOUwcKv3NpJ

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability shopify-spy-agentdiet-dev-rag-chunk-url-html-distillation-with-bm25-8e8d3b4b -d '<json body>'
```

Example prompt: Distill the content at https://stripe.com/docs/payments into clean markdown using BM25 budget mode, focused on the query 'payment intents', and keep it under 500 tokens.

## When to prefer this

Choose this endpoint when you need to ingest web pages or HTML into an LLM pipeline while minimizing token consumption. It is particularly valuable when you have a specific focus query and want BM25-ranked relevance filtering, or when you need to stay within a strict token budget. Prefer this over raw fetch-and-parse approaches when token cost savings and compression ratio matter, or when you want structured/skeleton output modes rather than raw HTML.

## Known failure modes

- URL is unreachable or returns non-200 status — distillation fails with fetch error
- HTML payload is malformed or empty — parser may return minimal or empty output
- Token budget (max_tokens) set too low to capture meaningful content
- BM25 query yields no matching chunks within budget — returns minimal excerpt
- Rate limit or payment failure via x402 — returns 402 Payment Required
- Invalid mode enum value — request rejected with validation error

## How this service works

Distill a URL or raw HTML payload with BM25 budget control (Live x402 v2, IETF MPP & MCP endpoint at https://shopify-spy.agentdiet.dev/v1/rag-chunk)

## Output

Returns a data object containing the distilled content (markdown, structured JSON, skeleton outline, or BM25-ranked budget excerpt depending on mode), along with ROI proof metrics (raw token count, distilled token count, tokens saved, compression ratio percentage, estimated LLM cost saved in USD, ROI multiplier), optional cache status (HIT/MISS), and billing details.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "required": [
  "url"
 ],
 "properties": {
  "url": {
   "type": "string",
   "minLength": 1,
   "description": "Target webpage URL to fetch and distill (or builtin://stripe-docs)"
  },
  "html": {
   "type": "string",
   "description": "Optional raw HTML string to distill directly without fetching"
  },
  "mode": {
   "enum": [
    "markdown",
    "structured",
    "skeleton",
    "budget"
   ],
   "type": "string",
   "default": "markdown",
   "description": "Distillation mode"
  },
  "query": {
   "type": "string",
   "description": "BM25 focus query when mode=budget"
  },
  "max_tokens": {
   "type": "integer",
   "minimum": 20,
   "description": "Strict token budget ceiling when mode=budget"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "object",
 "required": [
  "status",
  "roi_proof",
  "data"
 ],
 "properties": {
  "data": {
   "type": "object",
   "description": "Distilled Markdown, structured JSON tables/metadata, skeleton outline, or BM25 budget excerpt."
  },
  "cache": {
   "type": "object",
   "properties": {
    "status": {
     "enum": [
      "HIT",
      "MISS"
     ],
     "type": "string"
    },
    "cache_key": {
     "type": "string"
    },
    "latency_ms": {
     "type": "number"
    }
   }
  },
  "status": {
   "type": "string",
   "description": "Execution status ('ok')"
  },
  "billing": {
   "type": "object",
   "properties": {
    "agent_id": {
     "type": "string"
    },
    "payment_method": {
     "type": "string"
    },
    "fee_charged_usd": {
     "type": "number"
    }
   }
  },
  "roi_proof": {
   "type": "object",
   "properties": {
    "raw_tokens": {
     "type": "integer"
    },
    "tokens_saved": {
     "type": "integer"
    },
    "roi_multiplier": {
     "type": "number"
    },
    "distilled_tokens": {
     "type": "integer"
    },
    "llm_cost_saved_usd": {
     "type": "number"
    },
    "compression_ratio_pct": {
     "type": "number"
    }
   }
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/shopify-spy-agentdiet-dev-rag-chunk-url-html-distillation-with-bm25-8e8d3b4b/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from shopify-spy.agentdiet.dev](https://www.zero.xyz/host/shopify-spy.agentdiet.dev/llms.txt)