PrepLayer.ai Near-Duplicate Document Detection

Choose this endpoint when you need fast, deterministic near-duplicate detection over a batch of text documents using a mathematically rigorous Jaccard similarity metric. It is ideal for data cleaning pipelines, corpus deduplication before ML training, content audits, and plagiarism screening where you need both a similarity decision and a compact fingerprint (SimHash) per document for downstream indexing. Prefer it over semantic similarity approaches when you want exact lexical overlap measurement rather than meaning-based comparison, and when batch processing up to 200 documents per call fits your workflow.

• Batch exceeds 200 documents — returns a validation error
• Threshold out of range (not 0–100) — returns a schema validation error
• Empty documents array — rejected with minItems constraint error
• Very short documents may produce unreliable shingle-based similarity scores
• Non-text or binary content passed as strings may yield meaningless SimHash values