963 stars · MIT · v0.4.1 (2026-01-20) · Track this in Rising ↗
Fast semantic deduplication of text datasets — strips near-duplicate chat turns before topic modelling so clusters come out clean.
Radar check
What we found when we opened it. Every bar is one fact, with its date.
Last code pushed 4 Oct 2026, 23 days before this check.
Latest release v0.5.0 on 30 Sep 2026.
Public since October 2024, about 2 years.
977 stars and 56 forks.
4 commits and 0 new issues in the last 90 days.
6 contributors. 4 pull requests merged in 90 days. A small team carries it.
No security policy file says how to report a problem. No published security advisories.
How the bars are scored
- Alive. Days since the last code push: a week or less 100, a month 90, three months 70, six months 50, a year 30, eighteen months 15, older 0.
- Releases. Age of the latest GitHub release, on a similar scale; no bar when the project publishes no releases.
- Maturity. Age of the project: eight years 100, five 90, three 70, two 55, one 40, six months 25, younger 10.
- Adoption. Stars on a log scale: 100,000 stars is 100, 10,000 is 80, 1,000 is 60, 100 is 40.
- Activity. Commits in the last 90 days: 300 or more 100, 100 85, 30 65, 10 45, any 25.
- Community. People who have contributed: 100 or more 100, 30 80, 10 60, 5 40, 2 25, one 10.
- Security. A published security policy 80, none 40; published advisories are listed, not scored.
Checked by Repo Radar on 25 Oct 2026 from the repository on GitHub. Covered in Edition 17.
▶Repo detailsthe review · specs · pros & cons · install
What it does
It turns each text into a list of numbers that stands for its meaning, groups the ones that sit close together, and keeps one item from each group. The models it uses are small and fast, so it runs on an ordinary laptop with no graphics card. It handles a single list, and also a training-and-test pair, which matters when a model is being trained. It can report which items it treated as duplicates, so its judgement can be checked rather than trusted. Version 0.5.0 arrived on 30 September 2026 and needs Python 3.10 or newer. Closeness in meaning is a threshold you choose: too tight keeps repeats, too loose deletes something wanted. It removes duplicates; it does not summarise, cluster for presentation, or label anything.
Why it matters
Who it suits. Anybody who processes a pile of text before doing anything clever with it. Support tickets, survey answers, scraped pages and chat logs are all full of near-repeats, and removing them makes every later step faster and the results cleaner. It is especially useful before grouping or summarising long conversations, because a long conversation repeats itself constantly. Skip it if the texts are short and standardised, where exact matching already catches everything.
Verdict. Worth twenty minutes, and the threshold is the only thing to be careful about: set it on a sample you can read, not on the whole set. Note that the first run downloads a small model, so it needs an internet connection once. ekzhu/datasketch is the classic alternative and matches on wording rather than meaning, which is cheaper and blunter. MinishLab/model2vec is the embedding library from the same group if the numbers themselves are what you need.
Summary (Key Points)from the README
Key features
- Fast text embeddings: Uses Model2Vec to embed text, with Vicinity handling similarity search.
- Large-scale speed: Deduplicating 1.8M text records takes about 74 seconds on CPU.
- Scalable search: Vicinity's ANN backends are meant for datasets with millions of records.
- Default ANN backend: USearch is the default, with recall of about 100% on smaller datasets.
- Flexible modes: Works on one dataset, across two datasets (for example, train/test leakage), or on multi-column datasets such as QA data.
- Custom encoders: Any modality works if the encoder implements the Encoder protocol, which needs only an `encode(inputs, **kwargs)` method returning a numpy array. Sentence-transformers models and timm vision models are shown as examples.
- Flexible inputs: Accepts simple lists, Pandas DataFrames, pre-computed embeddings, and HuggingFace Datasets converted to lists.
- Inspection tools: `DeduplicationResult` shows duplicates and their causes, and can list the lowest-similarity duplicates to help choose a threshold.
- Filtering results: Outlier and representative-sample functions return a `FilterResult`, and the `.filtered` attribute exposes the selected records.
Important
- ANN requirement: The README says datasets above 1M samples need ANN search, since exact matching would take too long.
- Exact matching option: `ann_backend=Backend.BASIC` gives a flat, exact-matching backend, but the README recommends keeping the default.
- Optional dependencies: The text examples need the `datasets` package, and the image example needs `sentence-transformers`. The image example also uses a timm vision model.
- Custom backends: Any Vicinity backend can be used, but the README points to Vicinity's docs for the full list of backends and arguments.
- Benchmark scope: The only concrete benchmark figure given is the 1.8M-record text run. Detailed results are in a separate benchmarks directory.
Written by an AI model from the project’s README; not part of the review, and it can be wrong.
- Fast on a normal laptop. No graphics card needed.
- It can show which items it considered duplicates, so its judgement can be checked.
- It handles both a single list and a train-and-test pair.
- Closeness in meaning is a threshold you choose, and both extremes do damage.
- It downloads a small model the first time it runs, so the first run needs an internet connection.
- The project publishes releases slowly; judge it by its code rather than by its release page.
source venv/bin/activate # if you use a virtual environment pip install semhash
