963 stars · MIT · v0.4.1 (2026-01-20)
Fast semantic deduplication of text datasets — strips near-duplicate chat turns before topic modelling so clusters come out clean.
▶Repo detailsthe review · specs · pros & cons · install
What it is
A Python library that turns each text into a list of numbers that represents its meaning, then groups the ones that sit close together and keeps one from each group. It uses small, fast models, so it runs on an ordinary computer without a graphics card.What it is good for. Anybody who processes a pile of text before doing anything clever with it. Support tickets, survey answers, scraped pages and chat logs are all full of near-repeats. Removing them makes every later step faster and the results cleaner.This is directly relevant to Grasppy. A long chat repeats itself constantly: the same question asked twice, the same point restated. Stripping near-duplicates before grouping turns a map with six near-identical branches into one with six distinct ones.
- Fast on a normal laptop. No graphics card needed.
- It can show you which items it considered duplicates, so you can check its judgement rather than trusting it.
- It handles both a single list and a train-and-test pair, which matters if you are training anything.
- "Close in meaning" is a threshold you choose. Set it too tight and you keep repeats; too loose and you delete something you wanted.
- It downloads a small model the first time it runs, so the first run needs an internet connection.
- The version number moves slowly. Judge it by its code, not by its release page: the last code landed on 24 May 2026.
source venv/bin/activate # if you use a virtual environment pip install semhash