1.4k stars · MIT | 2.2.2 on PyPI, 25 Sep 2026 | [PyPI page](https://pypi.org/project/wtpsplit/) · 2.2.2 (2026-09-25) · Track this in Rising ↗
Splits text into real sentences in 85 languages, even when the writing has no punctuation.
Radar check
What we found when we opened it. Every bar is one fact, with its date.
Last code pushed 25 Sep 2026, 32 days before this check.
Latest release 2.2.2 on 25 Sep 2026.
Public since January 2020, a little over 6 years.
1,374 stars and 84 forks.
1 commits and 2 new issues in the last 90 days.
16 contributors. 1 pull requests merged in 90 days.
No security policy file says how to report a problem. No published security advisories.
How the bars are scored
- Alive. Days since the last code push: a week or less 100, a month 90, three months 70, six months 50, a year 30, eighteen months 15, older 0.
- Releases. Age of the latest GitHub release, on a similar scale; no bar when the project publishes no releases.
- Maturity. Age of the project: eight years 100, five 90, three 70, two 55, one 40, six months 25, younger 10.
- Adoption. Stars on a log scale: 100,000 stars is 100, 10,000 is 80, 1,000 is 60, 100 is 40.
- Activity. Commits in the last 90 days: 300 or more 100, 100 85, 30 65, 10 45, any 25.
- Community. People who have contributed: 100 or more 100, 30 80, 10 60, 5 40, 2 25, one 10.
- Security. A published security policy 80, none 40; published advisories are listed, not scored.
Checked by Repo Radar on 27 Oct 2026 from the repository on GitHub. Covered in Edition 15.
▶Repo detailsthe review · specs · pros & cons · install
What it does
wtpsplit finds sentence boundaries using small trained models. The current family is called SaT and covers 85 languages; an older family is kept for reproducing earlier results. The main call takes a string and returns a list of sentences, and passing a list of strings is faster than looping. It can also split into paragraphs, return the probability of a break at each position, or enforce a maximum and minimum segment length. On English the project reports an accuracy score of 96.5 for its small three-layer model and 97.4 for the twelve-layer one, against 92.2 for the long-standing Punkt method and 69.6 for one rule-based tool. Small add-on modules adapt it to a particular style of text — legal documents, mixed languages, short posts — in 81 languages, and a new one can be trained from as few as 10 to 100 example sentences. It runs on an ordinary processor, with a faster path through ONNX and an option to use a graphics card. It finds boundaries and nothing else: no parts of speech, no translation, no summary.
Why it matters
Who it suits. Anyone building a pipeline over real written conversation, where messages are lowercase, run on, and often have no full stops at all. It matters most where the next step depends on the cut: chunking, grouping into topics, and search over the pieces all inherit whatever the splitter handed over. Skip it when the text is clean, edited prose, because splitting on punctuation is already good there and far cheaper.
Verdict. Worth an hour, and the three-layer model is the right default: close to the best accuracy at a fraction of the size. Load the model once when a service starts, never per request, and run it in a background job rather than while somebody waits — Procrastinate (Edition 5 #10) is the queue for that. One model covers many languages, so there is no second path to maintain. explosion/spaCy is the better choice if the pipeline needs much more than boundaries.
Summary (Key Points)from the README
Key features
- Multilingual coverage: SaT models segment text across 85 languages, evaluated on 8 corpora.
- Strong accuracy: sat-12l-sm scores 97.4 English and 96.0 multilingual macro-average F1; sat-12l-lora scores 97.3 and 95.9.
- Speed options: 3-layer models (sat-3l, sat-3l-sm) are recommended for speed-sensitive use, and the README reports they give a strong speed-performance tradeoff.
- Faster inference: optimize() compiles PyTorch models with TorchInductor (PyTorch 2.0+), and ONNX export offers a faster path for sat and sat-sm models.
- Length-constrained segmentation: min_length and max_length settings, with priors ("uniform", "gaussian", "lognormal", "clipped_polynomial") and language-aware defaults.
- Paragraph segmentation: SaT models predict newline probability, so they can also split text into paragraphs.
- LoRA adaptation: trained modules cover Universal Dependencies, OPUS100, Ersatz and TED styles in 81 languages, plus legal, code-switching and tweet domains; the README says 10-100 segmented sentences can already improve performance.
- Adjustable threshold: a higher threshold makes segmentation more conservative.
- Reproducibility: configs, evaluation scripts and raw results from the paper are included, and WtP models are kept for reproducibility.
Ways to install
| Platform or method | What it needs | Effort |
|---|---|---|
| Standard package install | Python environment with the library's dependencies | Default install |
| ONNX support | Separate install option for ONNX inference; ONNX export script for sat and sat-sm models | Optional extra |
| NVIDIA AITune (auto backend selection) | Linux, CUDA, and a separate install | Separate install |
| Community Rust port (wtsplit-rs) | Independently maintained port | Depends on the port |
Important
- Backend conflict: the torch.compile path does not work with ONNX or ort_providers, and it applies only to SaT and WtP PyTorch checkpoints.
- Slow first call: the first split after optimize() can be slow while graphs are built.
- CUDA graphs: mode="reduce-overhead" helps only when every forward pass uses the same shapes.
- AITune limits: it requires Linux and CUDA.
- Model matching: a LoRA adapter must be used with the same model variant it was trained on, such as sat-12l-sm versus sat-12l.
- Training data format: LoRA training sentences must not contain newlines, and the library raises an error if they do.
- Threshold override: when max_length is set, the threshold parameter is ignored.
- Newlines in output: length-constrained segments may contain newlines, which may need post-processing.
- Community ports: they are independently maintained and may differ in features or update timing.
Written by an AI model from the project’s README; not part of the review, and it can be wrong.
- MIT, and the small models run on an ordinary processor. No graphics card needed.
- It handles text with no punctuation, which is exactly the case that breaks the simple approach.
- One language-agnostic model covers many languages at once, so one code path serves them all.
- It is a model, so the first run downloads weights and it wants a few hundred MB of memory.
- It is slower than splitting on full stops. Put it in a background job, not in a request.
- Options interact in ways worth reading about: setting a maximum length makes the sensitivity setting inactive.
source venv/bin/activate
pip install wtpsplit
python -c "from wtpsplit import SaT; s = SaT('sat-3l-sm'); print(s.split('hello how are you all fine thanks'))"
