2.5k stars · MIT · v0.13.0 (2025-11-24) per ungh.cc/repos/huggingface/lighteval/releases/latest — ten months old, while ungh.cc reports code pushed 2026-09-17. Install from the repository for what the documentation describes · Track this in Scout
Runs more than a thousand standard public tests against a language model and reports the score.
▶Repo detailsthe review · specs · pros & cons · install
What it is
A Python package with more than a thousand ready-made evaluation tasks covering knowledge, maths, code and several languages. It runs a model loaded locally or reached through an API, and writes per-sample results so a score can be traced back to the answers behind it.
What it is good for. Choosing between models on evidence rather than on a leaderboard somebody else ran. The published numbers for a model were measured on somebody's hardware with somebody's settings, and the whole point of this is that you can repeat the measurement with yours. It matters most before a decision that is expensive to reverse: swapping the model behind a product, or deciding whether a small model you can run yourself is good enough to replace one you pay for. The per-sample output is the part people underuse — an average score hides which kind of question the model fails.
- MIT, and it is maintained by Hugging Face, so the task definitions track the community's standard ones.
- It compares a local model and a paid service under identical conditions, which is exactly the comparison that is otherwise unfair.
- Over a thousand tasks including multilingual ones, so a model can be checked in a language other than English.
- Its newest release, v0.13.0, is dated 24 November 2025 — ten months ago — although the code was last pushed on 17 September 2026. Install from the repository if you want what the documentation describes.
- That release introduced a breaking change to how tasks are defined, so older examples found online will not run.
- Windows is not supported, and a full benchmark run costs real money or real hours of graphics-card time.
promptfoo/promptfooIt tests your own prompt against many models and marks the answers against rules you write, where this one scores a model against standard public tests.
Track this in Scout- EleutherAI/lm-evaluation-harness
It is the older framework most published benchmark numbers come from, and this project borrows heavily from it.
Track this in Scout - confident-ai/deepeval
It also measures model quality, but it is aimed at testing a whole application's answers as part of a test suite.
Track this in Scout
python3 -m venv venv source venv/bin/activate pip install lighteval lighteval endpoint inference-providers \ "model_name=deepseek-ai/DeepSeek-R1,provider=hf-inference" \ "lighteval|gsm8k|0"
