14.3k stars · MIT · v0.4.13 (2026-08-31) · Track this in Scout
Runs standard question sets against a language model and reports the scores.
▶Repo detailsthe review · specs · pros & cons · install
What it does
You name a model and a list of tasks, and it runs each task's questions through the model and scores the answers. Several hundred task definitions ship with it, covering reasoning, reading comprehension, mathematics, factual recall and more, and each one carries the exact prompt format and scoring rule in a YAML file. It can drive a model loaded locally through Hugging Face Transformers, one served over HTTP by an API, or one running under a serving engine. Few-shot evaluation is built in: you set how many worked examples go into the prompt, which is how published figures are usually produced. Results come back as a table and as JSON, with the number of questions and a standard error for each score. Adding your own task means writing a YAML file, not code. It does not judge open-ended writing quality, it does not test an application built around a model, and it is slow by nature.
Why it matters
Who it suits. Anyone choosing between models or quantisation levels for something they run themselves, and anyone who needs a number they can defend rather than an impression. It is the right tool for the question "if I drop to the smaller file, what do I lose?". Skip it if what you are testing is a product built on a model, because the thing that breaks there is your prompt and your retrieval, not the model's score on a reasoning set.
Verdict. Worth the setup if you make model choices more than once, because the alternative is trying a few prompts and trusting a feeling. The honest caveat is cost: a full run on a sizeable task set can take hours and pin your graphics card, so pick two or three tasks that match what you actually do rather than running everything. Version 0.4.13 landed on 31 August 2026 and code was pushed on 14 September 2026. Pair it with the application-level tools rather than choosing between them: Edition 45 covered confident-ai/deepeval and Edition 26 covered promptfoo/promptfoo, both of which test the thing you built rather than the model underneath it, and Edition 27 covered huggingface/lighteval for a lighter take on this same job. Use this one for the model, those for the product.
- The task definitions are the ones published figures use, so your score is comparable.
- Works against a local model, an HTTP API or a serving engine without changing the task.
- New tasks are YAML files, so a domain-specific question set needs no Python.
- Slow and hungry. A serious run is hours of compute, not minutes.
- Scores move with prompt formatting and few-shot count, so two runs are only comparable if the settings match exactly.
- It measures nothing about the application you wrapped around the model.
huggingface/lightevalThe same job with a smaller surface and several backends, easier to start with and covering fewer tasks. Covered in Edition 27.
Track this in Scout
confident-ai/deepevalTests the application you built on a model, in the shape of ordinary unit tests, rather than scoring the model itself. Covered in Edition 45.
Track this in Scout
promptfoo/promptfooCompares prompts and models against your own cases from a configuration file, aimed at product decisions rather than published benchmarks. Covered in Edition 26.
Track this in Scout
python3 -m venv venv source venv/bin/activate pip install lm-eval
