5.9k stars · Apache-2.0 · giskard-agents/v1.0.5 (2026-10-20) · Track this in Scout
A Python library that writes tests for language-model programs and tries to break them on purpose.
▶Repo detailsthe review · specs · pros & cons · install
What it does
It is split into packages so you can take only the part you need. The checks package gives you a way to describe a scenario and then assert things about the answer: that it contains a string, that it matches a pattern, that it means roughly the same as an expected answer, that it is grounded in the documents it was given, and that it follows a stated rule. Some of those checks are done by another language model acting as a judge, and the grounded check is one of them, with a default judge model of gpt-4o-mini. The scan package does the breaking: it probes for prompt injection, which is text that tricks a model into ignoring its instructions, and for jailbreaks, which are attempts to get past its rules. It can also score the quality of a document collection the program reads from. The giskard-checks package reached version 1.0.5 on 2 October 2026 and needs Python 3.12 or newer. What it no longer does is the old version-2 work on ordinary machine-learning models over tables: that scan stays on version 2, which the project says is no longer actively maintained.
Why it matters
Who it suits. Anyone shipping a feature built on a language model who wants a test suite rather than a feeling. It suits the case where answers must stay inside a set of documents, because the grounded check is aimed exactly at that. It also suits anyone who has to answer a question about prompt injection before a launch, since the scan produces a list rather than an opinion. Skip it if the model is doing something small and internal where a wrong answer costs nothing, and skip it if your models are the older kind that read tables rather than text.
Verdict. Worth half a day, and the half day is mostly deciding what a correct answer actually is, which is the valuable part. Two practical notes. Python 3.12 is a hard floor and that is newer than many projects are on, so check before you plan. And the judge-based checks need a provider key and bill per check, so a large test suite has a running cost that an ordinary test suite does not. The honest limit is that a model judging a model is a useful signal and not a proof, so keep the cheap string and pattern checks as the backbone and use the judges for the parts that cannot be written down. For a test-runner feel there is confident-ai/deepeval; for red-teaming alone there is promptfoo/promptfoo.
- Apache-2.0, and split into small packages so you can install only the checks.
- The scan looks for prompt injection and jailbreaks, which are hard to test by hand.
- Telemetry can be switched off with one environment variable before the first import.
- Python 3.12 or newer is required, which rules out a lot of existing projects.
- The judge-based checks need a provider key and cost money every time they run.
- The older table-based scan is version-2 only, and version 2 is no longer actively maintained.
confident-ai/deepevalApache-2.0, the same idea shaped like a unit-test framework, so it fits a project that already runs tests on every change.
Track this in Scout
promptfoo/promptfooMIT, a command-line tool for comparing prompts and red-teaming; stronger on comparing two prompts side by side, weaker on assertions inside a test suite.
Track this in Scout
python3 -m venv venv source venv/bin/activate pip install giskard-checks # needs Python 3.12 or newer # add the breaking tests and one provider pip install "giskard[scan]" pip install "giskard[openai]" # turn telemetry off before anything is imported export DO_NOT_TRACK=1
