The best open-source LLM evaluation tools in 2026
The best open-source LLM evaluation tools right now are promptfoo, Inspect and LM Evaluation Harness, out of 18 we have reviewed. Each is ranked by our rating, with its price, our verdict and who it suits.
By ReviewRamble editors · Updated Nov 18, 2026 · How we review
How we ranked: by our rating out of 5, from reviews built on each product's own pricing, documentation and policies plus independent benchmarks where they exist, each checked on the date shown in the review. We take no affiliate commission.
- 1
promptfoo turns LLM regression tests and red-team scans into a YAML file and one CI command; its new owner is the part to watch.
Best for: Teams shipping LLM apps who want regression tests and security scans in CI.
Price: CLI free and open source (MIT); Community use includes red teaming up to 10,000 probes a month. Enterprise (managed cloud) and On-Premise are priced by quote.
Read our promptfoo review4.5/5 - 2
Inspect is the shared harness of government and independent AI safety testing; product teams checking prompts in CI will find it more framework than they need.
Best for: Researchers and safety teams running agent and capability evaluations.
Price: Free and open source under the MIT license.
Read our Inspect review4.4/5 - 3
The common yardstick for model benchmarks, and the wrong instrument for judging an application built on top of one.
Best for: Researchers and model trainers comparing models on standard benchmarks.
Price: Free and open source under the MIT license.
Read our LM Evaluation Harness review4.4/5 - 4
BrowserGym is the plumbing most web-agent research now runs on; adopt it for the shared interface and budget a day for each benchmark's servers.
Best for: Researchers and agent teams who need reproducible, comparable scores for browser agents across the standard web benchmarks.
Price: Free and open source (Apache 2.0); you pay for model calls and for hosting the benchmark sites.
Read our BrowserGym review4.2/5 - 5
A capable pytest-style eval library whose real cost is the judge model's bill, with a $200-a-month platform at the end of the funnel.
Best for: Python teams adding automated quality checks to LLM applications.
Price: Free and open source under the Apache 2.0 licence. Confident AI's hosted platform is a separate paid product.
Read our DeepEval review4.2/5 - 6
AgentDojo is the shared yardstick for prompt injection in tool-using agents, worth running before you ship a defence even though its public results have gone stale.
Best for: Security researchers and agent teams who need a recognised, reproducible number for how well a model or defence resists indirect prompt injection.
Price: Free and open source (MIT); you pay for the model API calls a run makes.
Read our AgentDojo review4.1/5 - 7
e2e is the first agentic test framework that treats model calls as something to remove, and its replay cache is what makes plain-English tests affordable in CI.
Best for: Product teams that want readable end-to-end tests across web and mobile without hand-maintaining every selector.
Price: Free under Apache-2.0; you pay for your own model. TesterArmy's hosted platform is separate: Free (5 runs), Hobby $99/month (250 runs), Startup $299/month (1,000 runs), Enterprise custom.
Read our e2e review4.0/5 - 8
Giskard puts multi-turn agent tests and an OWASP-mapped red-team scan in one Apache-2.0 library, but v3 is young and drops the tabular-model scan v2 users relied on.
Best for: Python teams that want agent scenario tests and a pre-launch red-team scan from one open-source library.
Price: Library free and open source (Apache-2.0). Enterprise platform (continuous red teaming, SSO, RBAC, on-premises or SaaS) priced on request.
Read our Giskard review4.0/5 - 9
Keploy is the quickest route from an untested backend to a replayable API suite with real dependency mocks, as long as you can grant it root on Linux.
Best for: Backend teams with little test coverage who want API and integration tests without building mocks or test databases.
Price: Open-source CLI free; Playground free (30 generated suites, 100 test runs a month); Pro $19/user/month including $19 of usage, then $0.16 per test generation, $0.22 per test run, $10 per 10,000 sandbox runs; Enterprise custom.
Read our Keploy review4.0/5 - 10
SkillEvaluator asks the question most skill scanners skip, whether a skill actually helps the agent, and answers it with paired live runs, at the price of a young, fast-moving tool and a model bill you size yourself.
Best for: Teams that publish agent skills for others and need evidence, in CI, that each skill is safe, distinct and actually improves agent results.
Price: Free and open source under Apache 2.0. Tier 1 needs no model; LLM checks and live runs bill at your model provider's rates, with free prototyping on NVIDIA Build for eligible Developer Program members, plus managed-sandbox charges if you use one.
Read our SkillEvaluator review4.0/5 - 11
The broadest open benchmark for asking whether a model is a security liability, best used as a pre-release gate for model choice rather than a test of your finished app.
Best for: Teams choosing or fine-tuning a model who need evidence on its security behaviour before it reaches production.
Price: Free under the MIT licence. You pay for the model calls, including the judge model, and for any test infrastructure such as AutoPatchBench's large build machine.
Read our CyberSecEval review3.9/5 - 12
Scenario is the clearest way to write multi-turn agent tests as normal unit tests, provided you accept that an LLM on both sides makes every run a little different and a little costly.
Best for: Teams building chat or voice agents who want conversation-level regression tests in their existing test runner.
Price: Free and open source (Apache-2.0); you pay for simulator and judge tokens. Optional LangWatch cloud: Developer €0 (50k events, 3 simulations); Growth €29 per core seat a month with 200k events, then €5 per 100k.
Read our LangWatch Scenario review3.9/5 - 13
A sound, OpenTelemetry-native eval library with a corporate owner behind it, held back by a small community and a pull towards Snowflake.
Best for: Python teams evaluating RAG apps or agents, especially those already storing data in Snowflake.
Price: Free and open source under the MIT licence. You pay for the judge-model calls each feedback function makes; Snowflake's hosted AI Observability is a separate product billed through Snowflake.
Read our TruLens review3.9/5 - 14
A well-documented way to test whether an agent stays inside its permissions, as long as you read the grade as a judge's opinion with error bars rather than a certificate.
Best for: Teams shipping agents that can act on money, accounts or customer data who want a repeatable permissions and injection audit in staging and CI.
Price: Free under Apache-2.0, self-hosted with your own model keys; you pay the judge provider's API calls. Paid audit tiers (Startup, Growth, Enterprise, Agentic Enterprise) are priced per agent per month, with prices not published.
Read our iFixAi review3.8/5 - 15
agent-qa is one of the more careful agentic test runners, with honest documentation and a useful memory, but its first GitHub release came only in October 2026, it sends telemetry by default and is not open source.
Best for: Small product teams that ship a web app plus native mobile apps and want plain-English regression tests they can run from a coding agent or CI on their own model.
Price: $0: no paid tier, seat fee or hosted platform. You pay for the model you connect (a Claude or ChatGPT subscription, an API key or a local model) and your own CI, browsers and devices.
Read our agent-qa review3.7/5 - 16
Calibrain Harness is an unusually honest way to ask whether a classifier has a trading edge, and its own numbers so far say the answer is no.
Best for: Quant researchers and model builders who want to test whether a fast classifier's probabilities carry an edge on crypto perpetuals without risking real money.
Price: Free and open source (MIT). Needs a Kairos (Minara Cloud) or Jev (TypeSafe) API key, billed by those providers, and free Bybit testnet keys.
Read our Calibrain Harness review3.4/5 - 17
Dokimi asks the question every skill pack and AGENTS.md should have to answer, and asks it carefully, but it is a month old with 11 stars and needs a patient engineer to run.
Best for: Engineers who maintain a skill pack, MCP doc server or large AGENTS.md and want evidence that it changes what an agent does.
Price: Free and open source under Apache-2.0. You pay for model calls: a triage analysis of one 19k-token skill was about $1.80 in the docs' example, and several checks need no model at all.
Read our Dokimi review3.4/5 - 18
Its metric names became the vocabulary of RAG evaluation, but the code has been left without a maintainer.
Best for: Teams with an existing Ragas suite who can pin a version and carry their own fixes.
Price: Free and open source under the Apache 2.0 licence. You pay for the judge-model API calls each metric makes.
Read our Ragas review3.4/5
Sources
The main official source behind each ranked review; every review lists its full set.
- promptfoo: promptfoo on GitHub
- Inspect: Inspect on GitHub
- LM Evaluation Harness: LM Evaluation Harness on GitHub
- BrowserGym: ServiceNow/BrowserGym on GitHub
- DeepEval: DeepEval on GitHub
- AgentDojo: ethz-spylab/agentdojo on GitHub
- e2e: tester-army/e2e on GitHub
- Giskard: Giskard-AI/giskard-oss on GitHub
- Keploy: keploy/keploy on GitHub
- SkillEvaluator: SkillEvaluator on GitHub
- CyberSecEval: CyberSecEval 4 README (PurpleLlama on GitHub)
- LangWatch Scenario: langwatch/scenario on GitHub
- TruLens: truera/trulens on GitHub
- iFixAi: ifixai-ai/iFixAi on GitHub
- agent-qa: vostride/agent-qa on GitHub
- Calibrain Harness: zadescoxp/calibrain-harness on GitHub
- Dokimi: Dokimi on GitHub
- Ragas: vibrantlabsai/ragas on GitHub
More guides
- Best open-source tools to run LLMs locally
- Best open-source AI agent frameworks
- Best open-source RAG tools
- Best open-source LLM fine-tuning tools
- Best open-source AI chat apps
- Best open-source AI coding agents
- Best open-source image and video AI tools
- Best open-source LLM gateways and observability tools
- Best Claude Code plugins and agent skills
- Best open-source tools for coding agents
- Best free courses for learning AI