The agent shipped. Demo was clean. Two weeks later a customer flagged that it kept “getting the right answer the slow way” — and when I pulled the trace, there it was: to answer one billing question, the agent called the search tool three times with slightly wrong parameters, ignored the first two results, and stumbled into the correct number on the third try. The final string was perfect. The trajectory was broken. And every test we had — all of them graded that final string — was green.
That gap is the whole reason this post exists. In 2026 we are shipping tool-calling, multi-step agents far faster than we can measure them, and the tooling market has responded with a wall of frameworks: DeepEval at 16.8k stars, Langfuse at 31k, Phoenix at 10.5k, Ragas at 14.8k, OpenAI Evals at 18.9k (all as of 2026-07-13). Faced with that, most teams do the natural thing — sort by stars and pick the top result. That is the wrong sort key.
Here is the argument in one line: you don’t pick an agent-eval framework by GitHub stars — you pick by the evaluation layer you need to test. Evaluate the trajectory, not just the final answer, and do it continuously across the lifecycle. This piece gives you a five-bucket taxonomy to make that call in minutes, a walkthrough of one representative tool per bucket with real numbers, and a lifecycle loop so your agent doesn’t glitch in production later.
I’m writing this from what I’ve seen shipping agents with teams, not from a vendor slide. Every framework below gets the same treatment and the same skepticism.
Why agent evaluation is a different problem now
A single-turn LLM call can be graded on its final string. You send a prompt, you get text, you check the text. An agent is not that. An agent reasons over multiple turns, calls tools, reads the tool results, and uses them to decide what to do next. That one difference — tool use inside a loop — breaks output-only testing.
Consider what “the answer is right” actually hides. Two runs of the same agent can produce an identical final answer while taking completely different paths: one picks the correct tool on the first try; the other selects the wrong tool, passes a malformed parameter, retries, and lands on the answer by luck. Output-only eval scores both as PASS. Then you swap the model or tweak the system prompt, the lucky path stops being lucky, and the “silent regression” you never tested for shows up as a production incident.
Microsoft Foundry states the requirement about as plainly as it can be stated: “you need to evaluate not just the final output, but also the quality and efficiency of each step in the workflow” (Foundry agent evaluators). That is the shift. The failures that matter in agents live at the step and tool level:
- Tool misuse — the agent calls a tool it shouldn’t, or skips one it should.
- Wrong parameters — right tool, malformed input, wrong result.
- Redundant steps — three tool calls where one would do, inflating latency and cost.
- Silent regressions — a prompt or model change quietly degrades the path while the final answer stays plausible.
None of those show up if you only diff the last message. So the frameworks worth knowing in 2026 differ mainly by which layer they let you measure — and that is exactly the axis you should be shopping on.
The mental model: five variant types
Instead of a ranked list of tools, hold this taxonomy in your head. There are five variant types of agent-eval framework, and each one is built to test a different layer.
| # | Variant type | What it measures | When you reach for it | Representative tools |
|---|---|---|---|---|
| 1 | Trajectory / process evaluators | Step-by-step: tool selection, tool-call accuracy, task adherence, path efficiency | You need to know why an agent failed, not just that it did | Microsoft Foundry agent evaluators; DeepEval agentic metrics |
| 2 | Open-source eval libraries | Metric functions (LLM-as-judge + statistical) wired into unit tests / CI | You want assertions that fail a PR when quality drops | DeepEval, Ragas |
| 3 | Observability + eval platforms | Live traces + evals + datasets over real production traffic | You need to debug and monitor an agent already serving users | Langfuse, Arize Phoenix |
| 4 | Capability benchmarks | Standardized tasks + leaderboards comparing model/agent capability | You’re choosing a base model or reporting comparative capability | τ-bench / τ³-bench |
| 5 | Provider-native / cloud eval | Managed evaluators inside a cloud/model platform, CI gates + monitoring | You’re already on that platform and want batteries-included eval | Microsoft Foundry; OpenAI Evals (caveated — see §5) |
Two honest notes about the table, because the overlaps are the point rather than a bug. DeepEval appears in two rows: it is a metric library and it does trajectory metrics. Foundry spans trajectory and provider-native. The taxonomy sorts by a tool’s primary job, and the thesis — pick by the layer you need — is precisely what lets you live with a tool that spans two rows. You’re not asking “which bucket does this tool belong to,” you’re asking “which layer do I need to test right now,” then reaching for whatever covers it.
The rest of this post walks one representative per layer, with concrete facts, so the buckets stop being abstract.
A walkthrough, one representative per layer
Trajectory / process — Microsoft Foundry agent evaluators
If your question is why did the agent fail, you want process evaluators. Foundry’s agent evaluators are grouped into three categories (agent evaluators, updated 2026-06-02): system evaluation (end-to-end outcome — Task Completion, Task Adherence, Intent Resolution, Task Navigation Efficiency, a 1–5 Customer Satisfaction score); process evaluation (per-step tool use — Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success); and a preview quality grader.
The design detail I like: these “act like unit tests for agentic systems,” taking agent messages and returning a binary Pass/Fail (some via a 1–5 score thresholded to pass/fail). And Task Navigation Efficiency compares the agent’s actual tool trajectory to a ground-truth sequence — exact_match, in_order_match, or any_order_match — returning precision, recall, and F1. That is a deterministic trajectory check, not an LLM opinion, which matters for the judge-reliability debate below. It’s LLM-as-judge where it needs to be (it recommends gpt-5-mini as the judge model), deterministic where it can be.
Open-source library — DeepEval
If your question is will this fail my PR when quality drops, you want an eval library wired into CI. DeepEval bills itself as “Pytest for LLM apps” — deepeval test run fails the suite on a threshold breach and drops into any CI/CD (confident-ai/deepeval, v4.1.0, Apache-2.0, 16.8k stars as of 2026-07-13). Its metric families span G-Eval (research-backed LLM-as-judge on custom criteria), DAG (a deterministic, graph-based judge), plus dedicated agentic, RAG, multi-turn, MCP, and multimodal metrics, with end-to-end and component-level tracing via @observe. It runs metrics against any LLM or local models and integrates with OpenAI Agents, LangChain, LangGraph, CrewAI, LlamaIndex, Google ADK and more. The paid Confident AI platform is optional on top — the library itself runs anywhere.
Open-source library — Ragas (with an honesty flag)
Ragas belongs in the same bucket but earns a caveat that actually helps you choose. It’s an objective-metrics library (LLM-based and traditional) built for RAG apps, and its standout feature is production-aligned test-set generation — genuinely useful when you “don’t have a test dataset ready” (explodinggradients/ragas, now vibrantlabsai/ragas, v0.4.3, Apache-2.0, 14.8k stars as of 2026-07-13). But look at the CLI templates: rag_eval ships today, while agent_evals, benchmark_llm, prompt_evals, and workflow_eval are marked “Coming Soon.” Ragas is battle-tested for RAG and emerging for agents. If you’re doing RAG, it’s excellent. If you need trajectory eval today, reaching for a RAG-mature library because it’s popular is exactly the star-driven mistake this post is warning against.
Observability + eval platform — Langfuse and Phoenix
If your agent is already serving users and your question is what actually happened in production, you want an observability platform — tracing plus evals plus datasets over real traffic. Two strong open-source options, and the difference between them is instructive.
Langfuse is an “open-source LLM engineering platform” — tracing (@observe, OpenTelemetry), versioned prompt management, evaluations (LLM-as-judge, code evaluators, user feedback, manual labeling, custom pipelines), datasets, and a playground. It self-hosts in minutes via Docker Compose, Kubernetes/Helm, or Terraform for AWS/Azure/GCP, or runs as Langfuse Cloud (langfuse/langfuse, v3.212.0, 31k stars as of 2026-07-13, MIT except the ee/ directory; part of ClickHouse since Jan 2026).
Arize Phoenix is an “open-source AI observability platform for experimentation, evaluation, and troubleshooting” — tracing via OpenTelemetry/OpenInference, response and retrieval evals, versioned datasets, experiments, a playground, and prompt management. It’s explicitly vendor/language/framework agnostic and runs local, in a notebook, in a container, or in the cloud (Arize-ai/phoenix, v17.28.0, 10.5k stars as of 2026-07-13, Elastic License 2.0).
The buying signal here isn’t star count — it’s the license and hosting fine print. Langfuse is MIT (with a carved-out enterprise directory); Phoenix is Elastic License 2.0. If portability and license terms drive your infra decisions, that difference matters more than any feature-list bullet.
Capability benchmark — τ-bench / τ³-bench
If your question is which base model should I even start with, you want a benchmark — standardized tasks and a leaderboard. τ-bench emulates dynamic user↔agent conversations with domain API tools and policy guidelines, and its key idea is reporting pass^k: success across k repeated trials, which exposes reliability instead of single-run luck (sierra-research/tau-bench, arXiv 2406.12045; superseded by τ³-bench, arXiv 2506.07982, which adds banking and a voice modality).
The numbers make the reliability point vividly. On τ-bench retail (tool-calling), claude-3-5-sonnet-20241022 scores 0.692 / 0.576 / 0.509 / 0.462 across pass^1 through pass^4; gpt-4o drops from 0.604 to 0.383 (as of 2026-07-13). The single-shot number looks fine; consistency erodes fast when you run the same task four times.
Two caveats to keep you honest: the τ-bench repo now warns its tasks are outdated, so cite τ³-bench for current figures — and, critically, a high leaderboard score tells you a model is capable, not that your agent works on your tools with your data. Benchmarks pick a base model. They do not decide whether you ship.
Provider-native — Foundry, and the OpenAI Evals contrast
If you’re already on a cloud or model platform, the batteries-included option is provider-native eval. Foundry (covered above) is the full-agent example. OpenAI Evals is the one I want to caveat carefully, because it’s popular enough (18.9k stars, MIT, as of 2026-07-13) that people assume it’s an agent evaluator. It isn’t, quite.
OpenAI Evals is “a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks” (openai/evals). You write custom evals with JSON data plus YAML params — no eval code needed for basic or model-graded evals — and its Completion Function Protocol can target a tool-using agent. But it grades the final output, not tool-call accuracy, tool selection, or trajectory efficiency, and it’s been less actively developed lately. So treat it as a caveated contrast, not a peer trajectory evaluator. It still earns a mention for three narrow reasons: it’s the provider-native choice if you’re already in the OpenAI ecosystem (runnable right in the OpenAI Dashboard); it gives the provider-native bucket a second, non-Microsoft example; and it’s the honest “general LLM-eval retrofitted for agents” case. If you want OpenAI-native trajectory eval, the stronger path is the OpenAI Agents SDK’s tracing and evals, not the Evals registry.
How to eval so it doesn’t glitch in production
Picking the right layer is half the job. The other half is when you run evals — and the answer is “continuously, on both sides of the ship line.” Think of it as a loop: shift-left before production, monitor-right after.
Pre-production (shift left). Before an agent goes near a user:
- Build a golden dataset. Real or synthetic representative tasks with expected outcomes and expected tool trajectories. If you don’t have one, this is exactly where Ragas-style test-set generation earns its keep.
- Grade outcome and trajectory. Run system-level pass/fail and per-step tool checks. This is the whole thesis operationalized — never grade only the final string.
- Gate CI. Wire the suite so a threshold breach fails the PR (DeepEval’s
deepeval test runis built for this). A regression that can’t merge can’t ship. - Add deterministic checks beside the judge. Where you have ground truth, use it — Foundry’s Task Navigation Efficiency (precision/recall/F1) or DeepEval’s DAG — so you aren’t fully dependent on an LLM’s opinion.
- Run pass^k. Execute key tasks k times and watch for reliability decay, the way τ-bench does. A task that passes once and fails twice is not shippable.
- Red-team before release. Adversarially probe for jailbreaks and unsafe tool use — Foundry ships an AI red-teaming agent built on Microsoft PyRIT for exactly this.
Post-production (monitor right). Once it’s live:
- Trace everything. Capture every step and tool call — OpenTelemetry/OpenInference is the common standard across Foundry, Langfuse, and Phoenix, so you’re not locked in.
- Run online eval. Score a sample of live traffic continuously, not just your offline dataset.
- Alert on drift. A model update, a prompt tweak, or a changed tool API can silently degrade the trajectory. Catch it with monitoring, not a customer email.
The payoff is direct: shift-left catches regressions before they cost you an incident; monitor-right catches the drift that offline tests can never anticipate. Skip either half and the “works in demo, breaks in prod” cycle continues. The one-line rule: evaluate the trajectory, in CI and in production.
Three debates worth settling for yourself
Leaderboard score vs. your own data. τ-bench gives you comparable capability numbers, and they’re genuinely useful for base-model selection. But a high leaderboard score says nothing about your agent on your tools — which is why Foundry, DeepEval, and Ragas all push “bring your own data.” Use benchmarks to pick a model; use your-data evals to decide whether to ship.
LLM-as-judge vs. deterministic checks. Nearly every framework here leans on an LLM judge (Foundry’s Quality Grader, DeepEval’s G-Eval, Langfuse, Phoenix, Ragas). It’s scalable, but it adds a second model’s bias and variance to your measurements. The counter isn’t to abandon judges — it’s to pair them with deterministic checks where ground truth exists: DeepEval’s DAG, Foundry’s precision/recall/F1 trajectory match. Trust, but verify.
Open-source library vs. managed platform. DeepEval and Ragas (Apache-2.0, run anywhere) trade convenience for control and portability. Foundry and OpenAI (managed, batteries-included) trade portability for support and integration. Langfuse and Phoenix are open-source platforms that split the difference — self-host or cloud — but on different licenses (MIT vs. Elastic License 2.0). There’s no universally right answer; there’s the answer that fits your portability, license, and ops constraints.
The decision rule
Here’s the whole post compressed into a single question tree. Start with what you’re deciding, and the layer picks itself.
- Choosing a base model? → capability benchmark → τ³-bench.
- Catching PR regressions before merge? → open-source eval library → DeepEval or Ragas.
- Need to know why a step failed? → trajectory/process evaluator → Foundry agent evaluators (or DeepEval agentic metrics).
- Debugging or monitoring live traffic? → observability + eval platform → Langfuse or Phoenix.
- Already standardized on a cloud/model platform? → provider-native eval → Foundry (or OpenAI Evals, with the trajectory caveat).
Notice what dropped out of that list: star counts. Not once did “which has the most GitHub stars” help you answer the question. The layer did all the work.
What to do this week
Pick one agent you already have running and instrument it through both outcome and trajectory eval — not someday, this week. Start where your pain actually lives: if silent regressions keep slipping through, stand up an offline CI gate first (grab a library from bucket 2 and fail a PR on a threshold breach). If it’s production breakages that hurt, start with tracing and online eval from an observability platform (bucket 3). Choose one tool from that one bucket and run it end to end.
You’ll learn more from instrumenting a single real agent through one layer than from another afternoon comparing star counts. The frameworks in this post are all good at something — the skill worth building in 2026 isn’t picking the “best” one, it’s knowing which layer you’re testing and reaching for the tool that measures it.
Frameworks, versions, star counts, and leaderboard scores verified 2026-07-13 against their primary sources; all of these drift, so re-check before you rely on a specific number.