Skip to main content

Lessons learned from one million evaluations

MA

Mac Merritt

Engineering

Evals for a coding agent usually start as a fixed set of tasks scored with a binary pass rate and selected based on what is publicly available. That can work in some cases, but several issues can arise that make this evaluation approach unhelpful. First, as agents and models improve, the pass rate on the fixed set rises, and often the remaining pass rate gaps are due to noise in the benchmark as much as agent and model performance. We call this saturation of a benchmark, and when it happens, the benchmark no longer measures the intended signal. We saw this recently when we added a new model to Kiro. On Terminal Bench 2.1, the pass rate matched our expectations. We knew that Terminal Bench 2.1 was approaching saturation, but assumed that its pass rate would still measure large regression risks. Then we started to hear anecdotes from internal test users that the experience with the new model was worse than with other models. When we took a closer look, we saw that despite the reasonable pass rate, there were tools failing on 40% of their calls because the tool description that had worked for other models caused the new model to call the tools with invalid inputs.

That relates to the second issue: pass/fail signals may not be sufficient. This result didn't affect pass rate because the model was able to iterate until it found the correct way to submit the request, but during that trial and error, tokens were used, latency went up, and the overall customer value was reduced. Customer experience depends on more than just whether Kiro could get the job done, which connects to the third issue: evaluations may not represent realistic customer experience. Customers need confidence that Kiro can get the job done quickly and at a high quality bar. To measure this, we need to verify that we run benchmarks reflecting real customer experience and track all the signals that could affect that experience. In Kiro we have run millions of evaluations as we work on improving the product for our customers, and during these evaluations we have encountered each of these issues.

This post is about what we have learned and how we now manage Kiro's evaluation suite to measure what matters to our customers.

The problem

Evaluation is essential for agent development. While we can reason about how a model might respond to a change, we cannot reliably predict it. A small change to a prompt, a tool description, or the model itself can shift behavior in ways nobody expected, and the only way to know for sure is to run it against a real model. This is why even highly experienced engineers and scientists often disagree about what will improve or hurt agent performance: for example, when and how to use sub-agents, what belongs in an AGENTS.md file, how many tools is too many, what kinds of memory are helpful or harmful, and whether to lean more on skills or custom MCP tools. This means we need an evaluation with trustworthy results that we can use to decide which trade-offs will help customers most.

Many evaluations are a fixed set of pass/fail scenarios, chosen because of availability and popularity, not necessarily because they reflect customer value clearly. This can cause three problems. First, the fixed benchmarks can become saturated. In this case, the agent can still solve the benchmark even when there is a regression, which makes it hard to find meaningful signal about what should be improved. Second, the coarse pass/fail grading can hide signals that matter to customers because it does not explain how the agent solved the task. Customers care about their cost, how long Kiro takes to perform the task, and the quality of the code beyond just whether it passed the testing requirements. A pass signal can hide all of these details. Third, the benchmark itself may fail to measure signals that matter to our customers. Terminal Bench measures an agent's ability to complete incredibly complex tasks, but it underemphasizes signals that are not part of its target use case and are thus underrepresented, such as different coding languages, DevOps use cases, and code understanding. These use cases are not core to Terminal Bench, but they are important to Kiro customers.

The approach

Maturing our evals meant building mechanisms to measure and improve how well evals reflect customer use cases.

  1. From static sets to dynamic benchmarks. Our goal is a benchmark suite whose scenarios match how developers use Kiro. One of the ways we do this is by measuring usage from internal Amazon users of Kiro through aggregated categories. We group usage by 20+ task categories, 25+ domains, programming language, and number of prompts per session, and we label our benchmark scenarios with the same categories. We then fill those categories with scenarios from three sources: scenarios we write ourselves, scenarios from other Amazon teams, and public benchmarks. When the suite has too few scenarios in a category compared with actual usage, we add more, and those gaps help us prioritize which benchmarks to focus on. The result is a versioned benchmarking suite that we update on a regular cadence as usage and capabilities evolve.
  2. From pass/fail to graded judgment. We track three layers of granularity with every benchmark. Pass/fail is the first metric we look at, and it serves as the composite indication of performance. Next we track output signals from each benchmark run that we know matter to our customers. The simplest form of this is latency and token usage, but in some cases it may include measures of code complexity or the size of artifacts generated. Finally, we look at LLM-as-judge ratings of the benchmark trajectories for signals that are difficult to measure deterministically but still matter to customers, such as code quality and response tone. When we find opportunities for improvement or regression in any of these areas, we review detailed trajectories from the benchmarks to understand why the metric is worse than expected and what options we have to improve Kiro. One of the biggest values of benchmarks is that we can then re-test improvement options to decide which one is most effective for the opportunity we are targeting.
  3. Building trust with live signals. The signals we use in moving from static to dynamic benchmarks help bring benchmarks closer to customer use cases, but they can still miss customer pain points for reasons outside the measured dimensions. A benchmark may match a customer in category, domain, language, and horizon, and still not reflect their experience because the benchmark is saturated, or our categories were not fine-grained enough, or any number of other reasons connected to the scenarios themselves. In addition to which scenarios we run, there is a risk of missing customer pain points because of how we run the benchmarks. Model selection, thinking level, tool selection, client configuration, and more can all affect agent quality and may be configured by customers in a way we might miss in evaluations. Our selection process helps us focus on benchmarks that are likely to represent a customer's experience, but ultimately real customer experience is the source of truth. Jeff Bezos once said, “When the data and the anecdotes disagree, the anecdotes are usually right.” This is the sentiment behind building trust with live signals. We use data from surveys and aggregated satisfaction ratings, and anecdotes from discussions with users. We also use surveys and conversations with external customers to check that internal users have a similar experience. This lets us consistently evaluate the evaluation system itself and ensure the data agrees with signals from real customers.
Flowchart diagram showing Kiro's evaluation cycle in 4 steps: Users provide feedback, Live signals collect data through surveys and ratings, Offline benchmarks measure performance in three layers, and Kiro improvements fix issues based on findings.

A real measurement

After we learned our lesson with that model launch, we applied this more systematic approach. We measured Kiro across scenarios from many benchmarks, including several private benchmarks to avoid contamination risk, Terminal Bench 4.0, Deep SWE, SWE Marathon, and more. We also paid attention to detailed metrics around tool failure rates, token efficiency, and latency. The data on tool failure rate helped us fix the inconsistent tool description problem we mentioned earlier, and the data on efficiency and the more complete set of evals have led to dozens of changes that improved features, fixed bugs in the Kiro harnesses, and improved efficiency in a way that directly translates to customer savings.

Lessons and future research

Even with a suite matched to usage categories, getting the right scenarios into it is difficult. Signals that look like success can still be misleading. For example, we recently considered an internal pull request as a benchmark scenario. It was merged and the user gave the agent 5 stars, so it looked like an ideal example of a successful task. When we checked the repository history, the next commit fixed bugs that the pull request had introduced. If we had added it, the benchmark would have treated buggy code as the correct answer. Our future research focuses on scaling benchmark curation so we can keep updating the suite while catching scenarios like that reverted 5-star pull request.

Our current process is also limited by how Kiro is configured during evaluations. We run evals to match customer usage for model selection and Kiro client. Future research will investigate how Kiro performs with complex customer configurations such as MCP servers, steering files, verbose skills, and heavily commented codebases.

Try it and further reading

Try Kiro. For the detailed method behind this, including the LLM-judge rubric, the A/B setup, and measured results, read Continuous Prompt Evaluation, and for the bigger question of whether agents are truly improving over time, read Are AI coding agents actually getting better?.