How Kiro fixes itself, and how we keep the fixes honest

One sentence in Kiro’s file-read and search tool descriptions made a false claim about how the harness behaves, and it reached the model 61,742 times across a few hundred sessions. It did nothing useful, and it’s unlikely anyone would have filed a ticket for it. Problems like that are hard to find, and fixes are just as hard to judge. For example, we nearly shipped one that passed more tasks and used fewer tokens, until we split the results by host and found almost all of the gain came from a single run where the agent happened to escape a loop it usually gets stuck in. Without that host, the fix was slower, more expensive, and less accurate. A loop that grades its own fixes would have shipped it.

We built an internal tracing system to improve Kiro, our AI coding agent, from its own runs. This system, internally named kiro-trace, reads those runs to find where Kiro fails or wastes effort, proposes a change, and keeps the change only if it survives checks built to reject it. Those checks are also there to catch results like the earlier one, which looked like a win and was really just luck. Models can propose changes, but we use deterministic code to accept or reject them.

In its first week, kiro-trace opened eight pull requests to Kiro. Seven merged, one was closed for unrelated reasons, and none were sent back in code review. All seven changed Kiro’s prompts or tools, its tool descriptions, argument checks, and response behavior. Each one targeted a specific pattern we had measured, and each improved it. A few of these appear later in this post with their results. The rest never surfaced as an error a user could see, and no ticket pointed at them. A person could have found them, but only by reading far more runs than anyone has time for. This is how Kiro keeps self-improving.

The following numbers come from Kiro’s internal usage at Amazon, and each figure is a delta from an internal experiment, not a production estimate. A companion post, Continuous Prompt Evaluation, covers how we evaluate human-written prompt changes with LLM judges and live A/B tests. This one picks up after that, at automated trajectory mining, change generation, and harness self-improvement.

Kiro’s core modules took 141 bug-fix commits last month, close to five a day, and those are only the failures a human caught and had time to fix. The worst problems usually don’t show up as clean bug reports. They hide in how the agent recovers from an error, how it sequences tools, and how it holds a long context, and you only notice them after reading through thousands of real runs. The tool-description sentence at the top of this post is one example of that kind of problem.

So the job is to find the places where Kiro fails or wastes effort and change Kiro so it improves without seeing regressions in other places. That is broader than bug fixing. It covers misleading tool descriptions, tool calls rejected on a technicality, wasted calls and tokens, poor recovery after errors, and instructions that no longer help. Nobody can read thousands of runs by hand to find these, so the work has to be automatic. The obvious shortcut, letting the agent improve itself end to end, is where most self-improving loops go wrong. A loop that grades its own fix will ship narrow hacks and move on. To do this properly, kiro-trace has to solve four problems at once: 1) find a failure worth fixing, 2) write a change that targets its cause, 3) keep the solution general instead of tuned to the benchmark, and 4) prove the change improved the results. The self-grading loop skips step 4, and without it there’s no way to know whether steps 1 to 3 were worth doing.

The loop in that picture checks each change only against the problem it was aimed at, so a new change can break something an earlier one fixed without this loop noticing. That happens when nothing outside the loop checks for regressions. In kiro-trace, candidates are also run on tasks Kiro already passes, the final candidate is compared on the full benchmark against an unchanged control, and the pull request still goes through normal code review before it merges.

The diagram below shows one pass through the loop. Evidence from benchmarks and human-approved live-traffic complaints comes in on the left. The loop reads the failure, diagnoses it, picks one lever to change, and runs the same Kiro with the change off and on. A three-stage gate (screen, regression, confirmation) then decides. Changes that pass become a pull request for a human to merge, and rejected ones go back to diagnosis.

We always keep three roles separate, because anything that both writes a change and decides whether it worked will tend to accept it. We learned that partly from our own mistake. While judging comparisons by hand, we once read the same run three different ways, which is why the decision now sits in code. Kiro is the agent being improved. The loop proposes and applies a change. The gate, which is plain code, decides whether the change survives. Outside those three, a human makes the final merge call. Here is what we learned building each part.

Running a full benchmark every round is slow and expensive, and the tasks are not interchangeable. Some time out, some are really long-context tests, and some fail because of the eval harness rather than the agent. So the loop uses what it already knows about each task, its failure mode and its cost and whether it is even a real signal, to assemble a small set that covers the failure modes without the dead weight. It iterates against that small set, which is fast and cheap. The full benchmark runs only once, at the end, after a candidate has passed on the small set. We compare a full run with the change against an unchanged control run, and a change that wins on the small set but not on the full run does not ship.

A generic LLM judge can read any run and suggest a fix, but it does not know your agent loop or your inference layer. The kiro-trace reviewer reads Kiro’s real code-review comments and developer discussions, so it flags what this team would flag. The ideas it reviews come from evidence. When Kiro fails a task, the loop runs a panel of reference agents on the same task, with the same inputs and budget. Reference agents are other strong models and agents whose runs we compare with Kiro’s. When one succeeds where Kiro failed, that difference is a concrete behavior the loop can turn into a fix. A separate research step pulls in outside work, from papers to public code to how peer agents solve the same problem, for the general technique. It doesn’t copy their code. Every candidate records the failure it diagnosed and the evidence it cited, so any fix traces back to its reasoning. By default it tries removing something before adding a rule, because modern models already know a lot.

Running a fix a few times and reading the average won’t give you consistent results, because one outlier can swing it, as in the earlier example. So a change runs an escalating set of checks, starting with the cheapest, and most candidates fail before the expensive checks run. Unlike a plain A/B test, we measure the delta rather than the score, re-running the control every time, so a reported gain is the contribution of that specific change, and not something a new model would have produced anyway. And a change has to show it did not regress against a set margin, because at small samples a weak test finds nothing significant by default but can introduce real harm. Finally, a contamination check throws out any pass where the agent used the network to look up the task’s answer, for example by fetching its source or tests, instead of working it out.

Because weak candidates get cut early, the changes that reach a human reviewer are usually meaningful ones. Here are some of them. Each number is an internal experiment delta.

• A false claim about harness behavior on every file read and search. The model saw it 61,742 times across 445 samples and nothing acted on it, so we removed it.
• Guidance that told the model to call a tool that does not exist. We deleted it.
• Redundant file re-reads cut about 83 percent, by not re-reading a file that has not changed.
• Two over-strict or under-documented tool contracts fixed, clearing 205 read-tool schema rejections and 167 sub-agent orchestration rejections across 80 sampled sessions.
• Type-strict tool inputs stopped failing on a technicality. When the model returned a boolean as the string “true” instead of a raw boolean, the call was rejected and the work stalled. Now those values are corrected on retry, so a call with the right intent goes through.
• A silent file-search bug that broke roughly one in eight searches. A bad spec returned empty results with no error, so the agent searched and found nothing. Fixed.
• Over-eager delegation cut at the source, by removing the blanket delegate instructions that spun up sub-agents for simple direct edits, while keeping delegation where it helps.

Adding a stricter reference agent to the panel turned up something we weren’t looking for. A relative-path bug that failed 13–17 percent of that agent’s reads showed up on the model we ship as only 1.6 percent. The more tolerant model was absorbing a spec defect without error. Correcting the spec helps every model that shares the contract, and that fix is in review now.

The largest change has not shipped yet and is under team review. By removing aggressively, the loop built a smaller tool-and-prompt profile that scored 82.4 percent against 77.5 percent pass@1 (the share of tasks solved on the first attempt) on Terminal-Bench 2.1, at roughly a third of the token cost. It hasn’t shipped because it would change the default tools and prompt for every Kiro user, which is a product decision more than a benchmark one. The team is deciding whether to make the smaller profile the default, and which of the removed tools should come back only when a task needs them.

The fixes above are small and separate. Compaction is worth walking through from start to finish, because it shows the loop working on one problem from diagnosis to release. Compaction is how Kiro summarizes a long conversation when it reaches its context limit, so a fresh instance can continue. The diagram below shows how we picked the fix: 11 candidate summary prompts (two from Kiro, five from outside work, four of our own), tested head to head on whether a fresh agent could continue from the summary alone.

After a compaction, Kiro could abandon its current task and confidently resume a much older one. The old design appended a raw, newest-first dump of past user messages to the end of every summary, so the summary ended on the oldest question, right before the “resume from the summary above” cue. When the triggering message was a bare “yes, do that,” its context had been summarized away while the old thread survived word for word, so the agent tied the confirmation to the wrong request.

kiro-trace built a benchmark from real internal conversations that had actually hit compaction, in four size bands up to about 1.5 million tokens. It judged a summary by whether someone could pick up the work from it: it handed each candidate summary to a fresh model with nothing else and scored whether it could carry on. The score covered four things: was the summary actionable, were the file paths there, were earlier decisions stated, and did the thread hold together.

kiro-trace read how a range of production agents and public handoff practices compact, and rebuilt each as a prompt, 11 in all. Five came from outside our own code. They varied widely. Some were long multi-section templates, some were terse fixed ones, one was a situation-action-result narrative, and one was a step that decides when to compact.

The fix was to carry less context across a compaction. Before, Kiro kept the summary plus a raw list of old user messages, and that list was what sent the agent back to the wrong task, so reordering it would not have been enough. The outside prompts showed us the range of designs to compare against, but the idea to drop the list came from our own diagnosis of the failure. The fix removed it. A compaction now keeps a short summary, written to a terse fixed template and capped in size, plus the most recent turns. It ends with a static resume cue that names no task. A session can have several tasks in flight, and guessing one is what caused the wrong binding. Retention went from 0.82 to 0.92, and it shipped.

The loop keeps that same record for every candidate: the failure it diagnosed, the evidence and outside technique it used, the one thing it changed, and the result it measured. No part touches the task’s own answer.

kiro-trace keeps improving the loop alongside Kiro. It started from checks drawn out of our own prior research and code, then grew from mistakes. Each new check exists because the loop or the team was fooled once in some specific way, and most of those were mistakes in how we measured, not bugs in Kiro.

In one case, a correct fix scored 0 out of 5 because its output never reached the scorer. In another, a task’s setup gave the agent a way to undo its own mistakes, which hid a real bug. And a stale checkout once led us to open a pull request for something upstream had fixed the day before. Each of these became a check in the loop’s code.

Every confirmed fix also adds a deterministic assertion about its run to a permanent regression suite, so a failure fixed once does not return without being caught. Those checks come from one team today. The next step is to build them based on Amazon’s own resolved incidents and tickets, lessons the company has already paid for, so customers end up with a more reliable agent and a better developer experience.

kiro-trace finds problems in Kiro’s own runs, proposes one change at a time, and keeps a change only when deterministic checks and then a human reviewer accept it. In one week that produced seven merged fixes, and the compaction work raised retention from 0.82 to 0.92.

• Separate the proposer from the judge. Put accept and reject in code the model cannot reach.
• Decide on deltas, and re-run a control every time, because a new model alone can move the score.
• “No significant harm” is not the same as no harm, and small harms add up quickly across changes. Test against a margin.
• Turn every confirmed fix into a permanent regression check on the mechanism that failed, so it covers more than the one outcome.
• Try removing before adding. The safest wins are often removals, especially as models get better.

For Kiro users, that means fewer wasted calls and more careful delegation, and Kiro keeping better track of what it was doing in long conversations. No change from kiro-trace reaches users until it has passed the gate, with checks built to reject it, and then a human code review. We judge an agent experiment first by what it rejects. Most candidates never make it past the gate, and the ones that do are the changes that can meaningfully improve the developer experience.

You can try Kiro today.