Lily-Eval: turning a codebase's merged PRs into a resettable environment for a coding agent

Lily is InstaLILY's coding agent. A small team of the company's most senior engineers built it on LangChain Deep Agents, and I worked on it with them over the summer of 2026. Part of my work was on the harness itself: extensions for subagent and multi-agent delegation, so Lily could split complex tasks the way InstaLILY's work needs. The larger part was a question every change to that harness raised. Did this make Lily better on our code, or did it only feel better?

Lily-Eval answers that question. It rewinds a codebase to just before each of its merged pull requests, asks an agent to redo the change blind, and grades the result against what actually shipped. I led the design of Lily-Eval, which I built with a few colleagues; my main piece was the package that turns a repository's merged pull requests into graded tasks. The resulting private, resettable suite measures how well an agent can understand, modify and work inside that specific codebase.

InstaLILY published the method and the results in a two-part research report, Build the Environment, Then Optimize the Harness. The report has no byline. I led the evaluation work in Part 1 and helped synthesize the final draft. Part 2, the optimizer that climbs the environment, was the wider team's work. I summarize it briefly near the end, and my take on it is a separate post.

The question public benchmarks can't answer

Public benchmarks like SWE-bench test a generic agent on generic open-source repositories. That number is useful, but a company deploying a coding agent has a narrower question: can this agent ship code that meets our bar, on our codebase, under our platform's rules?

Answering that needs ground truth. Diffing the agent's output against a reference measures mimicry, and asking a model "is this good?" gives a number nobody can defend. Ramp and Harvey have each built private benchmarks from their own work, and both stopped at measurement. We wanted something an optimizer could climb, which raises the bar on how much the score can be trusted.

InstaLILY had one structural advantage: it builds on a shared platform template that is branched into a family of production codebases. That gives the environment a built-in generalization test: hold one codebase out of tuning entirely and evaluate on it cold. A gain that carries over means the agent learned the conventions the branches share, not one repository's quirks.

Every merged PR is a graded exam

The core idea is that every merged pull request is an exam we already hold the answer key to. The change shipped, so the before and after give ground truth for free. We turned that into what the report calls the rollback contract. For each case:

1. Rewind the repository to the commit just before a real, shipped change.
2. Describe the problem in plain English, the way a coworker or end user would: business language, no file paths, no hints about how the original fix worked.
3. Let the agent implement blind. It never sees the reference solution or the grading tests.
4. Score against the shipped version as the answer key.

Because the answer key is free, the environment is also resettable. The same real change can be replayed as many times as needed with the right answer always known, which is what later lets an optimizer train against the suite instead of taking it once.

Curation: most PRs don't make the cut

The tempting move is to mine every merged PR and call it a dataset. Curation goes the other way. One invariant governs everything: a case is valid only if its gold behavior fails before the change and passes after it. Before any test enters the suite, it has to show that a do-nothing agent scores zero and the real shipped fix scores one.

The subtlest filter is about what kind of failure counts. The "before" failure has to be a behavioral failure. A test that fails only because a function doesn't exist yet would reward an agent for creating an empty function with the right name. Beyond that, the gates reject mega-merges with no single intent, umbrella UI changes too broad to attribute, and low-value hygiene PRs. Every rejection is recorded with its reason. Eight automated gates enforce these rules on every case.

A surviving PR becomes one record with two payloads that pull in opposite directions on purpose. The request the agent sees is vague about implementation and written in business language. The validation assertions are exact about behavior, and the agent never sees them. That separation makes a task realistic on one side and gradeable on the other.

Each record compiles into a sandboxed task in Harbor, the Terminal-Bench team's task format: an instruction file, an environment built at the PR's base commit, a verifier that writes the reward, and the shipped fix kept aside as an oracle that only proves the task is solvable.

A score you can trust

Real product changes aren't graded by unit tests alone, so the tasks aren't either. Every check in the suite is one of three kinds, trusted in proportion to how hard it is to fool.

Tier	Asks	Graded by
Outcome	Does the feature behave correctly?	Deterministic code. The floor; it can zero a case on its own.
Conformance	Does the change follow the platform's rules?	Deterministic structural checks. Adjusts the score, never gates alone.
Judged	Is it well-crafted code a senior engineer would approve?	An LLM judge panel on a fixed rubric. Reported, never trusted alone.

The judges are kept apart from the work they grade. The one that writes functional tests sees only the plain-English request, never the agent's code, and the one that scores craft never runs tests. The report covers the rest of the grading pipeline. An optimizer will exploit any soft spot in its reward, so if a persuasive diff could talk a judge into a high score, a harness search would learn to write persuasive diffs.

One run is a random variable

The same agent, on the same task, at the same settings, does not reliably produce the same verdict. Across our head-to-head runs, 26% of task verdicts flipped between attempts of the same agent, with no infrastructure failures involved. That is the true variance of what is being measured, and we report it instead of averaging it away. It also means a gap between two agents means nothing until it beats what this variance produces on its own.

Lily vs. Claude Code

The first real use of the suite was a head-to-head between Lily and Claude Code, both running the same model (claude-opus-4-8 at maximum effort). Three production codebases, 46 tasks, 3 attempts each, 276 trials. Neither agent scored a perfect 1.0 on any single trial.

Metric	Lily	Claude Code
Mean reward	0.51	0.55
Behavioral outcome, pass@1	56%	49%
Behavioral outcome, pass@3	71%	58%
Tasks won on behavioral pass	11	7
Median time per attempt	9.0 min	7.6 min

The two headline measures disagree. Lily leads on the deterministic outcome bar at every k and wins more tasks outright. Claude Code leads on mean reward and on judged craft, and does it faster and leaner, with about 15% fewer input tokens. Where the two conflict, we treat the behavioral bar as the more authoritative one, because it is the deterministic floor. The report publishes both, so a reader can see where each agent wins. Whether either gap beats the run-to-run variance above is the question Part 2 opens with.

The anatomy of a failure

Aggregates say who is ahead; transcripts say why. We read every task that failed every attempt and sorted the failures into families, with the rubric fixed before reading. The report counts 22 such tasks, using a stricter bar than behavioral pass.

Family	Tasks (of 22)	What it looks like
Contract precision	11	The logic is right but a graded detail is wrong, like inventing a new field name instead of reusing the one the codebase already has.
Hard tasks	6	The task needs domain reasoning or a source of truth the agent doesn't reliably have.
Scoping	4	Right change, wrong place, or a change that balloons past the request.
Verification theater	1	The agent "verifies" a happy path that never touches the graded branch, then finishes wrong.

In most of these failures the model could do the work. It lost on a habit it never applied: check the codebase before inventing a name, run the code before claiming it works. A missing habit is something the harness around a model can enforce.

From benchmark to environment

Going in, we assumed the dataset was the valuable artifact. Partway through, a few of us realized the durable thing was the machinery that turns an organization's repositories and shipped history into a replica of how it builds software. Anyone can accumulate tasks. The replica has three properties that a pile of tasks lacks: it is resettable, it spans several codebases by construction, and it is versioned with the code it models. That is what lets an optimizer climb it instead of taking it once.

The climbing idea came from the meta-harness line of research. The Meta-Harness paper optimizes the scaffold around a frozen model instead of the model itself, and InstaLILY's version follows the same recipe: Lily's model stays fixed, and everything around it (prompts, context, tools, review loops, stopping rules) is open to search. Lily-Eval supplies the reward. That makes it an RL environment for tuning the harness to forward-deployed-engineering work and InstaLILY's own use cases, which is the job it ended up doing.

Part 2 of the report ran that search, and the stronger of the two harness changes it promoted was a review hook that makes the agent show evidence before it finishes, aimed at the contract-precision and verification-theater families above. The numbers and my take on them are in Build the Environment, Then Optimize the Harness.

What stays private and what doesn't

The tasks stay private, both because the codebases are proprietary and because published tasks invite contamination. The report says InstaLILY plans to release a package that builds this kind of environment for any organization: point it at your repositories and it mines your merged PRs, applies the curation gates, and compiles a sealed, resettable suite with the error bars built in. Generalizing the pipeline beyond InstaLILY's own platform is where I spent the end of the summer.

What's next

Next I want to point this at a codebase outside InstaLILY's template family and see which gates still hold. The held-out test above stays inside one family of branches, so it shows an agent can learn conventions those branches share. Whether the curation rules and the three grading tiers carry over to a repository with a different stack and different review habits is the open question I most want answered.