Build the Environment, Then Optimize the Harness

In July 2026, alongside its Series B announcement, InstaLILY published a two-part research report, Build the Environment, Then Optimize the Harness, on how we evaluate and tune Lily, the company's coding agent. The report has no byline and was a team effort. I led the evaluation work in Part 1 and helped synthesize the final draft. This page is my short version of the argument and of what Part 2 found.

What I worked on

I led the design of Lily-Eval, which I built with a few colleagues; my main piece was the package that turns a repository's merged pull requests into graded tasks. Part 1 of the report is that system and its head-to-head results, and it has its own write-up in Lily-Eval.

The argument

The usual way to improve a coding agent is to tune it directly: edit the system prompt, add a tool, run a handful of tasks, and keep whatever looks better. The report argues for a different order. First build an environment that reproduces your own engineering work and can be reset and replayed. Then measure how much its scores move when nothing has changed. Only after that should an optimizer touch the harness.

Our environment comes from merged pull requests, each one a task whose answer key already exists, so any task can be replayed as often as an optimizer needs. The report says it in two sentences:

A benchmark tells you where you are. An environment lets you climb.

Part 2 freezes Lily's model and searches everything around it, including prompts, context, tools, review loops and stopping rules.

What Part 2 found

Part 2 was built by the wider team on top of that environment, and it found three things.

Scores are noisy. Two runs of the identical agent differ by about ±0.05 on the full 46-task suite and about ±0.11 on a 14-task slice. A system-prompt candidate that gained +0.14 on that slice scored below baseline on an independent re-run. Without a rule that a gain must survive a re-run it was not selected on, it would have looked like a win. The same audit downgraded our own head-to-head (Lily ahead on behavioral pass@3, Claude Code ahead on mean reward) to "leans, not leads".

The only changes to survive were code in the harness, not prompt text. Fifteen candidates went through the gates and two were promoted, the stronger one a review hook that makes the agent show evidence before it finishes.

The gain carried to a held-out codebase the search never touched: 11 of 20 tasks solved against 8 of 20, still on the gpt-5.5 exploration model rather than Lily's deployment model. On eleven of the twenty held-out tasks the score didn't move under the new harness, which fits the report's view of where a wrapper stops:

The model chooses your ceiling; the harness decides how close to it you operate.

What I took from it

An optimizer will find whatever the scoring rewards, including noise. If the team had pointed it at the suite before measuring the null band and requiring re-runs, the +0.14 prompt candidate would have shipped. With those rules in place, thirteen rejections out of fifteen read as the gates doing their job.

If another team asked me where to start with agent evals, I would tell them to spend the first weeks on the tasks and the error bars before any tuning. Confirm each task separates a do-nothing run from the real fix, run the same agent several times to see how far its scores drift, and decide in advance what a gain has to survive before you believe it.