HIVE: an eval-gated loop for improving agent harnesses

HIVE starts from a bet my teammate and I wrote into our course proposal. Frontier models are already capable enough, and what holds agents back is the harness around them: the prompts, the tools, the extensions, and the way work is split between agents. If that is true, the harness is the thing to optimize. HIVE is a loop that does it while the model stays frozen. An agent runs a graded suite of tasks. A headless Claude Code overseer reads the scores and the raw traces and proposes an edited harness. A trusted controller keeps the edit only if it beats the current best by more than a fixed margin.

It is a two-person project for Cornell CS 4701, built on ideas from the Meta-Harness paper and the Mismanaged Geniuses Hypothesis. The research question comes from the proposal: can an evaluator-guided optimizer improve the topology and coordination rules of a multi-agent coding system built on Pi, compared with single-agent Pi and fixed hierarchical, peer-to-peer and manager-worker baselines? The proposal is joint work. I wrote the founding spec and built the code in the public repo so far, with coding agents implementing modules against that spec. HIVE has no experiment results yet. The scaffold runs end to end, the dry runs pass, and a few cents of real model calls confirmed the plumbing. Nothing on this page is evidence that the loop improves a harness.

The loop

A search starts from a seed harness, usually the stock single-agent setup, scored on the search split with k trials per task. Each generation, the overseer gets a scratch workspace holding an editable copy of the incumbent, the metric, the suite description, a catalog of the valves it may change, the controller's actual promotion rule, and the incumbent's traces. It makes one isolated change and writes a rationale. The controller validates the child, scores it on the same split, and promotes it only if it beats the incumbent by the margin and doesn't fall below the seed. The held-out split is scored once, at the end, and never enters an overseer's workspace.

The zero-cost dry run shows the shape. A scripted agent replays canned answers and a null overseer proposes unchanged copies, so it exercises the plumbing and nothing else. Trimmed output:

...
seed f8a391c8e95bb784: search score 0.5000
generation 1: asking overseer null to improve f8a391c8e95bb784
generation 1: evaluated; rejected (child does not beat the incumbent by more than the margin): child 0.5000 <= incumbent 0.5000 + margin 0.0500 = 0.5500; child 0.5000 >= seed 0.5000
generation 2: asking overseer null to improve f8a391c8e95bb784
generation 2: evaluated; rejected (child does not beat the incumbent by more than the margin): child 0.5000 <= incumbent 0.5000 + margin 0.0500 = 0.5500; child 0.5000 >= seed 0.5000
...
promotions: 0 of 2 generations (evaluated 2)
cost: $0.00 (stopped: completed)
execution: host (unisolated)

Every real search is meant to run the null overseer alongside. A loop that keeps the best of many noisy candidates will see its score climb by chance alone, and the null arm measures how much "improvement" the selection rule finds in pure noise.

Three seams and one registry

Every role that can be swapped has one small contract in hive/contracts.py, and implementations are picked by name through a single registry. Adding a framework, a use case or an improvement strategy means writing one class. The controller doesn't change.

A harness is data. Each candidate is a directory of framework-native files whose id is its content hash, so identical proposals deduplicate and are never paid for twice. Only the adapter understands the files, and nothing outside an adapter imports a framework.

Pi is the primary framework because it has the widest editable surface, mostly files the overseer can change without a build step. Deep Agents and Strands each run as a subprocess in their own pinned virtualenv. Extensions were the main design question. I chose not to build a portable extension API, because the three frameworks give the "same" hook different semantics. A context edit applies to one call in Pi but persists in LangChain, and a retry discards the first reply in Strands but keeps it in LangChain. So the contract shares the shape around the code instead: one extensions/ directory in every candidate, a per-adapter spec that maps 10 neutral lifecycle points onto the native APIs, and a load report that the controller checks after every run. A file that didn't load fails the trial as a harness load error, so an adapter can't quietly skip broken code and make a candidate look like it ran as written.

The evaluator is the product

That is principle P2 in the spec, and much of the code follows from it. If the score is wrong, the overseer will faithfully climb the wrong hill, and score bugs don't show up in the scores.

The model gateway is the clearest example. Every model request from every framework goes through a proxy owned by the controller. Each trial gets its own token, and the provider key never enters a trial. The gateway refuses any other model, any other reasoning effort and any sampling parameter, enforces a per-trial call limit, and bills from the provider's own usage numbers. After a run, its ledger is the authority for cost, and any violation marks the trial as a model mismatch. Before trusting it, I had Claude Code agents attack it through a fake upstream and the real OpenAI SDKs. It held against 14 attack families, including path desync, duplicate keys, lookalike model names, limit races, key echo and billing on client disconnect.

Errors are kept apart from failures. A trial that hits a quota or a timeout is recorded as errored, not as a wrong answer. The reported mean leaves errored trials out, but selection counts them as zero, so a candidate can't climb by pushing hard tasks into errors. The overseer gets read and edit tools only, no shell and a dollar cap per proposal, and the controller harvests nothing from its workspace except the candidate copy and the rationale.

What I took from Harness²

Harness² is a paper submitted to arXiv on October 9, 2026, with code released by Google Research. It improves a harness per task at test time without a grader, which is a different problem: HIVE runs an offline search over a graded suite and needs one harness that holds up on held-out tasks. I left out its verifier-free selection and per-task adaptation, and adopted four ideas:

• Contrastive evidence. The overseer's brief carries the last two generations of children, promoted and rejected, each with its diff, its report, its parent's report and its traces, so the overseer can see what its earlier edits did to behavior.
• Trace-cited rationales. The "Observed" part of every rationale must cite an existing evidence path. If it doesn't, the controller rejects the proposal as unjustified before spending anything. The check is mechanical with no LLM judge, since a judge would add cost and variance to a check that is only a floor.
• Scoped proposals, off by default. With scopes set, each generation gets one independent proposal per remit (topology, single_agent). The best of N noisy children is inflated by the winner's curse, so the null control must run with the same N.
• Container isolation. Each trial runs in its own Docker container on an internal network whose only path out is the gateway. Denied egress is recorded on the trial. The Harness² release's own experiment guide documents a validation run in which a Codex executor downloaded its task's public grading rubric without being told to. That is the failure this closes.

Where it stands

The public repo has the contracts, the loop, the three framework adapters, the two demo suites and the gateway. The Harness² changes and the Docker backend are finished in my working copy but not yet pushed. On that copy, 1,957 offline tests pass.

Real model use so far is about seven cents of plumbing checks: Pi with gpt-5-nano on both demo suites, a test that the overseer couldn't read or write outside its workspace, and one full generation with a Claude Sonnet overseer. The overseer made one generic prompt change, the child tied the seed, and the margin rule declined it. The stock seed already scores at or near the ceiling on both demo suites, and its score moved between identical reruns. The suites prove the plumbing but can't show improvement.

I built it spec first: the founding principles doc and task logs record each decision before the code, and Claude Code subagents implemented modules test-first, followed by adversarial review passes on the adapters and the gateway.

Next

The first real step is the noise-floor run: score the seed at least three times per task on a suite with headroom, measure the minimum detectable effect, and refuse to claim anything smaller. That needs a harder suite (the default plan is a 30 to 40 task slice of Terminal-Bench 2.0) and a cheap base model that puts the seed between 30% and 70%. After that come lower-bound selection and the multi-agent topology arms that the research question is about.