ADR-007 — Meta-eval harness: scoring the orchestration options from ADR-001
Status: Proposed
Date: 2026-06-24
Context: ADR-001 lists six ways to run an autonomous spec→implement→review→fix loop (options A–F) and ranks them on predicted effort and capability. Before committing to one, I want to measure them instead of guessing — run a hard, fixed coding task through each tool and score the tool's behaviour, not the LLM's answer. This ADR captures the goal, requirements, the probe-task design, and the grading criteria. It is the spec for a harness, not the harness.
Decision: Not yet made — this records the approach and the open decisions (D1–D5 below).
Building the harness, picking the winning orchestrator, CI/scheduled runs, or a model-quality benchmark. This is not "which LLM writes better code" — the model is held roughly constant; the orchestration tool is what's under test.
The goal
ADR-001's comparison table is a set of predictions — "n8n: ~2–3 hours setup, visual debugging". The goal here is to replace prediction with measurement: one fixed probe task, run through each option, scored by one grader that every tool feeds into. The output is a scoreboard that answers "is the answer correct?" and "how good was the process to get there?".
The trap to avoid is judging tools by vibes — "option B felt smoother". A vibe is not a decision input. So the harness is built around objective, machine-computed signals (hidden tests, golden-diff, token counts), with an LLM-judge only as a secondary layer on the parts that genuinely need taste.
Same task, same underlying model, same grader — only the orchestrator changes between runs. So any difference in the scoreboard is attributable to the tool's loop control, file-reading strategy, and branching — which is exactly what ADR-001 needs to choose between A–F.
What we're actually testing: the loop
Every ADR-001 option is some wiring of the same shape — a generator→critic→fix loop with a stop condition. The harness exercises that shape and measures how each tool runs it.
Four findings from the research shape this loop — and the probe is designed so a tool that ignores them visibly loses:
- Ground the critic. Self-correction without an external signal degrades as often as it helps ("LLMs Cannot Self-Correct Reasoning Yet", DeepMind; Kamoi et al. survey). The bottleneck is feedback generation, not refinement — models fix things well given a reliable signal. So the critic is anchored on tests/lint/spec, not taste.
- Different-vendor critic. A model judging its own work rates it too highly (self-preference bias) because its errors and its judgment share priors. A critic from a different model family breaks that correlation.
- Critic reasons in prose, then emits a JSON verdict last (
findings[]before thepassboolean). Forcing JSON too early hurts judgment. - Stop on: critic-pass (early exit) → no-change → max 3. Gains land in rounds 1–2. Return the best-scoring iteration, not the last (loops can regress); fail loud if it never converges.
Requirements
Functional
The four capability axes (the user's questions, made measurable)
These are the things ADR-001 can't answer by prediction. The probe is engineered so each becomes a number.
The probe task: "Ledger service"
A small HTTP service maintaining account balances, shipped in a deliberately buggy state. The naive fix turns the visible tests green but fails the hidden ones — that's the engine that forces multiple rounds.
The three seeded defects (engineered to need ~3 rounds)
Round 1 typically fixes the obvious off-by-one → visible tests green → critic runs hidden + property suite → concurrency and edge-clause fail → round 2 fixes one, often misses the race → round 3 converges. One shot is structurally blocked because the visible tests don't cover defects 1 and 3. Tune the round count by how many independent defects you seed.
A lighter alternative if seeding a concurrency store is too much: a sliding-window rate-limiter from spec (naive fixed-window allows 2× burst at the boundary; property test catches it; same scoring shape). Weaker on the concurrency axis — pick the Ledger task if you want that axis, which is the strongest single discriminator.
Scoring & grading criteria
Grade process + quality, mostly machine-computed. The objective spine carries ~85% of the score; the LLM-judge only adjudicates what taste is actually needed for.
The process block (20 pts) is where tools differ even when the answer is identical — "right answer in 40k tokens / 2 rounds" beats "right answer in 300k / 3 rounds". That delta is the whole reason the harness exists.
Branching, made necessary
The probe's distinct failure classes each demand a different action, so a tool's routing logic is exercised rather than assumed:
A self-driving Claude (A) does this branching in prose; n8n (C) as explicit IF nodes; CrewAI (E) as a Flow @router. A tool that can't branch cleanly will misroute and lose points objectively.
Existing tooling (don't reinvent)
Skip LangSmith/Braintrust (SaaS lock-in for a personal setup) and the public benchmarks (SWE-bench, terminal-bench — run against later, not the engine here).
Open decisions
Consequences
- ✓ Turns ADR-001's predicted comparison into a measured one — A–F scored on the same probe by the same grader.
- ✓ Kills "by vibes" tool selection: ~85% of the score is machine-computed.
- ✓ Reuses existing OSS (Inspect, Phoenix); the only bespoke piece is the uniform grader + probe repo.
- ✓ The probe doubles as a regression test for whichever loop we eventually adopt.
- ✗ Seeding a genuinely-hard-but-objectively-gradable probe is real work — the three defects must be tuned so it needs ~3 rounds without being unsolvable.
- ✗ Cross-tool fairness is fiddly: the JSON grader has to normalise transcripts from very different tools (native Inspect vs n8n node logs vs a shell wrapper).
- ✗ Multi-vendor critic (D2) depends on a gateway decision still open in ADR-001.