Memory for coding agents, graded by execution.
Existing memory benchmarks ask a model questions about synthetic conversations and let an LLM judge the answers. This one gives a real agent real work in a real repository, where success depends on something learned in earlier sessions, and grades the artifact by execution: tests pass or they do not. No judge anywhere in the primary endpoint.
Design in six decisions
the whole benchmark, compressedOfficial integrations, frozen and vendor-reviewed
Every product enters through its own published Claude Code integration: plugin, MCP
server, or lifecycle hooks. Each adapter's config is hash-pinned in
config.frozen.json, and each vendor is publicly invited to review it before
the run. The invitation, the response, or the documented silence is committed to the
repository.
Each product carries its own shipped integration
Every memory arm is the same CLAUDE.md bundle plus that product, wired exactly as
its vendor ships it: its own skills, its own MCP server, its own instruction text. The
baseline is claude_md, not bare: nobody runs a coding agent
memory-free, so nothing is measured against a strawman.
The instruction is a treatment, not scaffolding. Equalising it across arms measures a denominator no vendor ships; letting each carry its own measures what a user installs. The official run does the second and publishes every arm's instruction size beside its result. The equalised variant exists as a separate, labelled ablation.
One neutral experience feed, each product's own write path
The corpus is verbatim recorded agent session transcripts. Every adapter ingests identical bytes; what its extraction pipeline keeps is part of what is measured.
Executable endpoints only
Checkers run the artifact against oracles the sandbox never contained. A do-nothing session scores zero. Every task ships a naive reference solution that must fail and an informed one that must pass, asserted in CI.
The admission gate
A grid cell is discarded, not scored, unless every arm can prove its treatment was applied: MCP tools listed at session init, lifecycle hooks demonstrably fired with output, sandbox files digest-verified, and no arm holding another arm's tools. Discard counts are published per arm.
Only an arm with a memory surface can fail to wire, so the rule protects one class of arm's worst outcome and no other's. Every headline is published beside an intention-to-treat column. A timeout is an outcome, not a wiring fault, and is never retried.
Costs are end-to-end
Ingestion tokens and session tokens land in one per-arm ledger, alongside wall time and negative-transfer counts. Deltas below the preregistered minimum effect are reported as noise, not as findings.
The arms
same feed, same tasks, same gateEach arm exists to remove one explanation for a result.
bare is the reference damage is defined against. placebo is
project-shaped prose with no memory content, matched to the baseline bundle on line count and
whitespace tokens, and it separates "memory helped" from "any extra context helped".
claude_md is the honest competitor: a well-written static instruction file costs
nothing per query and is what most teams actually have. A memory product that cannot beat it
has not earned its tokens.
What each arm isolates, in full →
protocol is the control two readers asked for, and it is
the one that changed the question. It carries the complete memory instruction and no memory,
so it separates the memory from the telling. Because every memory product
requires that instruction, "does memory beat no memory" is not directly answerable; what is
answerable is whether a product earns back the cost of asking for it.
fs_grep and recall_prefetch bracket the
product. The first is the cheap answer, the whole corpus on disk with grep over it; the second
runs retrieval in the harness with the task prompt already in hand, which is the ceiling the
live arm is trying to reach. Neither is ranked as a competitor.
How they decompose the problem →
One arm is not running, and its absence is a
recorded choice: oracle_memory injects the exact evidence, so under the
absent condition it would hand over an answer the corpus is defined not to
contain. It returns when its bundles are condition-aware.
Where this actually stands
2026-08-31| piece | state |
|---|---|
| harness | runs; sandbox, admission gate, paired statistics, cost ledger, MCP preflight and bounded retry |
| tasks | 34 executable tasks, each with a naive and an informed reference asserted in CI. Three of them need two sessions combined, which single-fact retrieval cannot do |
| harm suite | five corpus conditions across 73 task-conditions; eleven tasks carry all four damage conditions, above the threshold of eight for reporting one as a result |
| arms | eight in the approved run: bare, placebo, claude_md, protocol, fs_grep, recall, mempalace, recall_prefetch. Two products, each wired through its own published integration and pinned to a released version, and each handed the same instruction byte for byte |
| corpus | 4,900 documents per condition, up from 196. It was rebuilt because the old feed was too easy: bm25 hit@1 fell 0.485 to 0.182 and voyage hit@10 fell 1.000 to 0.879 on the new one |
| runs | every run so far is bring-up, not result: five pilots, a harm-suite first pass, a calibration, a diagnostic and one full grid. The grid is treated as calibration because it showed the feed could not separate the arms. No arm-level number from any of them is quoted |
| leaderboard | empty by construction: a number reaches it only from a published run summary |
| reproducibility | checkable, not yet re-runnable. python -m scripts.verify_run --all re-derives the ledger, the endpoints and the discard set from the published sessions with no credentials and no money. Re-running is the gap: bare, placebo, claude_md and fs_grep need only the CLI and a model key, while recall needs a database, an embedding key and a built index that the compose stack does not start |
The dated status page in the repository carries the run-by-run table, the open blockers in the order they have to be cleared, and the command that re-derives every claim on this page: docs/STATUS.md. Every change to the instrument that moves a published number is recorded in docs/audit/.
This benchmark is built by the authors of recall, which competes in it.
That is exactly why the methodology is preregistered, the harness is open, every adapter
config is vendor-reviewable before any run, and all results are published, including the
ones recall loses. The full run's protocol is committed under
preregistration/ before a single session starts.