Sapience: what is current in an evolving codebase

A reasoning architecture built around an LLM (any LLM -- we run it with 8B open weights and with closed flagships) that extends it past its context window and across sessions. This page is the current measured record. Methods are the conservative choices throughout: pre-registration, multi-seed where variance warrants it, mechanical ground truth instead of LLM judges wherever the task permits, Wilson intervals on small n. Reproduction bundle on request.

Coding: who knows the current state of the code?

Public coding benchmarks resolve inside a single context window (hard SWE-bench instances finish well under 50K tokens in our runs), so they cannot distinguish a system that accumulates knowledge about a codebase from one that re-derives everything per run. Repo-Evolution targets the accumulation regime directly: the agent experiences the real git history of four production repositories (Django, LiteLLM, huggingface_hub, vllm) as ordered time-slices, then answers currency and supersession questions -- the current value of a setting that changed at least twice, which value replaced which, whether a value was ever a default. Gold is derived mechanically from git; there is no LLM judge in the scoring path. Each probe additionally passes a no-memory screen (a reader without the history answers 0 of 77 correctly), which removes parametric-knowledge contamination from the endpoint.

The margin has a mechanism, and it is an architecture property rather than model capability. The store records each change as a supersession edge at write time; a currency question then reduces to traversing a chain to its head. A vector store over the same update stream holds several near-identical embeddings of the same setting with nothing marking which is current -- write order is discarded at ingestion and similarity search cannot recover it. The same mechanism explains the model-size result: an 8B open model on the store lands within a few points of the frontier model on the store (identical 26/27 on the largest repo), because the store resolves the answer and the model mostly transcribes it. Both configurations clear every store-less baseline by 30-65 points.

Task-level pilot: complex alterations on a large codebase

Repo-Evolution above measures recall. The Repo-Task pilot measures work: does the same agent make a real code change more reliably, or at lower token cost, when it carries an accumulated memory of the codebase? The design is pre-registered. Tasks are real commits from two large production repositories (litellm, huggingface_hub), replayed as instructions to re-derive the change the maintainers actually made -- including multi-file alterations. The same agent and the same model (Sonnet) run every task under three configurations: unassisted baseline, with Sapience, and with Sapience plus market features (a variant that front-loads the memory and labels it). Co-primary metrics are task completion and tokens-to-done. Two grading strata ran so far: a diff-equivalence stratum (6 tasks; the pool exhausted at 6 because the leakage screen dropped five further candidates whose task statements revealed the answer) and a tests-graded stratum (11 tasks run; 2 more selected tasks were not reached when the pre-registered budget cap fired). The screens are strict on purpose: a task whose statement, or whose checked-out tree, gives the answer away measures nothing.

The pilot has produced two observations so far, and each is a single observation -- one task, one seed, descriptive only. First: on the one task all three configurations completed in the diff-graded stratum, a commit-churn task where the store held the prior-value chain, the Sapience configuration completed at roughly 11,400 tokens vs roughly 97,500 unassisted. A comment in the checked-out source partially confounds the attribution, so this stays labeled a single observation, not a pilot result. Second: on one jointly completed task in the tests-graded stratum, the Sapience configuration finished at about 133,000 tokens while the unassisted baseline hit the 300,000-token cap still running (its change graded complete on the tree at stop, but it had not declared done), so the baseline's token count is censored at the cap and the measured saving is a lower bound. The other jointly completed task in that stratum ran about 25% more expensive with Sapience. Split sign at two joint completions -- no efficiency conclusion until the powered run reports.

Across five measurement rounds, agents with the memory available consulted it voluntarily in roughly 0-8% of episodes -- even when the store demonstrably held the needed information (store coverage confirmed for 5 of 6 tasks' target symbols in the diff-graded stratum). Delivery in these runs is scaffold-guaranteed: the harness surfaces the memory rather than waiting for the agent to ask for it. The product's forced-surfacing and briefing design exists because of exactly this pattern.

Status: powered run (pre-registered, multi-seed) in progress; results will be reported either way.

Matched pairs: same weights, with and without the architecture

Identical model and question set per pair; the architecture is the only manipulated variable.

Published benchmarks

Repo-Evolution is ours; these are not.

Full provenance, per-cell denominators, and the rest of the battery ship in the preprint. Memory benchmarks are included because a reasoning architecture that learns across time should dominate them as a side effect -- and does; they are not the product.

At the frontier

The cross-vendor picture, measured: readers running alone collapse as the corpus grows past their window; the architecture holds a bounded working set and keeps answering at every scale we have measured, now out to 100M tokens.

Under the official BABILong scorer, on the leaderboard's own HuggingFace cells, the qa3 ladder reads 86 / 87 / 87 / 84 / 86% across 128K-10M, against 37% at 10M for ARMT, the best published entry at that scale and a model fine-tuned on these tasks. Past 10M the line stays flat: 81 / 80 / 80 / 80.0 / 79.7% at 2M / 5M / 10M / 30M / 100M, three seeded runs at n=300 per scale, official scorer, on generator-built corpora item-paired to the 4K-1M ladder. The 100M value sits 0.3 points below the pre-registered [80, 90] band. Beyond 10M the corpora are the official recipe scaled up (official generator unmodified, needles byte-identical to the ladder, PG-19 training split as filler because the official test pool is ten times too small per 100M item), and the pipeline is near-deterministic, so "3 seeds" is not 900 independent items. The reader payload is ~685 tokens at every scale. Compute for the 30M and 100M cells: about $3.

Taken with the other four BABILong tasks at 10M (qa1, qa2, qa4, qa5, each at 100), the average across qa1-5 at 10M is 97.2 under the official scorer, against 76.6 for the previous best published entry, the fine-tuned ARMT. Submission open: booydar/babilong PR #19, with per-task predictions, seeds and configs. Every number in this section carries the same disclosure - the retrieval pipeline is task-adapted, as the fine-tuned specialists are task-trained; the reader is Sonnet 4.5 with its weights untouched; and we published a zero-adaptation audit of how much the adaptation contributes (thread). Three of the five 100s (qa1, qa4, qa5) fall to extraction scripts with no LLM at all; the reasoning load lives in qa2 and qa3.

Two scoped firsts, each checked against the literature on 2026-08-27: these are the first BABILong qa3 (multi-hop) measurements beyond 10M tokens, and the first BABILong/RULER-class reasoning measurements at 100M tokens. Magic.dev reported in-house HashHop retrieval numbers at 100M tokens in 2024 on an unreleased model; no standard-benchmark (BABILong/RULER-class) results at that scale have been published by anyone. Frontier readers alone, for comparison: at 1M the same model alone (Sonnet 4.6, n=90 paired) reads 34.4% under the matched protocol; no clean multi-vendor 1M cell exists yet, and the Gemini and GPT-5.5 attempts at that scale are pilot rows we do not quote. At 128K, where the facts sit inside the window, Gemini 3.1 Pro and Opus 4.7 both land at 33.3% (n=12-15, single seed, directional only), about half the same-scale Sonnet-alone plateau. No Opus 4.7 or GPT-5.5 score at 1M is quoted on this page; the cells we hold at that scale are pilot cells we do not treat as clean.

Token accounting

A conventional agent re-sends its accumulated history with every request, so input tokens grow with session length; KV caching reduces the unit price of that re-read but not its size. Reads from the store scale with the query instead of the history. As a cost model on our benchmark setup: a few hundred input tokens per answer from the store vs ~304,000 re-sent once the accumulated history exceeds the window -- roughly $0.006 vs a hypothetical $30 per query at the 10M-token tier (hypothetical because past ~1M the history no longer fits in any window at all). Caching and store-reads are independent savings and compound.

Where should learning live?

Everything above is a frozen model reasoning better because a structured store sits beside it. The next question is separate: when a system should learn from use -- get permanently better at your work as you do it -- where should that learning be written? Two answers are being funded. One writes it into the weights (fine-tune or adapt a model per user or per domain). One writes it into the context (hand the model more text at read time). Our bet is a third: a structured substrate the model reads, where each update carries its order, its supersession, and its provenance. We are testing this the way we test everything else on this page -- a pre-registered uplift study, three arms on one corpus (a frozen frontier reader with retrieval, the same reader over the substrate, and a domain-trained open model), scored on lookup, application, and transfer strata, with the predictions and the kill condition written down before the eval ran.

The first results are in, and the first arm reported against a prediction we had written down. On a static corpus, with application questions answerable from inside it, the substrate condition tied strong retrieval exactly -- an even split, p=1.0 -- and both beat the closed-book model by the same margin. We pre-registered that this outcome would hand the within-corpus claim to retrieval, so that is what we report. A code audit afterward narrowed what the tie settles: on this run the store was populated by keeping each source verbatim, with the consolidation step -- the write-time ordering, supersession, and cross-source synthesis that are the actual bet -- never engaging, so the result adjudicates plain ingestion rather than the substrate we are arguing for. That consolidating version is now pre-registered as the follow-up test. Reading the missed questions one at a time surfaced a further limit that belongs to the test rather than to either method: about half the held-out answers were post-cutoff results present in no document either condition could retrieve, unanswerable from the evidence by construction, so the battery is being rebuilt with a stratum whose answers are derivable from the corpus before the follow-up runs.

What the substrate wins is the regime the rest of this page measures: the corpus that moves rather than sits still. Values that update and supersede, knowledge that accumulates across sessions, questions whose evidence is past the window entirely -- that is where write order, supersession chains, and a bounded working set do work no snapshot retriever reproduces, and it is the coding currency result and the multi-hop and 10M results above. On the transfer stratum the substrate also reaches source material dense retrieval cannot surface by construction, which replicates the discovery-reach result in our other work; we state the construction rather than score it as a head-to-head win. One further measurement bears on the weights answer specifically, and it is a different task class from the within-corpus tie: on a matched 8B model, the same material consolidated into the weights raised held-out accuracy by +5.6 points, while handing it to the model in context at read time lowered it by 8.7 points -- where knowledge helps depends on how it is delivered, and more text in the window is not the same thing as learning. The third arm, a domain-trained model, is training now; we will report where it lands either way.

One property separates the substrate bet from the weights bet regardless of which wins on accuracy. Learning written into a model's weights belongs to whoever holds the model, and cannot be attributed, ported to another model, corrected, or combined with what a second person learned. Learning written into a structured, signed store stays with its owner, moves across models, and, because every object records who established it, can compose across people without being surrendered. It also captures what never reaches a training corpus in the first place -- the dead ends, the negative results, the supersession history of real work, none of which gets published. That is the axis the weights bet cannot reach, and it is why we are building the substrate rather than a better fine-tune.

Why this is architectural

The three results above share one cause. A transformer's context window is a fixed-size working memory: whatever an agent learns either fits in it or is gone, and everything it keeps must be re-read on every request. Sapience adds the tiers that are missing -- writes go to a structured store at the moment they happen, with order, supersession, and provenance preserved, and reads pull back only what the question needs. Each benchmark isolates one consequence. The coding result is write-order preservation: "what's current" becomes a deterministic traversal, which no amount of retrieval quality or model scale recovers once ingestion has discarded the order. The matched pairs are the read path working where no window reaches -- decomposed, the single-fact and aggregation constituents these questions reduce to are retrieved essentially at ceiling even at 10M tokens, with overall per-constituent recall at 75-82% and long ordered sequences the identified weak case. The token accounting is that same read path costed: reads scale with the query, not the history. Model capability moves none of these, which is what the 8B result demonstrates -- and why we call this an architecture rather than a feature on top of a model.

Feel it: replay a real probe

Pick a setting below. You see the question, then how that setting's default actually changed across the repository's git history (oldest to newest), then what each system answered when asked for the current value. Correct and incorrect are scored mechanically against git -- the same boolean recorded in the run files, no judge in the loop.

Replay of recorded benchmark runs -- no live inference; full rows in the reproduction bundle. All five arms read with the same model (DeepSeek-V3.2); Sapience is the edge-traversal configuration.