The language model is a part, not the platform - everything else survives swapping it.
What actually happens when one AI session writes a fact and a different session, days later, on a different vendor's model, answers with it. Write gate, supersession, retrieval walk, overnight consolidation, and the token arithmetic - with every number's n and seeds attached.
Three components, three kinds of state. The model: frozen weights, swappable. The harness: prompts, tools, the work itself. And between them the piece that's usually missing: a learning system with its own state and its own learning rules, which writes at capture time and consolidates offline. The language model is a part, not the platform - everything below survives swapping it.
Work arrives as conversation, code, documents. At capture it's distilled into typed knowledge objects: a claim, its evidence, its condensed reasoning, provenance signed to an author, and typed edges to related claims. Not transcripts - transcripts are what context windows are for, and context windows are the thing that doesn't scale. A knowledge object is a few hundred tokens that stand for the few hundred thousand it came from.
Not everything writes. Encoding strength is gated by prediction error: what contradicts or extends the store writes strongly, the redundant fades before it ever competes for retrieval. This is the piece people underestimate - the intelligence of a memory is mostly in what it declines to keep. A store that keeps everything is a haystack with better marketing.
The concrete case first. In March you decide the API timeout is 30 seconds; in July you change it to 60. To a similarity search those two sentences are nearly identical text, equally close to any future question about timeouts - so it will happily serve the March value forever. Supersession is the fix at the storage layer, not the search layer.
When a new claim lands on an entity that already carries one, the two are linked at write time and the old claim is marked superseded. Retrieval walks each chain to its head. So a stale value cannot outrank its replacement, however similar the stale text is to the question - the ordering is structural, not semantic. In 81 update probes across our test suites, zero stale facts leaked back into answers.
We measured what this is worth directly: on date-obscured supersession chains, structure-aware retrieval beats similarity retrieval by 67 to 78 points (3 seeds, discordant-pair McNemar). The steelman control matters: hand-ordering the same retrieved items recovers parity - which isolates write order as the signal the structure carries and similarity search discards.
A question comes in. Retrieval is iterative: fetch, read, notice what's missing, fetch again - walking edges between objects rather than grabbing the top-k nearest neighbors and hoping. On multi-hop questions this is the difference between finding a fact and composing a chain.
The cleanest demonstration is RULER variable tracking, because the task is mechanical: variables assigned through chains (X7 = X3, X3 = X1, X1 = 12345), answers impossible to guess parametrically. At 1M tokens stored: 24/24. At 2M, 5M, and 10M: 16/16 each (2 seeds, exact match, corpora sha-pinned). The audit checked every row: the retrieval surfaced exactly the 5-line assignment chain, in order, every time.
And on real data, where nothing is mechanical: multi-hop questions over real Wikipedia (MuSiQue), scored by official exact match with no LLM judge, same model in both arms so the memory is the only variable. Full context, truncated as the corpus outgrows the window: 19% falling to zero by 8x the window. With the memory underneath: 43% declining gently to 28% past a million tokens - because the middle of the chain is still there when the question needs it.
Offline, the system compacts specifics into structure. It clusters objects across sources and sessions and synthesizes higher-order claims that exist in no single stored record - then those synthesized claims are themselves stored, gated, and supersedable. Fast gated capture feeding slow statistical integration is the division complementary-learning-systems theory describes in brains; the mapping here is mechanism-level, not metaphor. On questions whose answers exist in no single stored input, consolidation lifts accuracy from 26.7 to 66.7 percent (McNemar: 12 items flipped in favor, 0 against; 3 seeds).
The store also schedules what an attached model learns. Consolidation renders a nightly training curriculum from what the store currently holds - new material interleaved with replay of settled knowledge, superseded facts excluded - and a small adapter learns it beside a frozen base model. Delete the adapter and the base is restored exactly; the curriculum is inspectable text with per-fact provenance. In pre-registered controlled runs (≤8B models, two model families, 3 seeds each), consolidation-scheduled replay retained 50 to 75 points more prior knowledge than sequential fine-tuning while learning the new domain as well or better. And replaying structured objects beats replaying the same content as raw prose by 10 to 12 points at a matched token budget - the structure itself carries into weights. This path is measured in the lab; what ships today is the store, retrieval, and consolidation over a frozen model.
Here's the calculation that makes the whole design make sense. Take a working history of 1M tokens and a model that re-reads it per query, versus a memory that retrieves what the question needs:
The flatness is by construction: the retrieved payload stays near-constant in size, so accuracy-flat-at-10M is scale-invariance of the mechanism, not a claim that any model reasons over 10M raw tokens unaided - which is also why answers stay fast and cheap no matter how much has piled up.
Published nulls, because a system you might build on deserves the failure map: within a single window, single shot, static corpus, we tie a frozen frontier model with strong retrieval - +0.00pp, p=1.0, losing condition published before the run. The advantage begins where facts change, evidence exceeds the window, or work accumulates, and is absent where none of those hold. Trajectory-style questions - reconstructing the path of changes rather than the current head - score 0 to 13 percent and are unsolved. Asking the system to reconstruct the exact sequence in which past events happened - not what is true now, but the order it became true in - fails outright, 0 percent under the strict scorer: we keep the current state of the world, and the state is not the path. The weight path is the least mature mechanism in the system. And knowledge objects are text-only today - multimodal claims (a figure, a screenshot, an audio clip bound to the same provenance and edges) are designed but not built.
Everything above traces to a result file and the script that produced it. The RULER full suite - protocol, harness, corpora generators - goes public this week; the MuSiQue harness re-runs in an afternoon on a rented open model, and we'll send the reproduction package to anyone who asks. If you re-run it and get different numbers, we genuinely want to know - that is the fastest way to find our next bug.
And the follow-up we most want to run: letting the system adapt itself to a new task family overnight - consolidation and the learning loop doing the tuning, no engineers - and measuring again. "It adapted, we didn't" is the continual-learning claim in benchmark form, and nobody has published it yet.
The paper (architecture + full benchmark battery) is at spnc.ai. Next post: what consolidation finds that nobody wrote down - the discovery side of the same machine.
After this article was written, the numbers above grew into a leaderboard claim. We ran all five BABILong tasks at 10M tokens under the official scorer: qa1, qa2, qa4 and qa5 at 100, qa3 at 87.7 - an average of 97.2, against 76.6 for the best published entry, a model fine-tuned on these exact tasks. That would place the system #1 overall at 10M on the public leaderboard; the submission is an open PR with per-task numbers, seeds and configs: booydar/babilong#19. The retrieval pipeline is task-adapted, and three of those 100s deserve suspicion rather than applause: qa1, qa4 and qa5 fall to naive extraction scripts with no LLM at all, so the benchmark's reasoning load lives in qa2 (34 to 100 over the naive floor) and qa3 (32 to 87.7).
Then we did the thing leaderboards don't do. We ran the same suite through our shipped generic pipeline with zero task adaptation and published that number too: 24.7. The mechanism is clean - generic embedding retrieval surfaced zero gold facts on qa1 and qa2, because bAbI's synthetic template grammar is essentially invisible to semantic similarity, while the language model is 92% correct whenever the right facts reach it. The gap measures retrieval visibility on synthetic text, not reasoning; on the real corpora throughout this article (Wikipedia multi-hop, repo histories, conversations) the generic pipeline is the one producing the published numbers. We think every leaderboard entry should ship its own gaming coefficient. This audit cost $3.58 to run.
