Recursive Language Models and the End of the Context Window
Context stops being tokens inside the attention window and becomes an environment the model queries.
I keep a small list of the walls I hit over and over building agents, and one of them finally has a name. For a long time it was just a feeling: you give the agent more to work with, more documents, more history, a longer conversation, and past a certain size it quietly gets worse. Not broken. Worse. It starts missing things it caught earlier, it hedges, it gives up on tasks it should be able to do. The name is context rot, and it doesn’t respect the number on the box: a model can advertise a 200k or million-token window and still degrade long before you fill it. The size of the window was never the real constraint. What the model does with everything inside it is.
I’ve spent a lot of issues circling this. Harnesses were about keeping an agent coherent across sessions when a single window can’t hold the whole job. The three primitives were about giving an agent hands so it isn’t reasoning from memory alone. Every one of those was me, the builder, deciding in advance how to break the work down so no single model call has to swallow more than it can chew. That instinct is good. It’s also the thing a paper out of MIT this past December quietly turned on its head, and I haven’t stopped thinking about it since.
Context as an environment, not a window
The idea is called a Recursive Language Model, and the cleanest way I can put it is this. Instead of pouring the entire context into the model’s prompt and hoping it holds, you hand the model only the question. The context, the giant pile of documents or history, sits outside the model as a variable in a little code environment, a REPL. The model can see how big it is. It can’t see all of it at once, and that’s the point.
From there the model works the context like data. It peeks at a sample, greps for a pattern, slices the pile into chunks and calls a fresh copy of itself to read each slice and report back, then combines those answers into a final one. No single call ever sees the whole thing, so no single call ever chokes on it. The model steering all this stays nearly empty, because it reads summaries of the work, not the raw material.
There’s a quiet reliability win buried in that word “slice.” Anyone who has pointed an agent at 500 documents has watched it process the first 75, lose the thread, and report back as if it finished. When the slicing is code the model wrote, that failure mode disappears: a loop over 500 chunks runs 500 times or it errors. Coverage stops being something you hope the model remembers and becomes a property of the structure.
This is the move that reframes everything: context stops being tokens inside the attention window and becomes an environment the model queries. You don’t stuff a haystack into the prompt and pray. You put the haystack in a variable and let the model write the search. This isn’t RAG, by the way. There’s no index built ahead of time; the model greps the raw context on the spot, which is what makes it work on one-off inputs too arbitrary to index. And it isn’t a normal tool loop either, which is the part I find genuinely beautiful.
Agents decompose the problem. This decomposes the context.
Here is the sentence that reorganized how I think about it. Every agent pattern I’ve ever built decomposes the problem. A Recursive Language Model decomposes the context, and lets the model decide where to cut.
When I design an orchestrator with worker subagents, I’m encoding my own intuition about how the job splits. Research goes here, drafting goes there, review happens last. That’s human expertise baked into a workflow up front, and for well-understood tasks it’s the right call. But the decomposition is mine, fixed before the model ever sees the specific input.
The recursive approach inverts that. It hands the model the pile and lets it decide, per query, whether to sample first, or grep, or partition and map across the pieces, shaped by what’s actually in there. For inputs too big or too irregular for me to have good intuitions about, that turns out to matter a lot.
What surprised me is that the model doesn’t flail when handed that freedom. Left to decide, it settles into recognizable moves, fan the work out and merge the results, generate several passes and keep the best, loop until a pass turns up nothing new. Nobody hard-coded those. They’re the shapes a model reaches for when you stop pre-deciding the decomposition for it, and watching them emerge is most of what convinced me this isn’t a gimmick.
The proof is the boring part, and that’s the point
If you’ve read this newsletter for a while you know my one-note conviction: the model isn’t the bottleneck, the harness is. I’ve said it enough that I owe you evidence, and this is the cleanest I’ve seen.
On a long-context benchmark, a small model in this recursive setup beat the full frontier model by a wide margin, at roughly the same cost per query. Not a better model. The same model, arranged differently. And fed the equivalent of thousands of documents, it held its accuracy at millions of tokens, far beyond any current context window, where the plain approach fell apart. Roughly two orders of magnitude past what the window alone allows.
That gap isn’t a model result. It’s a harness result. The intelligence was already there. The arrangement was what unlocked it. I can’t think of a cleaner demonstration of the thing I keep insisting on.
There’s a neutrality bonus hiding in here too, and it’s one I care about after the lock-in issue. Because the steering model and the worker calls are just calls, nothing forces them to be the same model or the same provider. You can put a frontier model in the orchestrator seat and cheap open-weight models underneath doing the grunt reads, or flip it. The pattern is model-agnostic by construction, which is exactly the kind of architecture that lets you walk away clean instead of getting funneled into one vendor’s stack.
Now the part the launch posts skip
I don’t want to sell you a miracle, because it isn’t one yet. The published implementation wasn’t tuned for speed. The recursive calls run one after another with no caching, so a single query can take seconds to several minutes, with no hard guarantees on cost or runtime. And the headline numbers came from what the author himself called a smoke test, not a hardened production system.
So I’d file this where it belongs. The idea is durable. The engineering is early. The reframe will outlast this implementation and probably this paper. What it’s already changed for me is smaller than a rewrite of my stack: it changed the question I ask at the start of a design. I used to ask how much context I can fit, and how to pre-chop the job so nothing overflows. Now I ask whether the model needs to see all of it at once, or whether I can hand it the pile and let it decide how to read. Usually the answer is that I was doing work the model could do itself.
— Ali Ibrahim
If this reframed something for you, pass it along or subscribe so the next one finds you.
Exploring the frontier of AI agents, one system at a time.
Wrestling with how to give an agent more context than fits in a window? Ask the Agentailor agent. It’s built to answer exactly those questions.
P.S. — a few things I published on the blog recently, if you want to go deeper:
MCP v2: What’s Changing, What’s Deprecated, and Why — MCP goes stateless in v2, and sampling, roots, and logging are on the way out. A walk through what breaks across the SDKs, why the protocol is shedding weight, and whether it’s worth migrating yet.
Is LangChain Worth It in 2026? — An honest practitioner’s read on the v1 stack: create_agent, middleware, deploying, observing, and MCP, and where it earns its keep versus where you’re better off composing your own.
Top 7 AI Agent Evaluation Frameworks (2026) — The open-source eval tools worth knowing, from DeepEval and promptfoo to agentevals and Strands Evals, sorted by what they’re actually for: trajectory checking, CI gating, red-teaming, and scoring agents in production.
More from me at Agentailor.
Sources
Recursive Language Models (Alex L. Zhang, Tim Kraska, Omar Khattab; MIT CSAIL). The paper this issue is built on, and the source of the context-rot reframing and the benchmark results.
How to Use RLMs in Deep Agents (LangChain). Where I first saw the pattern applied to production agent orchestration.
Introducing Dynamic Subagents in Deep Agents (LangChain). The companion piece on dispatching subagents programmatically.
Building Effective Agents (Anthropic). The orchestrator-workers framing I lean on when contrasting problem decomposition with context decomposition.