Why Does Your AI Suddenly Get 'Dumber' Mid-Conversation?
You've probably seen this pattern: at the start of a session, your AI is sharp, precise, and follows instructions well. But after dozens of messages, several file reads, and many tool calls, the answers start drifting. It forgets decisions you already agreed on, re-asks questions that were already answered, or misquotes technical details that are still sitting right there in the conversation.
This isn't a bug, and it isn't a sign the model is "dumb." It's a direct consequence of how the context window actually works — and understanding that mechanism is the first step before you can fix it.
The Context Window Is a Finite Resource, Not Free Empty Space
Many developers treat the context window like a hard drive: as long as it isn't full, just keep stuffing it — conversation history, search results, document contents, tool-call logs. The assumption is that as long as you haven't hit the hard token limit, everything is fine.
That assumption is wrong. Context is the set of tokens the model has to process on every inference pass, and every new token consumes some of a finite "attention budget." This scarcity comes from the transformer architecture itself: every token can attend to every other token across the entire context, producing n² pairwise relationships for n tokens. As context length grows, the model's ability to capture all of those relationships with precision gets stretched thinner and thinner.
The analogy to human working memory is apt: we have a limited capacity to hold active information in mind. Cramming more data into the context window doesn't make a model smarter — if anything, it forces the model to split its "attention" across more competing signals at once.
The guiding principle of good context engineering is actually simple: find the smallest possible set of high-signal tokens that maximizes the likelihood of the desired outcome. It isn't about how large your context window is — it's about how efficiently you fill it.
Context Rot: Why Accuracy Drops as the Window Fills Up
This phenomenon has a specific name: context rot. Needle-in-a-haystack style benchmarking research has uncovered a consistent pattern across models: as the number of tokens in the context window increases, a model's ability to accurately recall information from that context decreases — even though the information is technically still present.
The important nuance here: this isn't a hard cliff, it's a performance gradient. Models remain highly capable at longer contexts, but their precision on information retrieval and long-range reasoning tends to decline compared to shorter contexts. Some models degrade more gently than others, but the underlying characteristic shows up across the board.
A concrete example from a research-agent experiment makes this tangible: when an agent read eight documents of roughly 40,000 tokens each with zero context management, its context ballooned past 335,000 tokens. A breakdown of what was actually sitting in that context showed that more than 96% of it was file-read content — mostly documents the agent had already processed and taken notes on, but which kept getting reprocessed on every subsequent turn, competing for the model's attention against more relevant information.
There are two practical consequences of context rot:
- On models with a smaller window (say, 200K tokens), the agent simply stops once the limit is hit — the API rejects the next request and the task halts mid-task.
- On models with a larger window (say, 1 million tokens), the agent keeps running, but answer quality degrades because early details get buried under an ever-growing pile of accumulated context.
So waiting for bigger context windows isn't a long-term fix — the window still fills with stale, no-longer-relevant data; the wall is just further away.
Three Core Primitives for Managing Context
There are three complementary core techniques for keeping context lean on long-horizon tasks:
1. Compaction
Compaction is the practice of taking a conversation that's approaching the context window limit, summarizing its contents, and reinitiating a fresh context window built on that summary. The model is asked to distill what matters — architectural decisions, unresolved bugs, next steps — while discarding redundant tool outputs and messages that are no longer relevant.
In testing, with the compaction trigger set at a given threshold, a research session that would otherwise have ballooned to roughly 335,000 tokens had its peak held down to around 169,000 tokens once automatic summarization kicked in. High-level facts (organism names, key figures) tended to survive in the summary, while obscure specifics (a single value buried in an appendix table) usually didn't.
The art lies in deciding what to keep versus what to discard. Overly aggressive compaction can drop subtle-but-critical context whose importance only becomes apparent later. That's why summarization instructions are worth customizing — for instance, explicitly asking the model to preserve every quantitative figure alongside its source.
2. Structured Note-Taking (Memory Files)
Also called agentic memory, this technique has the agent regularly write notes to persistent storage outside the context window, then pull those notes back in when needed in a later session. It's the same pattern behind Claude Code's to-do lists, or a custom agent maintaining something like a NOTES.md file.
The benefit shows up clearly on cross-session tasks. In one experiment, a second session with no access to memory had to re-read all eight source documents from scratch (peaking at 333,977 tokens). A second session that could read the first session's saved notes only needed to read four new documents and relied on memory for the rest — its peak context dropped to roughly 172,623 tokens, with four fewer file reads.
The key trade-off: memory gives you lossless fidelity on whatever the agent chose to save, but its usefulness is entirely bounded by how good the agent's judgment is about what's worth writing down. Sparse or disorganized notes won't help much.
3. Tool Result Clearing
Every time an agent calls a tool — reading a file, hitting an API, running a search — the result gets appended to the conversation and keeps counting against the token budget on every subsequent turn, even long after the model has processed it and moved on.
Tool result clearing replaces old tool-result blocks with a short placeholder, while keeping the record that the call happened (the tool name and its input). If the agent needs that data again, it simply calls the tool a second time. This is the cheapest of the three primitives, since it costs no extra inference — it's just a mechanical edit to the message list.
In testing, a session with no context management peaked at 335,279 tokens, while an equivalent session with clearing enabled stayed capped around 173,137 tokens despite reading the same number of files. The trade-off: old tool results are genuinely gone from context until re-fetched, so this technique works best when re-fetching is cheap (like re-reading a local file) and worse when the underlying call is slow or rate-limited.
Sub-Agents: Separating Context Windows for Long Tasks
Beyond the three primitives above, there's an architectural approach: sub-agents. Instead of one agent trying to hold an entire project's state in a single context window, specialized sub-agents handle focused tasks with their own clean context windows.
The lead agent acts as coordinator with a high-level plan, while sub-agents do the deep exploratory work — potentially burning tens of thousands of tokens or more — but return only a condensed summary (typically 1,000–2,000 tokens) back to the lead agent. The result is a clear separation of concerns: the detailed exploration context stays isolated inside the sub-agent, while the lead agent focuses on synthesizing and analyzing results. This pattern has shown substantial improvements over single-agent systems on complex research tasks.
This sub-agent approach is also relevant for work that spans many sessions — a coding project that runs for hours or even days, for example. In experiments building a complex web app across many context windows, two failure patterns kept showing up: the agent would try to one-shot the entire build and run out of context mid-implementation, or a later session would look around, see partial progress, and prematurely declare the project done.
The fix was a two-role structure: an initializer agent that sets up the working environment in the very first session (a setup script, a structured JSON feature list, an initial git commit), and a coding agent in every subsequent session that's asked to work on exactly one feature at a time, then leave a clear trail — a descriptive git commit and a progress-notes update — before the session ends. That way, a new session can read the git log and progress file to understand exactly where things stand, instead of guessing what happened before.
How to Diagnose Your Context Problem Before Picking a Fix
A common mistake is reaching for all of the above at once without first understanding the actual problem. Each primitive targets a different kind of context growth. Here's how to diagnose it:
| Symptom You're Observing | Likely Cause | Best-Fit Primitive |
|---|---|---|
| Long conversations, lots of back-and-forth dialogue and reasoning | Whole-transcript growth | Compaction |
| Context dominated by large, re-fetchable file/API results | Tool-result bloat | Tool Result Clearing |
| Work spans sessions/days, needs to resume where it left off | No cross-session persistence | Structured Note-Taking (Memory) |
| Deep research or exploration across many sources | Context mixing detailed exploration with synthesis | Sub-Agent Architecture |
A few practical questions worth asking about your own workload:
- Are your sessions short enough that they never naturally approach the context limit? If so, you may not need compaction at all — it's lossy by design (specific details get summarized away), so there's no reason to pay that fidelity cost for headroom you don't need.
- Does your agent genuinely need to see old tool results in full again? An agent doing cross-document analysis that compares passages side by side can't re-fetch its way back to a cleared result fast enough; clearing would just force redundant reads and hurt more than it helps.
- Should every session really start from zero? For a user-facing chatbot where each conversation is meant to be independent, adding memory would carry over state you don't actually want.
Once you know which problem you actually have, the next step is testing configuration — trigger thresholds, how much to retain, summarization instructions — against your real workload rather than generic assumptions. Concrete signals, like how many tokens got freed or what actually survived a summary, are directly observable and should drive your tuning.
Conclusion
Context engineering represents a real shift in mindset: from simply writing a good prompt, to carefully curating which tokens earn a place in the model's limited attention budget at every step. Not every workload needs compaction, memory, and clearing all at once — the useful question isn't "should I use all three?" but "which of these context problems does my workload actually have?"
If you're building an AI-powered application, automation, or agent-based service that starts to feel forgetful or sluggish as task complexity grows, the cheapest first move is to diagnose the actual bottleneck before reaching for a fix — not stacking every technique at once.
References
- Anthropic Engineering, "Effective context engineering for AI agents"
- Claude Cookbook, "Context engineering: memory, compaction, and tool clearing"
- Anthropic Engineering, "Effective harnesses for long-running agents"
