Why the Context Window Is Always the Problem
Any engineer who has run an AI agent on a long-horizon task — a large codebase migration, a multi-document research project, or anything spanning several sessions — eventually hits the same wall: a context window that fills up. The issue isn't only raw token capacity. Research into context rot shows that as the number of tokens in a context window grows, a model's ability to accurately recall information from that context degrades, and this happens well before any hard token limit is reached. A model's attention behaves like a finite budget, and every new token drains it a little more.
For an agent running in a loop — reading documents, calling tools, writing notes — the volume of potentially relevant data keeps accumulating with every turn. This is where context engineering comes in: not just writing a good prompt, but actively curating which set of tokens gets to occupy the context window at each step.
This article covers four practical primitives engineers use to handle this on long-horizon workloads, plus how to figure out which one your workload actually needs.
1. Compaction: Summarizing a Session So the Next Window Starts With What Matters
Compaction is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new window with that summary as the starting point. It's a whole-transcript operation: user messages, assistant responses, tool calls, tool results, and even prior compaction summaries all get flattened into one dense summary.
The art of compaction lies in the selection: what gets kept versus what gets discarded. Overly aggressive compaction can drop subtle but critical context whose importance only becomes clear later. In Claude Code's implementation, for example, the model preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs.
In an experiment using a research agent that reads eight lengthy review documents (roughly 320K tokens total), compaction cut the peak context from around 335K tokens down to about 169K tokens. Testing what actually survived in the generated summaries revealed a consistent pattern: high-level facts central to the task — such as organism lifespan figures or major comparative conclusions — tended to survive, while obscure specifics like a single statistic buried in an appendix table usually didn't.
One important detail: custom compaction instructions completely replace the default summarization prompt rather than supplementing it. So if you write your own instructions, you're responsible for the full framing — for example, explicitly telling the model to preserve every quantitative figure alongside its source.
2. Structured Note-Taking: Writing Durable State Outside the Context Window
Structured note-taking, or agentic memory, is a technique where the agent regularly writes notes to persistent storage outside the context window, then pulls them back in at a later point. This provides persistent memory with minimal overhead — similar to Claude Code maintaining a to-do list, or a custom agent keeping a NOTES.md file.
Unlike compaction and tool clearing, which operate on the currently active window, memory solves the cross-session problem. When a brand-new session starts with an empty context window, compaction and clearing don't help at all — memory is what bridges that gap.
In a two-session test on the same research agent: a second session with no memory access had to re-read all eight source documents from scratch, peaking at roughly 334K tokens of context. A second session with access to the first session's saved notes only needed to read the four documents not yet covered, since it pulled the existing comparative findings straight from the memory files — peak context dropped to about 173K tokens with four fewer file reads.
The quality of what memory gets you depends heavily on the agent's own judgment about what's worth saving. A few strategies that help in practice: giving explicit topical guidance about what kind of information should be recorded, instructing the agent to keep its memory directory organized (renaming or deleting files that are no longer relevant), and running a dedicated initializer session at the start of a multi-session project to set up the memory structure before substantive work begins.
3. Tool Result Clearing: Dropping Stale Observations Without Losing the Reasoning Trail
Every time an agent calls a tool, the result gets appended to the conversation as a tool_result block. That block keeps counting against the token budget on every subsequent turn, even long after the agent has processed it and moved on. For tools that are re-callable — file reads, API queries, searches — carrying the verbatim result forward is often unnecessary.
Tool result clearing replaces old tool_result blocks with a short placeholder, while the preceding tool_use block stays intact. That means the model still knows it made the call and with what input, but the bulky payload itself is gone. It's a sub-transcript operation — it only touches tool results, leaving user messages, the agent's own reasoning, and the record of the call untouched.
This is the lightest-touch and cheapest of the primitives: no inference cost, just a mechanical edit to the message list. In testing, clearing held the peak context around 173K tokens versus a baseline that climbed to 335K, with four separate clearing events each freeing roughly 160K tokens.
The trade-off: older file-read results are genuinely gone from context. If the agent needs that content again, it has to call the tool again — cheap for a local file read, considerably less cheap for a slow or rate-limited API. One technical detail worth noting: clearing invalidates cached prompt prefixes, so a minimum-tokens-cleared setting is worth configuring to make sure enough tokens get cleared each time to justify that cache invalidation.
4. Initializer Agent Plus Incremental Worker: The Long-Running Harness Pattern
For tasks spanning hours or even days across many context windows, compaction alone isn't enough. Experiments building a production-quality web app with a coding agent surfaced two consistent failure patterns: first, the agent would try to do too much in one shot, running out of context mid-implementation and leaving the next session to inherit a half-built, undocumented feature. Second, in later sessions, an agent that noticed some progress had already been made would often declare the project finished prematurely, while large parts of it still didn't actually work.
The fix is a two-part harness pattern. An initializer agent runs on the very first session with a specialized prompt to set up the environment: an init.sh script to start the development server, a progress log file (such as claude-progress.txt) tracking what's been done, a structured feature list in JSON (rather than Markdown, since models tend to be more careful about not accidentally corrupting JSON) with every feature initially marked as failing, and an initial git commit.
An incremental coding agent then runs on every subsequent session, instructed to work on exactly one feature at a time and leave the environment in a clean state afterward: committing to git with a descriptive message, updating the progress file, and only marking a feature as passing after genuine end-to-end testing — not just a unit test or a quick command-line check. Every new session follows the same three opening steps: check the working directory, read the git log and progress file to understand recent history, then pick the highest-priority unfinished feature from the feature list.
This pattern effectively gives the agent a form of collective memory across sessions without carrying the entire conversation history forward — the same underlying spirit as structured note-taking, but applied specifically to ongoing software development work.
How to Diagnose Which Primitive Your Workload Actually Needs
These four primitives address different kinds of context growth, so the first step is identifying which problem your workload actually has, rather than stacking every primitive on by default.
Here's a practical diagnostic map:
| Symptom in your workload | Relevant primitive | Watch out for |
|---|---|---|
| Long back-and-forth conversation, reasoning text piling up | Compaction | Specific details can be lost in the summary |
| Large, re-fetchable tool results (file reads, API calls) dominating context | Tool result clearing | Cleared data must be re-fetched if needed again |
| Work spans multiple separate sessions | Structured note-taking / memory | Quality depends entirely on what the agent chooses to save |
| Large project needs hours of work across many context windows, risk of half-finished features | Initializer + incremental worker | Needs explicit structure: feature list, progress log, end-to-end testing |
Once you've picked a primitive, measure its effect concretely: track the token trajectory turn by turn, log when compaction or clearing actually fires, and compare re-read counts or peak context across sessions. In the research-agent experiments above, combining all three context primitives at once (compaction, clearing, memory) pushed the peak context from a 335K-token baseline down to roughly 170K — but it also added more knobs to tune and more interactions to reason about.
That doesn't mean every workload needs all of them. A chatbot where every conversation is meant to be independent doesn't need cross-session memory. Sessions that are naturally short and never approach the context limit don't need compaction, since its lossy nature only trades away fidelity for no real benefit. And an agent that genuinely needs to compare passages from multiple tool results side by side can be hurt by clearing, since it forces repeated, redundant re-fetches.
Conclusion
There's no single primitive that's the best choice across the board. Compaction shrinks the whole window once it's grown too large, clearing drops stale, re-fetchable data from inside the window, memory moves information outside the window so it survives across sessions, and the initializer-plus-worker pattern gives explicit structure to projects spanning many context windows. The right question isn't which primitive is most sophisticated, but which specific context problem your workload actually has right now. Start there, then measure the result directly against your token trajectory and your agent's output quality.
References
- Anthropic Engineering, Effective context engineering for AI agents, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- Claude Cookbook, Context engineering: memory, compaction, and tool clearing, https://platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools
- Anthropic Engineering, Effective harnesses for long-running agents, https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
