Agent systems · Memory

Memory that gets quieter, not lost.

Keep recent conversation exact. Compress older history into a hierarchy. Let the agent open any summary to recover the original detail.

Soumil ChughSeptember 20267 minute read
01 · The problem

A conversation keeps growing. The context window does not.

Long-running agents eventually face a bad choice: compress a large block of history all at once, or drop it.

A hard cutoff creates uneven memory. Recent turns remain detailed while everything before the boundary suddenly becomes vague or disappears. Important decisions, file paths, and user preferences can be lost even though the agent still needs them.

Design goal

Let detail fade gradually with age while preserving every original turn outside the model's context.

02 · The idea

Recent turns stay exact. Older turns become progressively coarser.

The prompt contains a small set of summaries plus the recent turns in full.

The full conversation lives in files. The context contains only the live window and the summaries currently active at each tier. Each summary includes a path, so it also acts as an index into the archive.

03 · Rollups

The same rule repeats at every level.

Choose a window size N. With the default N = 8, a rollup begins when 16 loose turns accumulate. The oldest eight move into a tier-1 directory and a summarizer writes one bounded summary.

The illustration uses four turns for space; the default design uses groups of eight.

When enough tier-1 groups accumulate, the oldest groups roll into tier 2. Tier 2 groups later roll into tier 3. The algorithm is identical at every level and has no fixed maximum depth.

A token safety threshold can trigger an early rollup when one unusually large turn would otherwise overflow the live window.

04 · Retrieval

A summary is a map, not a replacement.

Suppose the agent remembers that an API decision was made months ago but needs the exact endpoint. It follows the path attached to the broad summary, opens the next level, and keeps narrowing until it reaches the original turn.

Ordinary list, read, and search tools are enough to navigate the tree.

The agent can also search the session files directly. This matters because summaries are intentionally lossy: they help locate evidence, while the original turn remains the source of truth.

05 · Scale

Context grows slowly even when the conversation grows dramatically.

The live window remains roughly constant. Only the number of active summary tiers grows, and it grows logarithmically with the number of turns.

Tier depth4
Maximum active summaries64
Maximum summary context32K tokens
N = 8 · summary budget = 500 tokens · plus 8–15 recent turns verbatim

Summarization cost is also amortized. With N = 8, the hierarchy needs about one summarizer call for every seven completed turns over time.

06 · Storage

The directory tree is the memory state.

Write onceEvery completed turn is stored as an immutable file.
Move, never duplicateA turn exists at the root or inside exactly one tier.
Resume from diskLoose turns rebuild the live window; summaries rebuild older context.
Branch safelyA session can fork while immutable history remains shared.

This makes recovery simple: walk the directory tree. A new session starts empty, clearing memory changes the active session rather than deleting history, and a branch creates an independent future from the same past.

07 · Tradeoffs

What this design solves—and what it does not.

It preservesEvery original turn, gradual awareness of old topics, bounded recent context, and an inspectable path to evidence.
It acceptsHigher tiers lose nuance because summaries are summaries. The agent must drill down before relying on an important detail.
It requiresEvaluation of summary faithfulness, rollup correctness, retrieval quality, and whether the agent knows when to inspect deeper levels.
It leaves separateShared memory across agents, cross-session search, and editing old turns. Those need their own designs.
The central idea

Use lossy summaries for orientation and lossless files for truth. The agent carries a compact map, not the entire territory.

More engineering work

Memory is useful when an agent can recover the right detail at the right time.

Explore the portfolio →