How Agent Memory Actually Works

The easiest way to make an agent worse is to give it “memory” without deciding what memory is for.

At first, memory sounds like pure upside. The agent should remember the project, the user, the plan, old decisions, past mistakes, and the command everyone forgets until Friday afternoon.

Then the system starts to sag. Every run loads a growing blob of instructions. Old preferences leak into unrelated projects. A stale summary survives longer than the decision it summarized. A vector search retrieves something semantically nearby and operationally wrong. The agent remembers, yes. It just remembers like a group chat with no pinned messages and too many confident people.

That is when memory stops being a feature and becomes architecture.

Agent memory is not one store. It is a set of decisions about what the agent should load, carry, recall, update, and forget. Storage matters, but it comes later. The first design question is consumption: when does this information need to influence the next action?

For most local-first agents, memory falls into three jobs:

• Load: stable context the agent should see when work starts.
• Carry: active state the agent needs while the current task is alive.
• Recall: durable knowledge the agent can retrieve when a future task calls for it.

Get those three jobs right and the rest of the design becomes less mystical. Get them wrong and even a fancy memory system becomes a filing cabinet tipped into the prompt.

This piece is the map for a short series. The follow-ups can zoom in on each lane: current memory as the agent’s loaded brain, short-term memory as working state without context bloat, and long-term memory as retrieval, records, and recall.

Memory Is A Runtime Contract

The useful mental model is this:

loaded context + active state + retrieved records -> next useful action

That is not a complete agent architecture, but it keeps the memory conversation honest.

Loaded context shapes the run before the agent acts. Active state preserves what is happening now. Retrieved records bring durable information back only when relevant. Together, they form the model’s practical view of the task.

This is also why “memory” is an overloaded word. A project instruction file, a chat session, a task checkpoint, a user preference, an architecture decision, a research note, and a reusable workflow can all be memory. They should not all behave the same way.

A project instruction may be loaded every time. A test failure may matter for ten minutes. A user preference may persist for months but only apply to one user. A publishing workflow may be a reusable procedure, not a fact to blend into every answer.

The architecture is deciding which lane each thing belongs in.

Load: The Context That Starts The Run

Loaded memory is the small body of context the agent receives up front. In a coding agent, this often means files like CLAUDE.md, AGENTS.md, rules files, project briefs, skill descriptions, or user preferences.

Anthropic’s Claude Code docs are a useful concrete example. They describe CLAUDE.md as the place for context you would otherwise re-explain: repeated corrections, code review lessons, project standards, common workflows, and onboarding knowledge. They also distinguish managed, user, and project scopes: memory needs an audience.

Loaded memory should be short, stable, and inspectable. It is a good home for:

• the agent’s role,
• project-level rules,
• coding standards,
• safety constraints,
• preferred commands,
• workflow boundaries,
• source-of-truth pointers.

It is a poor home for full transcripts, old scratch work, noisy research dumps, and “maybe relevant someday” material. Always-loaded memory has a tax. It spends tokens, competes for attention, and can conflict with newer task context.

Claude Code’s memory docs also make another useful distinction: memory files can be audited and edited, and some files are read on demand rather than loaded at startup. Not every durable instruction deserves a front-row seat in every run.

One more trap: loaded memory is usually advisory context, not enforcement. If something must happen every time, make it deterministic. A hook, test, lint rule, permission check, or CI gate is stronger than a sentence saying “remember to do this.”

A healthy loaded-memory file looks more like this:

Project: Coffee With Humans
Audience: software engineers learning practical AI engineering.
Draft boundary: internal drafts stay private until approved.
Voice: source-driven, clear, relaxed, no hype.
Publishing rule: never publish without explicit confirmation.
Pointers: use docs/content-catalog.md and docs/editorial-guide.md.

That is useful startup context. A full article draft and every image prompt ever generated are not.

Carry: The State Of The Current Work

Carried memory is what keeps the current task coherent. It answers: what are we doing, what has happened so far, what decisions are still active, and what should happen next?

OpenAI’s Agents SDK docs describe sessions as a persistent memory layer for runs. Reusing a session lets later turns receive prior conversation history and persist new items. The docs also cover editable history, custom session storage, resumable human-in-the-loop flows, and compaction.

That is carried memory. It is not the agent’s identity or durable knowledge. It is the state of this thread, task, run, or approval flow.

LangChain’s memory docs draw a similar boundary: short-term memory is normally scoped to a thread and can include conversation history plus stateful data such as uploaded files, retrieved documents, or generated artifacts. The same docs warn that long histories become costly, slow, and distracting even when they technically fit.

This is where many agents lose the plot. They preserve too much transcript and too little state.

The model usually does not need every false start. It does need:

• the current goal,
• relevant constraints,
• decisions made,
• evidence behind those decisions,
• files or systems touched,
• tool results that still matter,
• unresolved questions,
• the next action.

That is why compaction is not just “make the chat shorter.” Good compaction preserves the working state. Bad compaction preserves the conclusion and loses the reason.

For an engineering task, carried memory might be a session history, but it might also be a task file:

## Goal
Rewrite the agent memory article around a stronger decision model.

## Active Decisions
- Replace object analogy with load/carry/recall.
- Keep the article distinct from the context-engineering piece.
- Preserve OpenAI, Claude Code, LangChain, and survey sources.

## Evidence To Preserve
- Sessions support persistent run history and compaction.
- Claude Code uses scoped memory files and editable memory.
- LangChain separates thread-scoped short-term memory from long-term memory.

## Next Action
Rewrite the draft, then check for old analogy residue.

Notice what this does not do: preserve every paragraph we rejected. It carries the state needed to continue the work.

Recall: Durable Knowledge For Later

Recalled memory is what survives beyond the active task and comes back only when it is useful.

LangChain’s docs describe long-term memory as information retained across conversations or sessions, often saved in scoped namespaces. They also use a helpful vocabulary:

• semantic memory: facts and concepts,
• episodic memory: past experiences and outcomes,
• procedural memory: reusable instructions or ways of doing things.

These are design clues, not storage products.

“The user prefers TypeScript examples” is semantic memory. “The last deploy failed because generated image paths changed” is episodic memory. “How to publish a Coffee With Humans article” is procedural memory. Each wants different retrieval, update, and review behavior.

LangChain’s Deep Agents docs make the local-first version concrete with filesystem-backed memory. An agent can load memory files at startup or read them on demand. Skills can act as procedural memory: a short description is visible first, and the full skill loads only when relevant. Keep durable knowledge organized, searchable, and scoped; do not preload it all.

This is where vector databases belong in the story, but they are not the story.

Vector search is useful when similarity helps find relevant notes, examples, or documents. But long-term memory may also need exact lookup, recency filters, permissions, namespaces, version history, or human review. Some memories should be retrieved by keyword, some by project, some by user, and some not at all after they expire.

The 2026 survey paper Memory for Autonomous LLM Agents frames memory as a write-manage-read loop. That is a more useful engineering shape than “put it in a vector DB.” The system must decide what gets written, how contradictions are resolved, when records are recalled, and how stale or sensitive memories are forgotten.

The write path is especially important. If an agent can write memory automatically, it can write mistakes automatically. “This migration failed under Node 22 on Windows” is useful. “Never use Node 22” may be cargo cult with a timestamp.

Durable memory should carry provenance:

What is this record?
Where did it come from?
Who or what wrote it?
When was it last confirmed?
Which user, project, or agent may use it?
When should it expire?
How can a human edit or delete it?

Without those fields, long-term memory becomes vibes with persistence.

The Sorting Test

Before choosing a store, classify the memory by how it should be consumed:

1. Should this shape almost every run?
   Load it, but keep it short. Use this for stable project rules, identity, boundaries, and high-value pointers.

2. Is this only needed while the current task is alive?
   Carry it. Use session state, checkpoints, task files, scratchpads, compact summaries, or resumable run state.

3. Could this help future work only when relevant?
   Store it for recall. Give it scope, metadata, retrieval rules, and a path back into context.

4. Is it a fact, an episode, or a procedure?
   Facts, past experiences, and reusable workflows age differently. They should not share the same update rules by default.

5. What happens if this memory is wrong?
   If the blast radius is high, add verification, review, expiry, or deterministic enforcement.

This catches both under-design and over-design. Not every memory needs embeddings, a graph, a reranker, and a little ceremony. Sometimes a markdown decision log is enough. Sometimes the best “memory” is a failing test that prevents the agent from forgetting reality.

What Breaks When The Layers Blur

Layer confusion is the root of many agent-memory failures. If recall records are loaded every time, the prompt bloats and stale context competes with the current task. If active task state is treated as durable knowledge, temporary guesses become future truth. If project rules live only in chat history, they disappear after compaction. If user preferences are global instead of scoped, one user’s taste quietly becomes everyone’s default.

There is also the privacy problem. Memory that crosses sessions can cross boundaries. A local agent that remembers personal preferences, client details, codebase decisions, and tool credentials needs clear scopes.

The point is not to make memory timid. Useful agents need memory. The point is to make memory legible. You should be able to inspect what was loaded, what is being carried, what was recalled, what got written, and why.

The Last Sip

Agent memory works when it is designed around use.

Load the small, stable context that should shape the run. Carry the active state needed to finish the current work. Recall durable knowledge only when it helps the next action. Then design the write, update, review, and forget paths with the same care as retrieval.

That is the foundation for the rest of this series. We can go deeper on loaded project memory, short-term working state, and long-term recall. But the core move stays the same:

• Current memory: what the agent should load at the start of the run.
• Short-term memory: what the agent should carry while the current task is alive.
• Long-term memory: what the agent should store, retrieve, review, and forget across tasks.

Do not ask where memory should be stored first. Ask when the agent should use it.

Sources On The Counter

• OpenAI Agents SDK Sessions is useful for understanding session-backed agent memory, resumable runs, and history compaction.
• OpenAI Agents SDK Python Memory shows memory as an explicit interface for reading, adding, removing, and clearing conversation history.
• Anthropic’s Claude Code memory docs explain scoped project memory, CLAUDE.md, auto memory, audit/edit flows, and why loaded memory should stay concise.
• LangChain’s memory overview separates short-term thread state from long-term memory and lays out semantic, episodic, and procedural memory.
• LangChain Deep Agents memory docs make file-backed persistent memory and on-demand loading concrete.
• Memory for Autonomous LLM Agents provides a broader research view of memory as a write-manage-read lifecycle with real governance and reliability problems.