What are the four agent memory types?

Context window (session), RAG (large changing corpus), CAG (stable cached knowledge), and database (structured read/write facts).

When should I use RAG vs CAG?

RAG when the corpus is large and changes often. CAG when knowledge is stable, fits in context, and the agent needs all of it every time.

Why not store everything in a vector DB?

Vector stores are for semantic retrieval, not transactional facts. Order status and user preferences belong in a database with primary keys.

What happens when you pick the wrong memory type?

Slow retrieval, hallucinated answers, runaway token costs, and agents that forget when the session ends.

How do I audit my agent memory?

For every piece of state in your agent code, ask: retrieve from corpus (RAG), need all every time (CAG), write facts (database), or session only (context). Move misplaced ones.

All notes

In this note (7 sections)

Architecture May 29, 2026Updated Jul 6, 2026 9 min

Wrong memory. Dead agent.

Last updated on Jul 6, 2026

Four memory types. Four use cases. Pick wrong and your agent forgets, hallucinates, or costs 10x.

Introduction

The most common failure I audit: RAG everywhere, or context-window-only, or a vector DB storing order status. Each mistake has a predictable symptom. Slow retrieval. Hallucinated answers. Runaway token costs. Agents that forget the moment a session ends.

This note is the four-type decision rule. For the three-paradigm view (implicit, explicit, agentic), see agent memory three types. For where RAG sits in the pipeline, see complete RAG pipeline.

Type 1: Context window (short-term)

Session memory. Free per turn. Gone when the session ends. Use for the current conversation, recent tool results, and the active task.

Do not stuff entire knowledge bases into the context window. You will burn tokens and hit limits by week two. Rule: if it only matters for this turn, keep it in context.

Type 2: RAG (long-term vector)

Retrieve only what is relevant from a large, changing corpus. Company docs, product catalog, support history. Lives in Chroma, Faiss, Pinecone, or pgvector.

Avoid RAG when knowledge is small and stable. A 20-page policy that fits in context does not need a vector index. It only adds latency.

Type 3: CAG (cached context)

Stable knowledge the agent needs all of, every time. Policies, pricing rules, system instructions. Lives in the prompt cache at the model layer. Up to 90% cheaper than uncached input tokens on cache hit.

Avoid CAG when knowledge changes frequently or does not fit in the context window. The content quality pipeline uses CAG for stable 80-item rubrics for exactly this reason.

Type 4: Database (structured)

For facts the agent reads and writes: order status, ticket state, user preferences. Postgres, MongoDB, Redis. Predictable, queryable, transactional.

Storing order status in a vector store means semantic search on a primary key. That is a bug, not a feature. Rule: if the agent needs to write, use a database.

The decision rule

One question, one memory type

Question	Memory type
Retrieve from a large corpus?	RAG
Need everything every time?	CAG
Need to write a fact?	Database
Only matters this session?	Context

Conclusion

Open your agent code today. For every piece of state, ask which of the four types it belongs to. Move the misplaced ones. Your latency and your bill will both thank you.

Key takeaways

1Wrong memory choice kills more agents in production than wrong model choice. There are four memory types and four use cases. Pick wrong and the agent forgets, hallucinates, or costs 10x what it should.
2Short-term memory is the context window: session memory that is free, disposable, and gone the moment the session ends. Use it for the current conversation, recent tool results, and the active task. If it only matters for this turn, keep it in context.
3Long-term vector memory (RAG) retrieves only what is relevant from a large, changing corpus like company docs or support history. Use it when the corpus is large and changes often. Skip it for a small, stable policy that fits in context, where it only adds latency.
4Cached context (CAG) keeps stable knowledge in the prompt cache, up to 90 percent cheaper than uncached input tokens. Use it when the agent needs all of it every time and it fits in context. Avoid it when the data changes frequently or is too large to cache.
5Structured memory is a database, for facts the agent reads and writes: order status, ticket state, user preferences. If the agent needs to write, use Postgres, MongoDB, or Redis, not a vector store. Semantic search on a primary key is a bug, not a feature.
6The decision rule is one question per piece of state. Retrieve from a large corpus means RAG. Need everything every time means CAG. Need to write a fact means database. Only matters this session means context.
7The common failure is using one memory type for everything: RAG everywhere, context-window-only, or a vector DB for rows that belong in a table. That is what produces slow retrieval, hallucinations, and runaway token costs.
8Audit your agent today. For every piece of state, decide which of the four it belongs to and move the misplaced ones. Latency and cost both improve.

Frequently asked questions

Get the visual notes by email

New agentic AI notes and breakdowns, plus what I am shipping for clients, one email on Thursdays.