Every time I see an agent architecture diagram where "memory" is a single box pointing at a vector database, I know the author has not had to operate one past the demo. Memory in an agent is not a single system. It is a stack of five layers, each solving a different problem, and you pick which layers you need based on what the agent actually does.
I am going to walk through the five layers, what each one is good and bad at, and the combination I would ship if you asked me to stand up an operational agent this week. Spoiler: Pinecone is almost never the right answer when you already run Postgres.
The Five Layers
Layer 1: In-Context Scratchpad
This is the default. The LLM keeps its thoughts, actions, and observations in the prompt itself. Zero infrastructure. It works for demos and single-shot tasks. It fails the moment your loop runs 15 iterations or the agent has to remember anything after the session ends. If the pitch you are looking at only has this layer, it is a demo.
Layer 2: Cache
Redis with a TTL of 24 to 72 hours is the usual answer here. This layer holds recent session state, recent tool call results, user preferences that do not belong in the structured store yet. The win is speed. The loss is that a key-value cache is not semantic, so you cannot ask it "have I seen a similar request?", only "what was request ID X?". If you do not already run Redis, do not add it for this. An unlogged Postgres table with an expires_at column covers the same ground with no new service to operate, and you can swap it out the day cache reads are actually your bottleneck.
Layer 3: Structured Store
Postgres. Boring, proven, and exactly right for the data that has obvious columns: incident ID, timestamp, agent version, trigger type, tools called, outcome, cost in tokens. This is your audit log, your analytics source, and the thing your oncall team actually reads when something goes sideways. Any agent you expect to pass an audit needs this layer, because it is the only record of what the agent actually did.
CREATE TABLE agent_runs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
agent_name TEXT NOT NULL,
agent_version TEXT NOT NULL,
trigger_type TEXT NOT NULL,
trigger_payload JSONB NOT NULL,
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
ended_at TIMESTAMPTZ,
status TEXT NOT NULL,
outcome JSONB,
tool_calls JSONB,
token_cost INTEGER,
escalated_to_human BOOLEAN DEFAULT FALSE,
confidence NUMERIC(3,2)
);
CREATE INDEX idx_agent_runs_agent_started
ON agent_runs (agent_name, started_at DESC);
CREATE INDEX idx_agent_runs_trigger
ON agent_runs USING GIN (trigger_payload);Layer 4: Vector Store
This is the layer everyone talks about and most teams over-index on. The job of the vector store is semantic recall: "have we had an incident that feels like this one?" You embed a summary of past incidents, you embed the current situation, and you pull the top K most similar cases into the prompt. Done well, this turns a reactive agent into a learning one.
My strong opinion: if you already run Postgres, use pgvector. It is a Postgres extension, not a separate service. You get semantic search without adding another system to operate, another auth boundary, another place your on-call has to know about at 2 AM. Pinecone, Weaviate, Qdrant are all fine products. They are overkill for most teams.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE agent_memory (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
agent_name TEXT NOT NULL,
summary TEXT NOT NULL,
-- 384 dims matches all-MiniLM-L6-v2, which runs locally for free.
-- Bump this to your provider's dimension if you move to a hosted API.
embedding VECTOR(384) NOT NULL,
source_run_id UUID REFERENCES agent_runs(id),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_agent_memory_vector
ON agent_memory USING hnsw (embedding vector_cosine_ops);
-- Query: top 5 similar past incidents for a given embedding
SELECT id, summary, 1 - (embedding <=> $1) AS similarity
FROM agent_memory
WHERE agent_name = $2
ORDER BY embedding <=> $1
LIMIT 5;Layer 5: Summarization
This one is not storage. It is compression. When a session runs long, or when you pull 20 past incidents from the vector store and they would blow your context budget, you pass them through an LLM to produce a condensed version. This is the layer that keeps your token costs from exploding on long-running agents. Most intro decks skip this because it feels trivial until you see a bill.
How the Layers Actually Interact
A single run for a DBA agent, end to end:
- Trigger fires. Agent wakes up with an alert payload.
- Before calling the LLM planner, query the vector store for similar past incidents. Pull top 5.
- Run those 5 through the summarization layer to compress into a few hundred tokens.
- Pass the compressed memory plus the trigger plus the system prompt into the planner. This is the scratchpad starting state.
- Loop: plan, call tool, observe, repeat. Scratchpad grows.
- At the end, write the structured outcome to Postgres and a semantic summary to pgvector. Next run learns from this one.
No single layer could do all of that. Scratchpad gives you session state. Postgres gives you audit and analytics. pgvector gives you "have we seen this?". Summarization keeps the context from exploding. The cache fills in for hot data you do not want to recompute on every run.
The Stack I Would Actually Ship
The One Mistake I See Most
Teams treat memory as an afterthought. They build the loop, ship it, and then bolt memory on three months later when the agent starts making the same mistake twice. By then the structured log is a JSON blob in S3 with no schema, and you cannot rebuild the history because you never captured it properly.
Design the Postgres schema before you write the first tool call. You do not need to build the vector layer on day one. You do need to capture every run with enough detail that you can backfill the vector store later from the same data.
The agent that remembers is a different product from the agent that reacts. Pick one on purpose.
References
- pgvector: https://github.com/pgvector/pgvector
- Anthropic, Building Effective Agents (agent design patterns): https://www.anthropic.com/engineering/building-effective-agents
- MemGPT paper (hierarchical memory for LLMs): https://arxiv.org/abs/2310.08560