A 2-Tier Memory Architecture for AI Agents: Inside Google Cloud's AlloyDB + Memorystore Guide

LLMs are stateless; enterprise agents are asked to run multi-day, long-horizon workflows. When a user returns days later, the context window is empty, unless your application reconstructs the past. The common first instinct is context stuffing: dump raw chat history and tool logs into the prompt. It works early and breaks at scale, multiplying token costs with every message, dragging simple prompts past 30 seconds, and triggering the model to lose critical instructions buried in the text. On October 2, 2026, Google Cloud published a more engineered answer: a two-tier memory architecture using Memorystore for Valkey for short-term buffers and AlloyDB AI for long-term persistent memory. Here is the architecture, the four memory types, and the benchmark numbers.

Short-term buffer in memory, long-term facts in the database: the division of labor is the point

The problem: stateful workflows vs a stateless model

Per Google Cloud, the core conflict is that agents are used for multi-turn conversations and long-running workflows, while LLMs remain stateless across sessions. When a user comes back days later, the model starts with an empty context window unless the application rebuilds past context using long-term memory. Two common shortcuts both fall short. Context stuffing, dumping everything that fits into the prompt, multiplies token costs with every message, drags response times past 30 seconds for otherwise simple prompts, and causes lost-in-the-middle degradation that buries critical instructions under mountains of text. Rolling summaries, asking an LLM to compress older messages periodically, trim prompt size but are inherently lossy: after a few rounds, subtle but important details get filtered out as background noise, and a few turns later the agent quietly breaks the constraints you set earlier.

# The architecture, in one object. Facts are from Google Cloud's
# blog post dated October 2, 2026.

TIERED_MEMORY = {
    "short_term": {
        "store": "Memorystore for Valkey",
        "role": "token-bounded sliding window of active turns",
        "why": "sub-millisecond, high-throughput lookups every turn",
    },
    "long_term": {
        "store": "AlloyDB AI (PostgreSQL)",
        "role": "facts, preferences and episodic memory across sessions",
        "why": "transactional integrity, governance, hybrid retrieval",
    },
    "reported_effect": "up to 70% reduction in token spend",
}

The two-tier memory architecture

Google Cloud recommends splitting memory into two tiers. The short-term session buffer caches active conversation turns in memory using a token-bounded sliding window, keeping the context window stable across rounds and even across devices while the latest messages stay fresh. This tier needs sub-millisecond, high-throughput lookups on every turn, which makes Memorystore for Valkey well suited to it. The long-term persistent tier stores important facts, user preferences and episodic facts across sessions, and needs transactional integrity, data governance, and hybrid retrieval across relational data and vectors, all of which AlloyDB AI provides. Google Cloud notes that although both tiers could live in a relational database, a two-tier design performs and scales better: active buffers are highly ephemeral and updated on every turn, so handling those rapid writes in an in-memory key-value cache prevents write amplification and table bloat in the primary database.

# The four memory types, mapped to their storage layer and life.
# Splitting short-term context from structured long-term rules is what
# keeps prompts lean without losing constraints.

MEMORY_TYPES = {
    # type:                (what it stores,               tier,        lifespan)
    "buffer":     ("recent raw conversation turns",  "valkey", "active session"),
    "summary":    ("compacted older turns",          "valkey", "multi-turn window"),
    "episodic":   ("past actions, events, tool output", "alloydb", "permanent"),
    "entity_rule":("preferences, constraints, vetoes",  "alloydb", "permanent"),
}

# Rule and constraint recall must not degrade as the conversation grows;
# that is exactly what naive context stuffing breaks.

Offload rapid writes to an in-memory cache to avoid write amplification

The four memory types

To organise long-term state, Google Cloud divides memory into four types. Buffer memory stores recent raw conversation turns in Valkey for the active session. Summary memory stores a compressed history of older turns, known as compaction, also in Valkey across a multi-turn window. Episodic memory stores past actions, events and tool outputs in AlloyDB permanently. Entity and rule memory stores user preferences, constraints and vetoes, also in AlloyDB permanently. By isolating short-term conversation context from structured long-term rules, the agent retrieves relevant context on demand instead of filling token windows with raw interaction logs. That separation is exactly why rule recall stays stable under tiered memory but degrades under context stuffing.

# The read and write paths. Two coordinated flows keep the context
# window bounded while persisting everything that matters.

READ_PATH = [
    "fetch the active sliding window from Memorystore for Valkey",
    "normalise the query in-database with ai.generate",
    "retrieve scoped entity rules + episodic facts via ai.hybrid_search",
]

WRITE_PATH = [
    "cache the finished turn in Memorystore for Valkey",
    "an async worker extracts structured entities from the exchange",
    "write them to AlloyDB, where auto-embeddings run transactionally",
]

# AlloyDB AI: ai.initialize_embeddings (up to 3,000 embeddings/sec),
# ai.generate (run Gemini from SQL), ai.hybrid_search (RRF fusion).

The read and write paths

The system runs across two coordinated paths. On the read path, when a user asks a question, the application fetches the active sliding window from Memorystore for Valkey, runs in-database query normalisation with ai.generate, and queries AlloyDB with ai.hybrid_search to retrieve scoped entity rules and relevant episodic facts. On the write path, after the response is generated the turn is immediately cached in Valkey, and an asynchronous background queue worker extracts structured entities from the exchange and writes them directly into AlloyDB, where transactional auto-embeddings compute and store vector representations. AlloyDB AI embeds these capabilities in the engine: ai.initialize_embeddings generates up to 3,000 embeddings per second and keeps them current within the same transaction using incremental_refresh_mode => 'transactional'; ai.generate runs foundation models such as Gemini from SQL; and ai.hybrid_search executes Reciprocal Rank Fusion in-engine, combining vector cosine similarity with PostgreSQL full-text search in a single call, with metadata filter pushdown.

# The benchmark Google Cloud published across 45+ turns with heavy
# tool execution. Naive context stuffing vs tiered memory.

BENCH = {
    "prompt_tokens_turn_45": {"stuffing": 747_033, "tiered": 83_262},
    "latency_turn_45_s":     {"stuffing": 33.5,    "tiered": 6.7},
    "cumulative_tokens":     {"stuffing": 17_900_000, "tiered": 4_090_000},
    "rule_recall":           {"stuffing": "degrades", "tiered": "stable"},
}

def pct_smaller(old, new):
    return round((old - new) / old * 100, 1)

pct_smaller(747_033, 83_262)     # 88.9% smaller prompt
pct_smaller(17_900_000, 4_090_000)  # 72.0% fewer session tokens

Entity extraction and hybrid retrieval happen inside the database

The benchmark: what it saves

Google Cloud published its internal benchmark across multi-turn development dialogues of 45-plus turns with heavy tool execution. Compared with naive context stuffing, tiered memory produced concrete gains. Active prompt size at turn 45 fell from 747,033 tokens to 83,262, an 88.9% smaller prompt. Turn-45 response latency dropped from 33.5 seconds to 6.7 seconds, 80.0% faster. Per-turn wait time went from 33.5 seconds to 4.2 to 6.7 seconds. Cumulative session tokens fell from 17.9 million to 4.09 million, a 72.0% saving. And rule and constraint recall did not degrade over turns, preserved by ACID guarantees. Google Cloud cautions that the numbers come from a simulated enterprise AI programming assistant workload and that real savings vary with prompt structure, query frequency and data volume.

# Governance you should wire in from day one. Multi-tenant agent
# memory is a security surface, not just a cache.

GOVERNANCE = [
    "index user_id / project_id / scope columns in agent_entities",
    "enforce PostgreSQL Row-Level Security (RLS) across tenants",
    "use Parameterized Secure Views to blunt prompt-injection SQL",
    "run scheduled compaction + time-based partition pruning",
    "keep long-term memory behind the database, not in the prompt",
]

# Google Cloud ships an ADK Memory provider (ADKTieredMemoryProvider)
# and exposes long-term memory to agents as a tool (longterm_memory_tool).

Governance and multi-tenant security

Running agent memory in enterprise production makes security boundaries and access controls mandatory. Google Cloud recommends indexing user, project and scope columns in agent_entities and enforcing PostgreSQL Row-Level Security so memory stores are isolated across departments, teams and users within the same database cluster. Parameterized Secure Views add a deterministic application-level layer of protection against malicious prompts and overly broad SQL. Automated memory lifecycle management matters too: combining scheduled SQL compaction queries with time-based partition pruning keeps database footprint and query latency predictable over time. The implementation path is packaged: extend the default ADK Memory provider, for example ADKTieredMemoryProvider, and expose long-term memory to the agent as a tool such as longterm_memory_tool, with a companion Codelab providing a working build.