The AI Agent Stack Has a Missing Layer — And It's Costing You Context
Your AI agents are smarter than they have ever been. They can reason through multi-step problems, generate production-quality code, hold nuanced conversations, and operate autonomously across complex workflows. The tooling around them has matured at breakneck speed: safety guardrails filter harmful content before it reaches users, evaluation frameworks catch regressions before they ship, orchestration layers chain tools and models into sophisticated pipelines, and routing gateways let you switch between providers without rewriting a single line of code.
And yet, every time a session ends, your agent wakes up with total amnesia.
It does not remember the user's preferences from yesterday. It does not recall the architectural decisions your team made last week. It cannot build on the patterns it discovered in previous runs. Every session is a cold start. Every interaction begins from zero. Every piece of hard-won context evaporates the moment the connection closes.
This is not a minor inconvenience. It is a structural gap in the modern AI agent stack — one that undermines the value of every other layer you have built.
This article is a tour of that stack. We will walk through four essential layers — safety, evaluation, orchestration, and routing — examining what each does well and where each falls short. Then we will look at the layer that ties them all together: persistent memory. Because until your agents can remember, they cannot truly learn.
The Safety Layer: NeMo Guardrails
Before an AI agent interacts with the world, it needs boundaries. NVIDIA's NeMo Guardrails is an open-source toolkit designed to give you programmable safety controls over LLM-powered applications. It sits between your users and your models, enforcing policies that keep conversations on track and outputs within acceptable limits.
NeMo Guardrails provides five distinct types of rails, each targeting a different phase of the interaction lifecycle:
- Input rails filter and validate user messages before they reach the LLM, catching prompt injection attempts and off-topic requests.
- Dialog rails control the flow of multi-turn conversations, ensuring agents follow predefined interaction patterns.
- Retrieval rails validate and filter content pulled from external knowledge bases during RAG workflows.
- Execution rails monitor and constrain tool calls and actions the agent attempts to perform.
- Output rails check the LLM's responses before they reach the user, catching hallucinations, toxic content, and policy violations.
The toolkit ships with built-in mechanisms for jailbreak detection, hallucination prevention, fact-checking against trusted sources, and content moderation. Its secret weapon is Colang, a domain-specific language purpose-built for designing conversational guardrail flows. Colang lets you define safety policies as readable, maintainable scripts rather than tangled prompt engineering hacks.
NeMo Guardrails integrates with multiple LLM providers and slots cleanly into existing application architectures. For teams building user-facing AI products, it is an essential layer.
But here is the gap. NeMo Guardrails operates entirely in the present tense. It filters the current request and validates the current response. It has no mechanism for remembering past safety incidents, learning from repeated policy violations, or adapting its behavior based on historical context. Every request is evaluated in isolation. If your agent encountered a sophisticated jailbreak attempt yesterday, the guardrails have no memory of it today. The safety layer protects each session — but it cannot learn across sessions.
The Evaluation Layer: DeepEval
Building AI agents without testing them is like shipping code without a CI pipeline — technically possible, professionally reckless. DeepEval, built by Confident AI, brings rigorous evaluation to LLM applications. Often described as "Pytest for AI," it is an open-source framework that lets you write unit tests, integration tests, and regression tests for your agents the same way you test traditional software.
What sets DeepEval apart is the depth of its metrics. The framework provides research-backed evaluation criteria for a wide range of agent behaviors:
- RAG quality metrics measure faithfulness, contextual relevance, and answer correctness for retrieval-augmented generation pipelines.
- Agent-specific metrics evaluate tool use accuracy, task completion rates, and multi-step reasoning quality.
- Conversational metrics assess coherence, knowledge retention, and engagement across multi-turn interactions.
- Safety metrics detect bias, toxicity, and harmful outputs using LLM-as-a-judge approaches.
DeepEval supports synthetic dataset generation, letting you stress-test your agents against edge cases you might never encounter organically. It integrates directly into CI/CD pipelines, so evaluation becomes part of your deployment workflow rather than an afterthought. And it works with the major LLM ecosystems — OpenAI, Anthropic, LangChain, and LlamaIndex are all supported out of the box.
For prompt optimization, DeepEval provides tools to systematically improve your prompts based on evaluation results, closing the loop between testing and iteration.
But here is the gap. Evaluation is transient by design. DeepEval runs tests, produces results, and those results live in your terminal output or CI logs. There is no built-in persistence layer for evaluation data. If you want dashboards, historical trends, or cross-run comparisons, Confident AI offers a paid cloud companion — but the open-source framework itself does not retain state. Your agents get tested rigorously, but the insights from those tests do not accumulate. Patterns that emerge across hundreds of evaluation runs are invisible unless you build your own storage and analysis pipeline on top.
The Orchestration Layer: LangChain
When your agent needs to do more than answer questions — when it needs to call APIs, query databases, chain multiple LLM calls together, or coordinate across tools — you need an orchestration framework. LangChain has become the dominant player in this space, providing a modular architecture for building complex LLM-powered applications.
LangChain's core strength is model interoperability. It provides a unified interface across dozens of LLM providers, so you can swap models without rewriting application logic. Its component library covers the full spectrum of agent capabilities:
- Document loaders and text splitters for ingesting and chunking knowledge bases.
- Vector stores and retrievers for building RAG pipelines.
- Tool abstractions that let agents interact with external APIs and services.
- Chain and agent primitives for composing multi-step workflows.
- Prompt templates and output parsers for structured interactions.
The broader LangChain ecosystem extends the core framework significantly. LangGraph adds support for stateful, multi-actor workflows with cycles and branching — critical for agents that need to loop, retry, or coordinate with other agents. LangSmith provides observability, tracing, and debugging tools for understanding what your agents are actually doing in production. The community has built hundreds of integrations, connectors, and extensions.
For teams building AI applications that go beyond simple chat, LangChain provides the scaffolding to connect models, tools, and data sources into coherent pipelines.
But here is the gap. LangChain provides memory abstractions — conversation buffers, summary memory, entity memory — but these are primarily designed for within-session context management. True cross-session persistence requires external infrastructure. You can wire up a database, configure a vector store, and build the plumbing yourself, but LangChain's core does not deeply integrate persistent memory as a first-class concern. The framework orchestrates what happens during a run. What happened in previous runs is your problem. Agents built with LangChain can be extraordinarily capable within a session and completely blank at the start of the next one.
The Routing Layer: LiteLLM
In production, you rarely use a single LLM provider. You might route complex reasoning to Claude, code generation to GPT-4, and simple classification tasks to a cheaper model. You need to manage API keys, track spending, handle rate limits, and fail over gracefully when a provider goes down. LiteLLM, built by BerriAI, solves this problem with a unified API gateway that sits in front of 100+ LLM providers.
LiteLLM exposes an OpenAI-compatible interface, meaning your application code talks to one endpoint regardless of which model is handling the request behind the scenes. This abstraction unlocks powerful capabilities:
- Virtual keys let you manage access and permissions without distributing raw API keys to every team member or service.
- Spend tracking gives you real-time visibility into costs per model, per team, per project — critical when LLM spending can spike unexpectedly.
- Load balancing distributes requests across multiple deployments of the same model, maximizing throughput and reliability.
- Fallback routing automatically redirects requests to backup providers when the primary is unavailable or rate-limited.
Performance is a genuine strength. LiteLLM reports P95 latency of approximately 8ms at 1,000 requests per second — the gateway adds negligible overhead to your LLM calls. It also supports emerging interoperability standards, including the A2A (Agent-to-Agent) protocol and MCP (Model Context Protocol) integration, positioning it well for the multi-agent future.
For teams running LLM workloads at scale, LiteLLM is the traffic controller that keeps everything running smoothly and cost-effectively.
But here is the gap. LiteLLM is a routing layer, not a state management layer. It excels at getting the right request to the right model at the right price, but it has minimal support for conversation memory or stateful session management. If a user talks to Claude through LiteLLM today and GPT-4 through LiteLLM tomorrow, there is no shared context between those interactions. The routing layer moves data through the stack efficiently — but it does not remember what it moved.
The Missing Layer: Persistent Memory with Engram
You now have the full picture of the modern AI agent stack. You can guard your agents against harmful inputs and outputs. You can evaluate their performance with research-backed metrics. You can orchestrate complex multi-step workflows across tools and data sources. You can route requests to the optimal provider at the best price. Each layer is mature, well-engineered, and essential.
And none of them remember anything.
This is the gap that Engram fills. Engram Memory Community is a self-hosted persistent memory system purpose-built for AI agents. It gives your agents the ability to store, search, recall, forget, consolidate, and connect memories across sessions — turning every interaction from an isolated event into part of a continuous, evolving relationship.
Three-Tiered Recall Architecture
Engram's recall system is engineered for speed across different access patterns:
- Hot-Tier Cache: Sub-millisecond lookups powered by ACT-R cognitive principles. Frequently accessed memories surface instantly, mirroring how human memory prioritizes recent and relevant information.
- Multi-Head Hash Index: O(1) candidate retrieval for known memory patterns. When your agent has seen something before, the hash index finds it without scanning the entire memory store.
- Hybrid Vector Search: 768-dimensional cosine similarity combined with BM25 sparse ranking via Reciprocal Rank Fusion. This dual approach catches both semantic matches ("things that mean the same thing") and lexical matches ("things that use the same words"), delivering more accurate recall than either method alone.
The result: repeat queries resolve in approximately 25ms, and novel queries complete in approximately 190ms. The system literally gets faster the more your agent uses it.
Bridge and Overflow: The Differentiators
Two capabilities set Engram apart from anything else in the ecosystem:
Bridge is an MCP (Model Context Protocol) integration that connects Engram's memory system to the tools your agents already use. Claude Code, Cursor, Windsurf, VS Code — Bridge makes memory portable. Your agent's accumulated knowledge travels with it across development environments, not locked into a single tool or workflow. This is not a theoretical integration; it is how Engram is designed to be used in practice.
Overflow provides automatic cloud backup when local memory capacity is exceeded. When your agent's memory store grows beyond what your local infrastructure can handle, Overflow compresses memories using TurboQuant compression, deduplicates redundant entries automatically, and syncs them to a cloud tier — all without manual intervention. Your agent never hits a memory ceiling, and your local-first privacy posture is maintained for active memories.
Seven MCP Tools
Engram exposes its full capability set through seven MCP tools that any compatible agent can call:
memory_store — Save a new memory with metadata and importance scoring
memory_search — Find relevant memories using hybrid vector + keyword search
memory_recall — Retrieve memories by context with cognitive-inspired ranking
memory_forget — Remove outdated or incorrect memories
memory_consolidate — Merge related memories into coherent summaries
memory_connect — Create explicit links between related memories
memory_feedback — Reinforce or decay memory importance based on usage
These tools give agents fine-grained control over their own memory — not just passive storage, but active memory management that mirrors how effective human memory works.
Privacy-First Architecture
Engram is self-hosted and local-only by default. Your memories live on your infrastructure, processed by your compute, with zero data leaving your network unless you explicitly enable the optional cloud tier. There are no API costs for the core memory system — you run it on your own hardware. For teams and individuals who cannot send proprietary context to third-party services, this is not a nice-to-have. It is a requirement.
Engram vs. Alternatives
| Engram | Mem0 | Zep | |
|---|---|---|---|
| Cost | Free, self-hosted | Free tier + paid cloud | Free tier + paid cloud |
| Privacy | Local-only by default | Cloud-dependent | Cloud-dependent |
| Scaling | Overflow with TurboQuant compression | Cloud-tier dependent | Cloud-tier dependent |
| Performance | ~25ms repeat / ~190ms novel | Varies by tier | Varies by tier |
| MCP Integration | Native Bridge support | Limited | Limited |
| Self-Hosted | Full feature set locally | Partial | Partial |
| Cognitive Model | ACT-R inspired recall | Standard vector search | Standard vector search |
How Memory Supports Every Other Layer
Persistent memory is not a standalone concern. It is the connective tissue that makes every other layer in the stack more effective:
- Guardrails + Memory: Safety policies that persist and adapt. When your guardrails detect a new attack pattern, memory ensures that knowledge informs future protection — not just for this session, but permanently.
- Evaluation + Memory: Test results and learned patterns that compound over time. Instead of evaluating each deployment in isolation, your agents build an institutional memory of what works and what does not.
- Orchestration + Memory: Agents that maintain context across complex workflows. A multi-step pipeline that spans hours or days can pick up exactly where it left off, with full awareness of decisions made in earlier stages.
- Routing + Memory: Session state that survives provider switches. When LiteLLM routes a conversation from one model to another, memory ensures continuity. The user never notices the handoff because the context travels with the conversation.
The Complete Stack
The modern AI agent stack is not a single tool. It is a layered architecture where each component handles a distinct concern:
| Layer | Tool | Purpose |
|---|---|---|
| Safety | NeMo Guardrails | Protect inputs and outputs |
| Evaluation | DeepEval | Test and measure quality |
| Orchestration | LangChain | Compose tools and workflows |
| Routing | LiteLLM | Direct traffic across providers |
| Memory | Engram | Remember everything that matters |
Each layer is essential. Remove safety, and your agents become liabilities. Remove evaluation, and quality degrades silently. Remove orchestration, and complexity becomes unmanageable. Remove routing, and costs spiral. But remove memory, and every other layer operates in isolation — repeating work, losing context, and starting from scratch with every session.
Memory is what turns a collection of capable but disconnected tools into a coherent, learning system. It is the layer that makes your agents not just powerful, but intelligent over time.
Engram Memory Community is open source, self-hosted, and ready to run. Pull the Docker image, start the container, and your agents have persistent memory in minutes.
Get started: Pull the Docker image, run docker compose up -d, and your agents have persistent memory in minutes.