The AI Agent Tech Stack Explained
This article explains the seven-layer production AI agent stack, from foundation models to deployment infrastructure. It covers each layer's function, code examples, and technology choices for prototyping, startup, and enterprise scenarios.
The AI Agent Tech Stack Explained - MachineLearningMastery.com
The AI Agent Tech Stack Explained - MachineLearningMastery.com
[sc name="banner-new-book"][/sc]
In this article, you will learn how the seven layers of a production AI agent stack fit together, from the foundation model down to deployment infrastructure.
Topics we will cover include:
What each layer of the stack does, from the foundation model and orchestration framework through memory, retrieval, tools, observability, and deployment.
How to implement each layer with working code, including a stateful agent, a memory system, a RAG pipeline, custom tools, and tracing.
Which combination of technologies to use at each layer depending on whether you are prototyping, scaling a startup, or operating in an enterprise environment.
Introduction
Picture this: you ask an AI agent to research three competitors, pull the pricing data from each of their websites, summarize the findings into a structured report, and drop it in a Slack channel by 9am. You hit enter. Thirty seconds later, the report is there.
What just happened under the hood is not magic, and it is not one thing. It is seven distinct layers of technology working in sequence, each one handling a specific job, each one capable of breaking in its own specific way. The model at the top gets all the attention. The six layers beneath it are what determine whether the agent actually works.
According to Gartner, 40% of enterprise applications will be integrated with task-specific AI agents by the end of 2026, up from less than 5% in 2025. That is not a gradual curve. That is a near-vertical adoption line, and the engineers and technical leads responsible for those deployments need to understand the full stack, not just the layer they happen to own.
This article goes through each layer in order, from the foundation model down to deployment infrastructure. By the end, you will know what every piece is, why it exists, how the layers connect to each other, and what to actually use at each level.
Layer 1: The Foundation Model
The foundation model is the cognitive core of an agent. It is where reasoning happens, language is understood, and decisions about what to do next are made. Everything else in the stack is either feeding context into it or acting on what it produces.
In practical terms, your main options in 2026 are OpenAI’s GPT-5.5, Anthropic’s Claude Sonnet 4.6 (or Claude Opus 4.8 for harder reasoning), Google’s Gemini 3.1 Pro, and open-weight models like Meta’s Llama 4 and Mistral Large 3. Each has trade-offs worth understanding before you commit.
GPT-5.5 is fast for everyday calls and reliable at tool-calling, and it has the most mature ecosystem of integrations and the widest community of developers who have already run into and solved the edge cases you will encounter. Claude Sonnet 4.6 handles long documents and nuanced instruction-following well at a lower price point than Anthropic’s Opus tier, which matters in document-heavy workflows; reach for Claude Opus 4.8 when a task needs deeper, longer-horizon reasoning. Gemini 3.1 Pro has a 1 million token context window, which is relevant if your agent needs to process large codebases or lengthy knowledge bases in a single pass. Open-weight models like Llama 4 give you full control over deployment and data residency, at the cost of the infrastructure overhead of running them yourself.
There is no longer a hard split between “standard” and “reasoning” model families, the way there was in 2025; OpenAI, Anthropic, and Google have each folded reasoning into a single model that decides how long to think. GPT-5.5 ships with adjustable reasoning effort levels (from none up to xhigh), and the same applies to Claude’s effort parameter and Gemini’s thinking levels. For most agent workflows, the default or low-effort setting is the right choice: fast and cheap. For tasks that require careful planning or mathematical reasoning, dialling the effort level up earns back its cost in correctness.
Layer 2: The Orchestration Framework
If the foundation model is the brain, the orchestration framework is the nervous system. It handles the control flow: deciding what the agent should do next, when it should call a tool, how it should handle the result, and how the whole reasoning loop stays coherent across multiple steps.
The pattern that most frameworks implement is called ReAct (Reasoning and Acting). The agent produces a thought, decides on an action, executes the action through a tool, observes the result, and then thinks again. This loop repeats until the agent produces a final answer. It sounds simple. In practice, it is where most production failures occur: the agent calls the wrong tool, gets stuck in a loop, or fails to recognise when it has enough information to stop.
A cyclical loop diagram of the ReAct loop (click to enlarge)
LangChain is the most widely adopted framework. It offers a large ecosystem of integrations and good documentation. The criticism that it adds too much abstraction is fair at the prototype stage, but less relevant once you need the features that abstraction provides. LangGraph, built by the same team, is better suited for stateful multi-agent workflows where you need fine-grained control over the execution graph. If your agent involves multiple specialists coordinating on a task, LangGraph is the cleaner choice.
CrewAI is designed specifically for multi-agent coordination. It lets you define agents with roles, assign them tasks, and have them collaborate within a structured workflow. It is higher-level than LangGraph and faster to get running, but gives you less control over the execution details. AutoGen, from Microsoft, takes a conversational approach to multi-agent systems. Agents interact with each other through a message-passing interface, which makes the interaction logic very readable.
Semantic Kernel is Microsoft’s enterprise-focused option, with production-ready support for C#, Python, and Java. If you are operating in an enterprise environment already running on the Microsoft stack, it fits naturally. LlamaIndex started as a document ingestion and retrieval framework and has since grown into a full agent framework, with particularly strong support for RAG-heavy workflows.
The right choice depends on what your agent needs to do. For a single-agent task runner: LangGraph or LangChain. For a coordinated team of specialized agents: CrewAI or AutoGen. For enterprise environments: Semantic Kernel. For document-heavy retrieval workflows: LlamaIndex.
Here is a minimal working agent in LangGraph that handles tool use and maintains state.
Prerequisites:
1
pip install langgraph langchain-openai langchain-community python-dotenv
How to run: Save as agent.py, add your OPENAI_API_KEY to a .env file, then run python agent.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
agent.py
Minimal stateful agent with tool use built on LangGraph
Python 3.10+ | LangGraph 0.2+ | LangChain 0.3+
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.messages import HumanMessage
from langgraph.prebuilt import create_react_agent
Load API key from .env file
load_dotenv()
Initialize the language model
temperature=0 for deterministic, focused responses in agentic tasks
llm = ChatOpenAI(
model="gpt-5.5",
temperature=0,
api_key=os.getenv("OPENAI_API_KEY")
)
Register the tools the agent can use
DuckDuckGoSearchRun requires no API key -- good for development
tools = [DuckDuckGoSearchRun()]
create_react_agent from LangGraph wires together the LLM,
tools, and a built-in ReAct loop -- no boilerplate required
agent = create_react_agent(llm, tools)
Run the agent with a sample query
The agent will decide whether to use a tool based on the question
result = agent.invoke({
"messages": [HumanMessage(content="What is the current market cap of Nvidia?")]
})
The final response is the last message in the messages list
print(result["messages"][-1].content)
What this does: create_react_agent handles the full ReAct loop automatically. The agent receives the question, decides it needs current data, calls the DuckDuckGo search tool, reads the result, and synthesizes a final answer. The messages list in the output contains the full trace of that reasoning process.
Layer 3: Memory Systems
Statelessness is the default behavior of any LLM. Every call starts from scratch, with no knowledge of what came before unless you explicitly pass that context in. For a one-shot question, that is fine. For an agent that needs to track a conversation, remember a user’s preferences, or build on work it did yesterday, it is a fundamental problem.
According to Atlan’s research on AI agent memory, 95% of enterprise generative AI pilots delivered zero measurable ROI in 2025, with failure attributed to context readiness rather than model quality. Agents are failing not because the model is wrong, but because the memory layer is not there.
There are four types of memory in a production agent, and each one handles a different job:
Working memory (in-context) is the active context window. It holds the current conversation, any documents you have passed in, and the results of recent tool calls. It is fast and requires no infrastructure, but it is session-bound. When the session ends, it is gone.
Episodic memory is a log of prior interactions. As described in the research on memory types, episodic memory stores what happened: timestamp, task, actions taken, outcome. This is what allows an agent to answer “What did we work on last Tuesday?” or “What did the user say about this project three sessions ago?“
Semantic memory is factual knowledge stored externally, including definitions, entity relationships, and domain-specific facts that the model was not trained on. This is where your RAG pipeline feeds in (more on that in the next layer).
Procedural memory encodes workflows and tool-use patterns, repeatable behaviors the agent should always follow. This lives in the system prompt or a version-controlled instruction file, and it shapes every response the agent produces.
Here is how to implement working and episodic memory together using LangChain’s recommended pattern for LangChain 0.3+:
Prerequisites:
1
pip install langchain langchain-openai python-dotenv
How to run: Save as memory.py, ensure your .env has OPENAI_API_KEY, then run python memory.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
memory.py
Working memory + episodic memory for persistent agent context
Uses the current LangChain 0.3+ pattern (legacy ConversationBufferMemory is deprecated)
import os
import json
from datetime import datetime
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage, trim_messages
load_dotenv()
llm = ChatOpenAI(
model="gpt-5.5",
temperature=0.2,
api_key=os.getenv("OPENAI_API_KEY")
)
── EPISODIC MEMORY STORE ─────────────────────────────────────────────────────
In production, replace this list with a database (SQLite, Postgres, Redis).
The structure here: each episode is a dict with timestamp, user input, and agent response.
episodic_store: list[dict] = []
def save_episode(user_input: str, agent_response: str) -> None:
"""Save a conversation turn to the episodic store."""
episodic_store.append({
"timestamp":
[truncated for AI cost control]