The Complete Guide to Tool Selection in AI Agents
A comprehensive guide on how to maintain tool selection accuracy in AI agents as the tool catalog scales, covering techniques like gating, retrieval-based selection, semantic routing, planning, fallback logic, and benchmarking.
The Complete Guide to Tool Selection in AI Agents - MachineLearningMastery.com
The Complete Guide to Tool Selection in AI Agents - MachineLearningMastery.com
[sc name="banner-new-book"][/sc]
In this article, you will learn why agent accuracy degrades as a tool catalog grows, and six practical techniques for keeping tool selection accurate and efficient at scale.
Topics we will cover include:
Why adding more tools to an agent causes tool hallucination and accuracy loss, not just slower responses.
How gating, retrieval, routing, and planning each narrow down what the model sees before it has to choose a tool.
How to build fallback logic and a benchmark harness so you can measure whether any of these fixes actually worked.
None of this requires a bigger model, just a smarter view of what the model sees before it acts.
The Complete Guide to Tool Selection in AI Agents
Click to enlarge
Introduction
You build an agent with five tools. It works flawlessly in the demo. Three months later, it has 40 file operations, CRM access, Slack, a calendar, and three different search APIs you bolted on for different teams. The same agent that nailed every demo now calls the wrong tool, hallucinates parameters borrowed from a different tool’s schema, or stalls mid-task waiting on a call that should never have been made.
Nothing about the model changed. The tool list did. This is not an edge case you’ll eventually run into. It’s the default trajectory of every agent that ships and then grows. Research analyzing MCP tool descriptions across the ecosystem has found that a high number contain at least one quality issue, and production benchmarks show agent accuracy degrading measurably once tool counts pass roughly 10 to 15. The RAG-MCP paper, published in May 2025, put hard numbers on the fix: retrieval-based tool selection more than tripled tool selection accuracy from 13.62% to 43.13% while cutting prompt tokens by over half on the same benchmark tasks.
Tool selection isn’t a minor implementation detail you patch later. It’s the architectural decision that determines whether an agent survives contact with a real tool catalog. This guide covers six techniques that solve it, in the order you’d actually deploy them: gating, retrieval, routing, planning, fallback logic, and the benchmark that tells you whether any of it worked.
Why Tool Selection Breaks at Scale
Every tool definition — its name, description, and parameter schema — gets sent to the model on every single request, whether that tool gets used or not. With 50-plus tools, this can consume 5 to 7% of the model’s context before the user’s actual message arrives, crowding out the conversation history and reasoning space the task actually needs.
The “lost in the middle” effect compounds this. Models recall information at the start and end of a context window far more reliably than information buried in the middle. With dozens of near-identical tool definitions stacked in sequence, the one tool that’s actually right for the job often sits exactly in that dead zone, overlooked not because the model can’t reason about it, but because attention is structurally pulled elsewhere.
The second failure mode is worse: tool hallucination. When an LLM’s attention spreads across too many similar-sounding tools, it either invents tool names that don’t exist or calls the correct tool while filling in arguments borrowed from a different tool’s schema. This is a hard failure. There’s no “slightly wrong” way to call a nonexistent function.
OpenAI documents a hard ceiling of 128 tools per agent, but real degradation shows up well before that limit; most production teams see accuracy drop noticeably once they cross 15 to 20 tools in active rotation. The fix isn’t a bigger context window. It’s controlling what the model sees in the first place.
Gating: Deciding Whether a Tool Is Needed at All
Before you optimize which tool to pick, ask a cheaper question first: does this turn need a tool at all? A meaningful fraction of agent turns are purely conversational: “thanks,” “what do you mean by that,” a follow-up clarification. Running full retrieval and tool-selection reasoning on every single turn means paying the full agentic overhead even when the answer is “no tool needed.”
A gate is a fast, cheap classifier — sometimes a small model call, sometimes just pattern matching — that runs before anything expensive does.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
gate.py
Prerequisites: none beyond Python's standard library (re)
Run: python gate.py
import re
CONVERSATIONAL_PATTERNS = [
r"^\s*(thanks|thank you|thx|ok|okay|cool|got it|sounds good|sure|great)\b",
r"^\s*(hi|hello|hey|good morning|good evening)\b",
r"^\s*what do you mean\b",
r"^\s*can you (clarify|explain that)\b",
]
ACTION_KEYWORDS = [
"send", "create", "search", "find", "look up", "schedule", "book",
"read", "write", "query", "summarize", "translate", "check",
]
def gate(query: str) -> dict:
"""
Cheap pre-filter that decides whether the full tool-selection pipeline
needs to run at all. Short-circuits conversational turns before
retrieval, routing, or planning ever fires.
"""
q_lower = query.strip().lower()
Tier 1: regex match against known conversational patterns -- near-zero cost
for pattern in CONVERSATIONAL_PATTERNS:
if re.match(pattern, q_lower):
return {"tool_needed": False, "reason": "conversational_pattern", "tier": 1}
Tier 2: if there's no action verb and the message is short, likely no tool needed
has_action_keyword = any(kw in q_lower for kw in ACTION_KEYWORDS)
if not has_action_keyword and len(q_lower.split()) tool_needed={result['tool_needed']} ({result['reason']})")
How to run (no dependencies required):
1
python gate.py
This costs almost nothing and catches a meaningful share of turns before they reach the expensive part of the pipeline. The threshold for “is this worth building” is low: if even 20–30% of your turns are conversational, gating pays for itself immediately in both latency and token cost.
Retrieval-Based Tool Selection
This is the technique with the strongest published evidence behind it. Instead of sending every tool definition on every call, you index tool descriptions in a vector store, embed the incoming query, retrieve only the top-K most relevant tools, and send just those to the model.
The RAG-MCP framework is the reference implementation of this idea, using semantic retrieval to identify the most relevant MCP tools for a query before the LLM ever sees the full catalog. The reported numbers are not subtle: tool selection accuracy rose from 13.62% with the full catalog exposed to 43.13% with retrieval-filtered selection, more than tripling accuracy, while cutting prompt tokens by over 50% on the same benchmark tasks.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
retriever.py
Prerequisites: pip install sentence-transformers faiss-cpu numpy
Run: python retriever.py
import numpy as np
from sentence_transformers import SentenceTransformer
import faiss
TOOL_CATALOG = [
{"name": "search_web", "description": "Search the web for current information on any topic"},
{"name": "read_file", "description": "Read the contents of a file given its path"},
{"name": "write_file", "description": "Write or overwrite content to a file at a given path"},
{"name": "send_email", "description": "Send an email to a recipient with subject and body"},
{"name": "create_calendar_event", "description": "Create a new calendar event with a title, date, and time"},
{"name": "query_database", "description": "Run a SQL query against the company database"},
{"name": "list_github_issues", "description": "List open issues in a GitHub repository"},
{"name": "create_github_pr", "description": "Create a pull request on a GitHub repository"},
{"name": "send_slack_message", "description": "Send a message to a Slack channel or user"},
{"name": "get_weather", "description": "Get current weather conditions for a city"},
{"name": "translate_text", "description": "Translate text from one language to another"},
{"name": "summarize_document", "description": "Summarize a long document into key points"},
{"name": "lookup_stock_price", "description": "Get the current stock price for a ticker symbol"},
{"name": "book_flight", "description": "Search and book a flight between two cities"},
{"name": "create_invoice", "description": "Generate an invoice for a customer with line items"},
]
class ToolRetriever:
"""
Embeds tool descriptions once at startup and indexes them in FAISS.
At runtime, embeds the incoming query and returns only the top-K
most relevant tools -- not the full catalog.
"""
def init(self, tools: list[dict], model_name: str = "all-MiniLM-L6-v2"):
self.tools = tools
self.model = SentenceTransformer(model_name)
descriptions = [f"{t['name']}: {t['description']}" for t in tools]
embeddings = self.model.encode(descriptions, normalize_embeddings=True)
IndexFlatIP = inner product search, which equals cosine similarity
when vectors are normalized -- the standard setup for this use case.
self.index = faiss.IndexFlatIP(embeddings.shape[1])
self.index.add(np.array(embeddings, dtype=np.float32))
def retrieve(self, query: str, top_k: int = 3) -> list[dict]:
query_emb = self.model.encode([query], normalize_embeddings=True)
scores, indices = self.index.search(np.array(query_emb, dtype=np.float32), top_k)
return [
{**self.tools[idx], "score": float(score)}
for score, idx in zip(scores[0], indices[0])
]
if name == "main":
retriever = ToolRetriever(TOOL_CATALOG)
queries = [
"What's the weather like in Lagos today?",
"Can you check if there are any open bugs in our repo?",
"Send a message to the engineering channel about the deploy",
]
for q in queries:
results = retriever.retrieve(q, top_k=3)
print(f"\nQuery: '{q}'")
for r in results:
print(f" {r['name']} (score={r['score']:.3f})")
How to run:
1
2
pip install sentence-transformers faiss-cpu numpy
python retriever.py
Only the top-3 tools out of a 15-tool catalog get sent to the model per query, an 80% reduction in tool definitions on every call, and the accuracy lift compounds because the model is now choosing between a handful of genuinely relevant candidates instead of scanning past a dozen near-misses.
Semantic Routing
Routing is retrieval’s lighter cousin, and it fits a different shape of problem. Retrieval answers “which specific tool” out of a flat list. Routing answers “which toolbox” — useful when your tools cluster naturally into categories (data, communication, scheduling) and you want to load only the relevant category’s tools rather than re-ranking the entire catalog every time.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
router.py
Prerequisites: pip install scikit-learn numpy
Run: python router.py
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
CATEGORIES = {
"data": ["query the database", "read a file", "write data to storage", "run a SQL query"],
"communication": ["send an email", "post a slack message", "notify the team", "send a message"],
"scheduling": ["create a calendar event", "book a meeting", "schedule an appointment"],
}
class SemanticRouter:
"""
Routes a query to a tool category using similarity against category
centroid embed
[truncated for AI cost control]