Teaching AI Agents Better Retrieval: Beyond Keyword Search

Agent retrieval is the process a coding or research agent uses to find the right information before it reasons, writes, or acts. When retrieval is weak, even strong models miss crucial evidence, rank the wrong sources, or answer with partial context.

MMichael SwindellJuly 21, 202616 min read

Agent retrieval is the process a coding or research agent uses to find the right information before it reasons, writes, or acts. When retrieval is weak, even strong models miss crucial evidence, rank the wrong sources, or answer with partial context. That gap between what a model could do with the right documents and what it actually does with poor search is often the real bottleneck. For teams building agentic systems, better retrieval design matters as much as model choice. Platforms like Levr matter here because agents need shared context, scoped tasks, and observable search behavior, not just a bigger prompt.

At a Glance

Better retrieval is often the fastest way to improve agent quality.

Core problem: Modern models reason well, but often fail to access the right knowledge at the right time.
Key concept: The Oracle gap measures the difference between perfect retrieval and actual retrieval performance.
Main cause: Agents often write keyword-heavy queries suited for grep or BM25, even when better semantic tools are available.
Practical fix: Use a multi-tool search agent with clear tool selection, semantic query framing, limited search rounds, and ranking-based rewards.
Why it matters: In legal, finance, and deep research workflows, one missed clause or amendment can invalidate the whole output.

Why do strong models still fail on knowledge-heavy tasks?

Strong models often fail on knowledge-heavy tasks because reasoning quality has improved faster than retrieval quality. In many cases, the model is capable of answering correctly, but it never sees the right evidence. The problem is access, not intelligence.

This is a useful framing shift. Teams often blame hallucination, model weakness, or poor prompting when an agent misses something important. But in many retrieval-heavy tasks, the deeper issue is simpler. The agent cannot reliably surface the right documents from a noisy corpus.

That matters most outside straightforward coding workflows. Legal research, financial analysis, and deep document review depend on finding subtle, high-value evidence inside large collections. If retrieval misses the right clause, filing, or amendment, the final answer can look polished while still being wrong.

This tracks with broader industry adoption patterns. The Stack Overflow Developer Survey 2024 shows widespread use of AI tools, but also persistent concerns around accuracy and trust. Retrieval quality is one of the clearest reasons those concerns remain.

What is the Oracle gap in agent retrieval?

The Oracle gap is the difference between a model's performance with perfect supporting documents and its performance with real retrieval tools. It reveals how much quality is lost because the agent cannot find the right evidence on its own.

This concept is valuable because it separates model capability from retrieval failure. If a model answers well when given the right context, but poorly when forced to search a corpus, then search is the bottleneck.

In the source material, this gap shows up on two knowledge-heavy benchmarks. One is a browsing-style benchmark with complex queries over a fixed corpus of 100,000 documents. The other is built around a large treasury-related corpus and asks difficult historical questions. In both cases, model performance drops sharply once retrieval is introduced.

The implication is practical. Before switching models or adding more prompt complexity, ask whether your agent is simply underpowered at search.

Why do agents write bad search queries?

Agents often write bad search queries because they are biased toward tools and patterns learned from code search and web search. That leads them to produce keyword piles instead of clean semantic requests, which weakens better retrieval systems.

There are three main reasons this happens.

Code-search bias: Many agents are trained around repository exploration patterns, where grep-style matching works well. That encourages literal token hunting.
Human web-search bias: Agents imitate common search engine behavior, which often means short lexical strings rather than structured semantic intent.
Benchmark bias: Some retrieval benchmarks favor entity-heavy, keyword-centric queries that reward lexical overlap more than richer semantic search.

The result is predictable. Give an agent a strong semantic retrieval system, and it may still interact with it as if it were a basic keyword matcher. Then the tool looks weak even when the real problem is tool misuse.

Why doesn't grep or BM25 fully solve agent retrieval?

Grep and BM25 are useful, but they are not enough for many knowledge tasks. They work best when the agent already knows the exact terms to match. They break down when meaning matters more than surface overlap.

This is the key misconception behind a lot of retrieval debates. Keyword tools are excellent for exact matches, identifiers, field names, and code symbols. They are not enough when the evidence is paraphrased, spread across documents, or expressed with different language than the query.

That is why deep research tasks expose the limits of purely lexical search. A hidden contract clause, an amended quarterly result, or a semantically related policy note may not share the exact words the agent guessed. If the system only rewards overlap, the agent misses the signal.

According to GitHub Octoverse, AI-assisted software development is becoming routine. As these systems take on more non-code knowledge work, retrieval quality becomes more important, not less.

How can you design an agent that uses retrieval well?

A good retrieval agent needs multiple search tools, explicit search planning, limited but flexible exploration, and a way to rank evidence once enough support is found. The goal is not endless searching. It is efficient evidence gathering.

One effective pattern is a short search loop with strong tool specialization. In the source material, the agent uses four main tools:

Overview search: A wide semantic pass that returns many chunks as summaries so the agent can understand the corpus without filling context.
Main semantic search: A narrower retrieval step that returns full payloads for top results.
Filter chunks: A metadata-based tool that sorts or narrows by facets.
Grep: A lexical match tool for exact terms or patterns.

The loop stays deliberately small. The agent gets a maximum of four search rounds, but each round can run parallel searches. That gives enough exploration without creating slow, expensive trajectories.

The agent also starts with two useful inputs: an initial semantic search result on the original query, and hints about available metadata facets. That preview gives the agent a rough map of the corpus before it plans deeper searches.

How do you get agents to write better semantic queries?

Agents write better semantic queries when you change the task framing. Instead of asking for a search query, ask for a concise sentence describing the evidence the agent wants to find. That shifts the model away from keyword stuffing.

This is one of the most practical ideas in the source material. Query quality improves when the agent must first state what evidence it needs. That forces an intermediate reasoning step before retrieval.

Several design choices support this:

Goal framing: The agent explains what evidence it is trying to uncover before it searches.
Tool-aware prompting: Semantic search is used for concepts and aspects, while grep is reserved for exact matches.
Sentence-based query writing: The agent writes a natural-language description of the target evidence, not a bag of keywords.
Prompt examples: A few examples show good semantic query form and how to split a larger question into separate aspects.
Corpus preview: Initial results help the agent match its language to the domain vocabulary it sees.

Together, these changes teach the agent that not every retrieval tool should be treated like grep.

How should a retrieval agent decide between semantic search and grep?

Use semantic search when the agent needs concept-level matches, paraphrases, or broader topic exploration. Use grep when the task requires exact words, names, or patterns. Good retrieval systems do not pick one tool. They route between them deliberately.

This split is easy to state but often poorly implemented. Many agents default to one retrieval pattern across every task. That creates obvious failure modes.

A better decision rule looks like this:

Use semantic search for: policy meaning, clause intent, related findings, conceptual summaries, and multi-document evidence discovery.
Use grep for: exact phrases, identifiers, dates, statute numbers, company names, or specific terms the agent already knows matter.
Use metadata filters for: date ranges, authors, document type, source, region, or any facet that can cut noise cheaply.

When teams run multiple coding agents across tasks, this kind of routing becomes operational, not theoretical. One way to manage it is through an agent-first control plane like Levr's workflow model, where intent, task scope, verification, and handoff are visible instead of hidden inside one long prompt.

Make retrieval behavior observable before it becomes a reliability problem.

If your agents search across large code and knowledge corpora, a shared control plane helps you track what they searched, what they found, and what still needs human review.

See how teams structure agent workflows

What kind of training helps a search agent improve?

Search agents improve when training targets strategy, not just answer quality. That means rewarding better tool choice, better queries, useful exploration, and sensible ranking. The model should learn how to search, not only how to respond.

The training approach described in the source material has two stages.

Supervised fine-tuning: A smaller model learns from a larger teacher model.
On-policy reinforcement learning: The agent improves through a custom search reward tied to retrieval and trajectory quality.

The choice of a small model is intentional. The objective is a search specialist that is fast and cheap, not a giant general-purpose model doing everything.

This is a useful design lesson for engineering teams. Not every agent in your stack should be the largest model available. A narrow, optimized search planner can be more useful than a slow all-purpose agent that searches badly.

What rewards actually matter for retrieval training?

Useful retrieval training rewards both the final ranked evidence and the path the agent took to get there. If you only reward the final answer, the agent may keep using sloppy search behavior that does not generalize.

In the source material, the reward has two components.

Retrieval reward

This measures the quality of the final ranked list using retrieval metrics such as NDCG, plus model-based judging. The judging checks whether the returned chunks are relevant, whether the set covers what matters, and whether the ranking is plausible.

Trajectory reward

This evaluates how the agent searched. It looks at query quality, efficiency, and exploration behavior. It also checks whether semantic queries are written as natural sentences and whether the amount of exploration is too little or too much.

This distinction matters in production systems. You want agents that are reliably good searchers, not agents that occasionally stumble into the right answer.

How much search exploration is enough?

Enough exploration means covering distinct aspects of the question without wasting rounds on redundant searches. The best search agents are neither shallow nor exhaustive. They are selective, parallel, and evidence-driven.

A practical approach is to split a complex request into a small number of separate search intents. In the source material, the agent breaks the task into up to four focused queries and can run them in parallel within a round.

That structure solves two common problems:

Under-exploration: The agent locks onto one interpretation too early and misses adjacent evidence.
Over-exploration: The agent keeps searching after it already has enough support, which adds cost and noise.

Once enough evidence is gathered, the agent stops and submits a ranked result set. That stop condition is as important as the search itself.

How do you apply these retrieval ideas in agentic development workflows?

Apply these ideas by treating retrieval as part of your delivery system, not just part of a prompt. Agents need scoped context, tool routing, verification, and memory across runs if you want search behavior to improve over time.

For developer teams, the connection is straightforward. Coding agents do not only search code anymore. They search issues, specs, docs, test history, architecture notes, and incident context. Weak retrieval turns into missed requirements, repeated work, and fragile pull requests.

One way to handle this in practice is with a shared control plane such as Levr. A typical pattern is:

Define intent in natural language: capture the task and expected outcome clearly.
Agents pick up scoped work: the retrieval problem is constrained by project and issue context.
Automated verification runs: search outputs can be checked against tests, criteria, or review gates.
Human review closes the loop: people validate whether the surfaced evidence actually supports the change.

When multiple agents need to share context or route work across tasks, Levr becomes relevant because message passing, shared memory, and routing affect retrieval quality too.

What mistakes should teams avoid when building retrieval agents?

Teams should avoid forcing one search method onto every task, rewarding only final answers, overfilling context windows, and hiding retrieval behavior from operators. Most retrieval failures are system design failures before they are model failures.

The most common mistakes include:

Using one tool for everything: semantic search, grep, and metadata filters serve different purposes.
Letting agents write unchecked keyword soup: this weakens semantic systems immediately.
Skipping corpus preview: agents search better when they see initial corpus structure and vocabulary.
Returning duplicate chunks: repeated evidence wastes context and reduces exploration quality.
Training only on answer accuracy: without trajectory rewards, bad search habits persist.
Ignoring efficiency: retrieval systems need to be fast and cheap enough for repeated production use.

If your team cannot inspect search trajectories, tool choices, and ranked evidence, debugging agent quality becomes guesswork.

Technical Deep Dive FAQ

What is retrieval in an AI agent?

Retrieval is the part of an agent workflow that finds relevant information before the model answers or takes action. That information can come from documents, code, tickets, reports, or any other indexed corpus. In practice, retrieval decides what context the model sees. If retrieval is weak, even a strong model can fail because it reasons over incomplete or noisy evidence instead of the best supporting material.

What is the difference between reasoning and retrieval?

Reasoning is how the model interprets information, combines facts, and produces an answer. Retrieval is how it gets the information in the first place. A model can be strong at reasoning but still fail on real tasks if it cannot find the right documents. That is why many agent failures come from search quality rather than model intelligence. The model is capable, but the evidence pipeline is poor.

What is the Oracle gap in practical terms?

The Oracle gap is the performance difference between a model given ideal supporting documents and the same model forced to retrieve those documents itself. In practice, it shows how much task quality is being lost to bad search. If the gap is large, the obvious optimization target is retrieval. That is often cheaper and faster than switching to a larger model or rewriting your entire prompt stack.

Why do semantic search systems fail with agent-generated queries?

Semantic search often fails because agents interact with it using keyword-centric query habits learned from code search, web search, or benchmark bias. Instead of describing the meaning of what they want to find, they dump tokens, names, and fragments into the system. That can work for lexical tools, but it weakens concept-based retrieval. The failure is not always the embedding system. Often, the failure is mismatched query formulation.

When should an agent use grep instead of semantic search?

An agent should use grep when it needs exact matches, such as identifiers, literal phrases, dates, names, or structured text patterns. Grep is also useful late in a search trajectory, once the agent already knows the exact string worth verifying. Semantic search is better earlier in exploration, when the agent is still mapping the space or when the relevant evidence is phrased differently from the original request. The best systems support both.

Why limit search rounds in an agent loop?

Limiting search rounds keeps the agent efficient and reduces the chance of noisy, repetitive exploration. A small cap forces the system to plan better, split the task into distinct intents, and stop once enough evidence exists. Unlimited search often looks powerful but creates latency, cost, and context bloat. A few rounds with parallel searches usually produce a better tradeoff between recall, speed, and ranking quality.

What is NDCG, and why is it used for retrieval evaluation?

NDCG stands for Normalized Discounted Cumulative Gain. It is a ranking metric used to measure how well a system orders relevant results near the top of a list. It matters in retrieval because the best documents should appear early, where the agent will actually use them. A retrieval system that finds relevant chunks but buries them deep in the ranking can still perform badly in practice. Good ranking is part of good retrieval.

Why train a smaller model for search instead of using a larger one?

A smaller specialized model can be the right choice when the objective is fast, cheap, repeatable search behavior. Retrieval planning does not always need the most capable general-purpose model. If the system is trained specifically on tool choice, query writing, and ranking, a compact model can perform well while reducing cost and latency. That makes it easier to run retrieval loops frequently in production workflows.

How do metadata filters improve agent retrieval?

Metadata filters reduce search noise before the model spends context on full document content. If the corpus includes facets such as date, source, author, or document type, the agent can narrow the space cheaply before deeper semantic or lexical search. This is especially useful in large enterprise corpora where relevance often depends partly on structured constraints. Filters are not a replacement for search. They are a force multiplier for it.

How do you make retrieval behavior observable in production?

Make retrieval observable by logging the original task, tool choices, query text, returned chunks, ranking decisions, and stop conditions. You also need to know which evidence actually influenced the final answer. In agent-first environments, this usually works best when retrieval is attached to tracked tasks and verification steps rather than buried inside raw chat history. That makes debugging, auditing, and improving agent performance far easier across teams.

Can shared memory improve retrieval across multiple agents?

Yes, shared memory can improve retrieval when agents need to build on prior searches, hand off context, or avoid repeating work. If one agent has already found key documents or learned the useful vocabulary of a corpus, that information can help later agents search more efficiently. This becomes even more important in multi-agent systems. Levr is relevant here because routing and shared memory affect retrieval quality over time."

Key Takeaways

Retrieval quality is often the hidden limiter on agent performance.

Teaching agents to search well requires better systems, not just better prompts.

Diagnosis: If a model performs well with perfect context but poorly with live search, retrieval is your bottleneck.
Design: Multi-tool search, corpus previews, sentence-based semantic queries, and bounded exploration produce better behavior.
Training: Reward both final ranked evidence and the search trajectory that produced it.
Operations: Treat retrieval as observable infrastructure tied to tasks, verification, and handoffs.

Better agents do not just think better. They find better evidence.