Deep Agents
Conceptual Guide
Why human insight is a strategic advantage for building better agents
November 3, 2026
8
min
Go back to blog
Create agents
Create agents
Key Takeaways
Human expertise defines what good looks like. Domain experts bring the judgment needed to identify quality, explain why an interaction succeeds or fails, and uncover gaps in automated evaluations.
Context is essential to evaluating agents. Reviewing the full conversation, including earlier messages and tool calls, helps teams understand not just what an agent answered, but how and why it got there.
Human feedback becomes the foundation for better evals. Structured scores and reviewer comments can be used to calibrate LLM judges, improve online evaluators, and build golden datasets for continuous testing.
As agents take on longer, more complex tasks, evaluating their performance means looking beyond the final AI-generated answer. Human review plays an important role in agent development: it helps teams define what good looks like, evaluate performance in context, and turn expert judgment into data that drives improvement.
In this blog, we explore three principles for effective human-in-the-loop workflows: defining quality through expert judgment, evaluating agents in context, and turning human feedback into reusable evaluation data that drives continuous improvement.
What is an agent trace?
An agent trace is a record of everything an agent did in a single attempt: user messages, model responses, tool calls, and intermediate steps. Teams use traces to debug behavior, score quality, and decide what to fix.
Often an engineer or an online evaluator flags a trace when something looks off. The flag is a signal, not an explanation. It usually is not enough to know how to change the agent.
The flagged trace goes into an annotation queue. A human reviewer opens the interaction and leaves feedback: did the agent do the right thing, and why or why not? Automation chooses what to review. The human supplies the judgment that builders act on.
In LangSmith, a queue item can be one run (a single step or span) or a thread (the full multi-turn session). With a thread, the reviewer sees the conversation the user had, instead of one step cut out of it.
Principle 1: Human perspective defines what good looks like
Each reviewer brings domain judgment the trace does not contain on its own: what “good” means in that job, what the user was trying to do, and what the agent should have done instead. This expert judgment is often highly specific to the company, role, or individual's expertise, and is a key part of building differentiated agents. Or as Snoop Dogg put it his commencement speech at CSU Long Beach: “Nobody can do you, like you do.”
When a domain expert reviews a trace, one of the most valuable parts of the review is their reason. An expert can say why an output was right or wrong, and how a high-quality agent should behave.
While reviewers can catch what automated evals miss, they also surface problems that sit beside the agent’s answer. For example, they might discover that an eval is scoring the wrong behavior, or that there is a gap in the rubric. One note in a queue can reveal a broader pattern worth fixing in the evaluator itself.
This expert feedback changes the evaluations, and ultimately how the agent is built. In an annotation queue, that perspective is stored on the run or thread as a score plus the reviewer’s reasoning, so builders can turn a single expert review into a clearer definition of quality for the next iteration.
Principle 2: Context is what makes the judgment hold up
A reviewer’s perspective is only as good as the interaction in front of them. If they only see one step from a multi-turn session, they can miss earlier messages, tool calls, and decisions that explain the final answer. That's why comprehensive HITL workflows involve reviewing the full thread.
Threads in annotation queues put the whole session in front of the reviewer. They can see what the user was trying to do, how the agent arrived at its answer, and whether that answer holds up. Editing one sentence is not the same as editing the essay that sentence belongs to.
A reply can look right on its own and still be wrong for something the user said earlier. The mistake may have happened several turns back, but only in the final message did it became obvious. As agents run longer and take more steps, that missing history gets harder to diagnose by hand.
For multi-turn agents, the unit of review should be the conversation. A thread in an annotation queue is that session as one review item, so the reviewer reads the interaction instead of assembling it.
Principle 3: Turn expert judgment into high-quality evaluation data
Building better agents depends on data that reflects what good performance actually looks like. Subject matter experts (SMEs) bring that standard, as builders usually cannot get it from automated metrics alone.
Human review is how that standard gets written down in a form the system can reuse. In practice it looks like this:
A reviewer opens a run or thread in an annotation queue and scores it against a rubric (pass/fail, 1–5, “empathy,” “correct tool,” etc.), often with a short note on why.
That labeled example becomes ground truth for that interaction: not just “bad,” but which dimension failed and what “good” would have been.
Builders use those labels in a few concrete ways:
Align an LLM judge: compare the judge’s score to the human score on the same traces. Where they disagree, update the judge prompt or rubric until they match more often.
Update online evaluators: if humans keep catching the same miss, change the evaluator that should have caught it, then re-run it on new traffic.
Build a golden dataset: promote strong human-labeled examples into a dataset and run offline experiments against them when the agent or prompt changes.
Next time an eval runs, it is scoring against criteria that came from expert review.
When that data is connected to the runs and threads it describes, builders can use it to identify failure modes, improve evaluators, refine rubrics, and build datasets for ongoing testing. The result is a feedback loop where expert knowledge becomes a reusable asset for agent development, not just a score in a review queue
Over time, this creates a reusable foundation for evaluation: expert judgment on a specific thread → structured scores and notes → better evaluators and datasets → scored evals that track the same definition of quality. The annotation is the raw material for evals that the team can run again without asking the expert every time.
Why we built threads in annotation queues
A reviewer does better work with their own expertise, the full interaction in front of them, and a way to leave feedback that a builder can use. Once an agent works across many turns and tools, one trace leaves out the context that judgment depends on.
A trajectory is another view of the conversation. It shows messages and tool calls in the order they happened, with the nested run tree and other duplicate information removed, in a readable view. The queue item is still the thread, because the thing being judged is the conversation. Reviewers read the trajectory as they annotate the thread.
Threads in annotation queues keep the review next to the interaction a person actually had. Domain experts can focus on the content of the conversation without first learning how to read a trace tree. Try out this feature yourself in LangSmith today.
Case Study: Fetch Rewards
Fetch Rewards uses annotation queues to turn human judgment into a reference point for its automated judges. Reviewers assess individual turns of the AI Rewards Assistant - an input and an output - but the transcript is only the starting point. To determine whether a response is good, they often need to investigate further. A fluent answer can still be wrong if it isn’t grounded in real data, so reviewers inspect trace metadata to see which tools the assistant called and whether its answer matches the results.
When a user asks about an offer, for example, they confirm that the assistant searched for offers and that its response accurately reflects the offer returned. They also check product-card images against the products they represent: for example, making sure a card for a 20 oz Coca-Cola doesn’t show a 2 liter bottle. Since a recommendation is useful only if the recipient can actually use it, reviewers look up the user’s offers to confirm that the suggestions were available to them.
Some cases are ambiguous: an image may be almost right, an offer technically valid but unhelpful, or an accurate response may miss the user’s intent. Reviewers weigh the evidence and document what they checked, what remained unclear, and why they assigned each score in an annotation-queue comment. Engineers then use that reasoning to identify where an automated judge’s logic differs from the human review and tune it accordingly.
Make Human Review Part of Your Agent Development Lifecycle
We believe human review is an important part of building better agents, and we've been investing in making it easier to bring that into your workflow with LangSmith.
Try reviewing your agent traces with a human reviewer. You might be surprised by the insights and useful data points you uncover! We'd love to hear what you find. Reach out to us on LinkedIn to share your discoveries.
See what your agent is really doing
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
Try LangSmith
Get a demo