待翻译：Customer Experience (CX) Agents in Production: Lessons from Lyft, Vodafone, and LATAM Airlines

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Conceptual Guide Customer Experience (CX) Agents in Production: Lessons from Lyft, Vodafone, and LATAM Airlines August 4, 2026 15 min Go back to blog Create agents Customer experience has become one of the fastest-moving categories for agents, partly because the ROI is relatively easy to measure. Faster responses can improve conversion, fewer escalations can reduce the cost per contact, and more successful resolutions can help retain customers. As CX agents move into production, the challenge shifts from building them to improving how they operate. Teams are learning from real interactions, refining agent behavior, and deciding when a conversation should become a structured workflow. Increasingly, they are also using those interactions to improve the broader customer experience. The teams furthest along treat agents as production systems that require continuous testing, deployment, monitoring, and iteration. This piece looks at how that approach is taking shape across three companies: Lyft, which built a self-serve platform that allows non-technical operations teams and product managers to configure and launch support agents Fastweb + Vodafone, which built Super TOBi and Super Agent to support both customer-facing conversations and internal call-center teams LATAM Airlines, which built Concierge, its travel-assistance agent, and Compass, a system that turns unstructured conversations into structured signals Drawing on additional examples from Cisco and Podium, we’ll explore the use cases emerging across customer experience, the technical and operational challenges teams encounter in production, and how LangSmith, Deep Agents, and LangGraph support continuous improvement throughout the Agent Development Lifecycle. The emerging patterns in Customer Experience agents Consumer-facing self-service agents are often the most visible starting point. They interact directly with customers through chat or voice, helping with tasks such as billing, account access, claims, and appointment scheduling. Their value is relatively easy to measure: faster responses can improve conversion, while more successful resolutions can reduce escalations and lower support costs. Podium’s AI Employee, for example, responds to inbound leads for car dealerships, HVAC contractors, and other local businesses. For these companies, responding within five minutes produces a 46% higher lead-conversion rate than responding within an hour. Frontline and rep copilots can be an even higher-leverage use case. Rather than speaking directly to customers, these agents work alongside human representatives and surface the next best action. Cisco’s CX organization uses this approach for network engineers. Its system narrows thousands of potential findings down to the handful that matter most, so even a vague request like “help” can be routed toward the right issue. Self-serve platforms emerge when engineering can no longer build every agent. Lyft’s platform allows operations teams and product managers to create a prompt and configuration file, then launch a new support agent without involving a machine learning engineer. Podium built a similar system around the same primitives it uses internally. This lets one underlying architecture support a wide range of use cases, from automotive sales to HVAC warranty support. Semantic routing and triage become critical when customer requests are incomplete or ambiguous. LATAM Airlines saw this with Concierge. Initially, 13% of messages were classified as out of scope. After reviewing the conversations, the team found that 95% were legitimate passenger needs the agent had not yet been designed to handle, including check-in and baggage questions. Adding a customer-care specialist reduced the out-of-scope rate from 13% to 1%. Evals become a shared language across technical and domain teams. As more people contribute to building agents, teams need a consistent way to define what good behavior looks like and determine whether an agent is ready to ship. Evals turn domain expertise into concrete, testable criteria that engineers, product managers, and operations teams can use to review performance and guide improvements. Lyft encountered this after opening agent development to non-engineers. The platform was no longer the primary constraint; prompt and evaluation quality were. The team introduced a structured prompt-writing framework and automated checks to catch contradictory instructions and incomplete conversation paths before they reached production. Together, these patterns show how the work changes once CX agents reach production. The following three teams illustrate how organizations are designing, evaluating, and improving these systems at scale. Three teams with CX agents in production Lyft: Turning support engineering into a self-serve platform Lyft’s AI Assist supports riders and drivers across issues such as account access, damage claims, charge reviews, and earnings disputes. The volume of trips Lyft facilitates necessitates an agentic system for support. Lyft facilitates 79 million trips each month, while AI Assist handles roughly 270,000 monthly interactions across seven or more production agents. The system has achieved a 65% deflection rate and a 35% AI resolution rate. Lyft sets an intentionally high bar for resolution, requiring the agent to solve an issue end to end rather than simply prevent the customer from reaching a human. For complex workflows such as driver damage claims, that can include collecting information and photos, retrieving data through tools, applying fraud signals, making a decision, and explaining the outcome to the driver (all within 15 minutes). Agent architecture Lyft’s current system uses a router-based, multi-agent architecture built on LangGraph. A meta-agent classifies each incoming request and routes it to a specialized subagent, with separate paths for riders and drivers. Each subagent is itself a complete LangGraph state graph registered as a subgraph node. When an intent agent determines mid-conversation that a request requires a more specialized handler (e.g. moving from a general driver-intent agent to a damage-claim agent), it returns control to the meta-agent for rerouting. This prevents the conversation from being forced down the wrong path. Lyft divides its agents into two categories: Specialized agents are built by machine learning engineers for complex, high-stakes workflows, such as damage claims involving image processing and fraud detection Configurable agents are the self-serve layer. They are initialized at runtime using a JSON configuration and a prompt from LangSmith’s Prompt Hub, which can be written by a domain expert rather than an engineer This approach reduced the time required to develop an agent from roughly six months for Lyft’s first driver agent to about two weeks for a new configurable agent. How Lyft builds evals As the platform became easier to use, prompt and evaluation quality started to become bottlenecks. Lyft built an evaluation flywheel that connects development and production. Before launch, the team runs simulated, multi-turn conversations in which an LLM role-plays the customer against the agent. Each simulation is defined around a task, user persona, and environment that reflects what the agent is likely to encounter in production. The resulting trajectory can be evaluated using a combination of code-based assertions and LLM judges, including whether the agent granted the correct concession, escalated appropriately, or resolved the issue within the expected number of turns. The diversity of those offline scenarios is important. Lyft uses offline evaluation as a launch gate, allowing the team to move quickly without treating real customers as test cases. An agent only progresses toward production when it meets the required quality threshold. The team learned early on that generic evaluation metrics weren’t enough. Initial measures such as response helpfulness, conversation naturalness, tool-use appropriateness, and conversation completeness produced scores, but didn’t tell the team what to change. Lyft instead worked with operations and quality experts to build narrow, behavior-specific rubrics based on how support interactions should actually unfold. The team also moved from broad scalar scores to simpler pass-or-fail outcomes. For example, an education rubric checks whether the agent provides useful educational content when it can solve the issue, but escalates once it becomes clear that it cannot. The agent fails if it repeats the same education too many times, escalates before making a reasonable attempt to help, or includes a factual error. A separate escalation rubric defines the expected behavior when a user asks for a human. The agent should push back once, then escalate after a repeated request. It fails if it escalates immediately, refuses to escalate after the second request, escalates before providing necessary information, or continues for several turns after it is clear that it cannot help. These rubrics are more useful than generic quality scores because each failure points toward a specific product, prompt, or workflow change. Lyft also calibrates its LLM judges against human reviewers. The team collects human labels and iterates on each judge until it achieves a sufficiently high agreement rate. This gives the team confidence that automated scores reflect the standards its operations and quality teams would apply themselves. The simulated user requires the same level of calibration. Lyft’s first LLM-generated customers were too articulate, patient, and cooperative, producing offline pass rates above 90% that did not reflect production behavior. Real users often write in fragments, omit context, repeat themselves, or arrive with a specific goal such as securing a refund or bypassing the agent. To make offline evaluation more realistic, Lyft fine-tuned its simulated user on real customer verbatims and introduced personas such as refund seekers, AI skeptics, and users determined to reach a human. Making the simulated customer less polished made the evaluation harder, but also made offline results more predictive of production performance. Once an agent launches, the same evaluation loop continues online. Every invocation is traced in LangSmith across development, staging, and production, including the agent’s reasoning, the educational content it retrieved, and the tools it called. This allows the team to identify whether a failure came from routing, context, tool execution, or the final response. LangSmith also makes the evaluation process accessible beyond the machine learning team. Product managers and operations specialists can define pass-or-fail criteria, write rubrics, and configure LLM judges directly. That brings the people who understand the support experience most deeply into the evaluation process rather than requiring engineers to translate every requirement for them. Lyft has configured automations that send failed production traces into an annotation queue. Product managers and quality reviewers then label the failure mode in free-form language, turning individual bad interactions into structured product insights. Those findings feed back into prompts, workflows, datasets, and future offline tests. What’s next The team is now working toward a more standardized evaluation harness. Today, many offline tests still begin as one-off scripts or notebooks. Lyft wants to replace those with versioned primitives (e.g. tasks, datasets, personas, and scorers) that teams can share and run automatically. That would make it possible to regression-test every prompt change, compare models on the same scenarios, and maintain an evaluation set that grows easily. Over time, Lyft also sees these traces becoming more than evaluation data. Successful trajectories can become supervised fine-tuning examples. The [truncated for AI cost control]