LangSmith LLM Gateway: Runtime Controls for Production Agents

LangSmith LLM Gateway: runtime controls for production agents

July 30, 2026

7

min

Go back to blog

Create agents

Key Takeaways

Avoid provider lock-in — one gateway for closed and open weight models, centralized controls, and the flexibility to change model providers

Control cost and reliability at runtime — set spend and rate limits, track usage, and configure model fallbacks across any model, provider, or host

Protect sensitive data with redaction — redact sensitive data like PII and secrets from model calls to prevent sensitive data from being sent to model providers

LangSmith LLM Gateway is now available in public beta.

Agents in production need runtime controls over model calls to avoid undesirable outcomes like cost overruns and outages. Spend caps, rate limits, model fallbacks, and sensitive data protections are all important controls, but without the right tooling, engineers have to write these controls manually within each agent for a given model provider.

That’s why we built LangSmith LLM Gateway. It’s the central governance layer that sits between your agents and the models they call, designed specifically for the controls that agent engineering teams need in production. LLM Gateway gives teams one place to enforce runtime controls across agents, models, and providers, helping them consistently govern model usage while avoiding vendor lock-in.

Try the public beta today

Why agents need a centralized governance layer

In production, the model layer can quickly become the source of costly mistakes and customer-facing outages. Here’s what this looks like in practice: your model provider has suffered an outage, and now your customer service agent is returning errors, leading to customer frustration and lost revenue. Or your coding agent gets stuck in a retry loop overnight, and by morning it’s made 10,000 LLM calls and you have a four-figure invoice.

In these cases, it’s not enough to understand what happened after the fact. You need to enforce policies before violations happen. But if controls are embedded in each agent’s code, every new agent and model provider adds another implementation path to maintain. A model gateway lets teams define policies once, then enforce them consistently across models and agents.

LLM Gateway controls

LLM Gateway provides the controls that teams need first when agents go into production: cost controls, rate limits, model fallbacks, and sensitive data handling.

Avoid cost overruns: LLM Gateway lets you monitor and control LLM spend for internal and external users before it becomes a problem. You can set time-bound caps and track spend at four levels:

Organization level — for company-wide controls

Workspace level — to keep teams within spend limits

API key level — useful for controlling spend on a specific project

User level — for individual spend controls

When a cap is hit, the agent receives a 402 response with a clear error.

Limit runaway traffic: Apply rate limits at the organization, workspace, user, and API key levels to prevent accidental abuse or traffic spikes from overwhelming provider limits and degrading reliability for other agents or teams.

Per-customer policies for spend and rate limits: For multi-tenant environments and resellers, LLM Gateway gives you customer-level control without customer-specific API keys. Use a custom request header to identify each end customer or team, then enforce separate spend caps and rate limits while routing all LLM calls through one API key. You can update limits and check spend across customers from one place, instead of building and maintaining that logic yourself.

Improve agent reliability: Define fallback rules across models and hosts, then enforce them for all applications. When a provider is down, rate-limited, or unavailable, agents can route to another model without custom fallback logic in each application.

Reduce sensitive data exposure: Detect, redact, and replace PII and secrets in requests before they reach model providers. LLM Gateway also redacts sensitive data from the traces it records in LangSmith. Currently, this feature is available only to Enterprise plan users.

Trace LLM Gateway events in LangSmith: Gateway events are logged as metadata in LangSmith traces, so you can monitor how controls behave with production traffic. When a request is blocked by a spend limit, rate limit, or other control, you can open the trace to inspect what happened in context and decide whether anything needs to change.

Preserving model choice

LLM Gateway provides a single interface for routing calls between your agents and model providers. It supports popular providers like OpenAI, Anthropic, and Fireworks and custom models with OpenAI- or Anthropic-compatible endpoints.

LangSmith Gateway is BYOK-first. You can bring your own provider keys and use LLM Gateway for routing, governance, and policy controls without adding extra instrumentation to every LLM call. For teams that want simpler procurement or a faster way to start building agents, Gateway also supports hosted inference for open models powered by Fireworks, available directly through LangSmith using Gateway Credits. Fireworks provides fast, efficient inference for open models, helping teams bring them into production through LangSmith LLM Gateway.

We believe open weight models are important because they give teams more control over AI systems. They give teams flexibility to optimize for cost, quality, latency, privacy, and deployment needs, and to tune models around the specifics of their business. Gateway is one way we’re making open models easier to adopt in production.

Open models are already delivering the performance teams need for production AI. The next step is making them easier to adopt in agent workflows, where teams need control over cost, latency, and governance. Fireworks makes open model inference fast, efficient, and reliable at scale, and we’re excited to partner with LangChain on LangSmith LLM Gateway to help teams bring that performance into production.

Lin Qiao, CEO, Fireworks

How teams are using LLM Gateway in production

Teams are already using LLM Gateway to put hard limits and shared controls around production agents. Some common patterns include public-facing agents that need spend caps, and multi-customer agents that need centralized billing, usage tracking, and per-customer limits.

LLM Gateway helped us centralize billing and cost tracking for our Deep Agents, giving us visibility into usage and limits across each of our customers. The integration was straightforward and gives us the observability we need to scale our agents.

‍Sean Rich, Cofounder and CTO, Blueberry AI

Every production agent needs a hard spend cap, especially public-facing agents, and the LLM Gateway takes that risk off the table by making sure costs never exceed the limits we set. Now we can sit down with a customer, align on spend and any other policies it supports, and have it all configured in a few clicks.

‍Hylke Sybesma, AI Engineer, Friday Digital Agency

Getting started with LLM Gateway

You can call LLM Gateway directly via API or use it with coding harnesses like Claude Code, Codex, and Deep Agents Code (dcode).

To get started, point your agents at the LangSmith Gateway endpoint, authenticate with your LangSmith API key, and add your provider API keys to workspace secrets. Then, configure your controls for fallbacks, rate limits, spend limits, and sensitive data handling. Update your base_url, and keep the remainder of your code as is.

Where LLM Gateway is going next

We started LLM Gateway with the controls we’ve seen teams need most. Next, we’re expanding that foundation with more ways to govern model calls in production.

Broader guardrails — more coverage for the risks that come with agents calling external models and interacting with production systems.

CI/CD controls — tools for deploying prompts and context, running A/B tests, managing blue-green and shadow deployments, and defining controls with Infrastructure-as-Code.

Try it today

LLM Gateway is available now in public beta for Plus and Enterprise plans.

Get started by logging into LangSmith or signing up, then selecting “LLM Gateway” in the sidebar. You can find setup details in our docs.

Enterprise plan users can request access to Data Protection controls here.

See what your agent is really doing

LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.

Try LangSmith

Get a demo