From prototype to production: High QPS for Databricks AI Search

From prototype to production: High QPS for Databricks AI Search | Databricks Blog

Skip to main content

Real-time search bars, recommendation and personalization systems, and entity resolution can run at peak production traffic without managing replicas, sizing nodes, or building load balancers in front of the index

Scale endpoints to thousands of QPS with a single configuration parameter, generally available today

Built-in production observability: endpoint QPS, latency, and health visible in the AI Search UI for every endpoint

Search is everywhere:  product discovery on retail sites, voice queries on smart TVs, recommendations in every feed, and identity matching during account lookups. Each fires a request per page view, per keystroke, per user action. At consumer scale, that translates into thousands of queries per second hitting the search index, with peak traffic often several times higher.

Search also sits on the critical path of revenue. In retail, shoppers who use search convert at two to three times the rate of those who simply browse. On streaming platforms, recommendations drive most of what users watch.

Reaching production-level QPS used to mean building a bespoke retrieval stack per app. Load testing across index sizes, query types, and filters. Manually sizing capacity. Wiring up load balancers in front of the index. Every new search use case restarts the work. The pain is universal across vector databases, search engines, and DIY stacks.

Announcing high-QPS scaling for Databricks AI Search

Today we're announcing high-QPS scaling for Databricks AI Search which is generally available. Standard endpoints can now scale to thousands of QPS with a single, human-readable parameter. You tell us your target. We provision the infrastructure to meet it.

Set target_qps on the endpoint at creation, or update it on any existing endpoint at any time via the SDK, REST API, or the endpoint UI. Databricks provisions the infrastructure to meet the target. No replica counts to manage, no nodes to size, no load balancers to wire up.

Unity Catalog governance and Delta Sync stay in place. The same endpoint that powered your prototype now scales to production traffic without leaving the platform.

When you need high-QPS scaling

Three real-time production patterns require it.

Search bars, like product search on e-commerce, content discovery on streaming and media platforms, voice search on connected devices. A typeahead box can fire a search call per keystroke, so QPS scales with active typing volume. Latency directly affects conversion. For an end-to-end architecture, see Building Real-Time Product Search on Databricks.

Recommendation / personalization systems, like "You may also like" panels on e-commerce, personalized feeds on media and streaming platforms. Every page view fires a recommendation query, so traffic spikes hit retrieval first. Recommendation latency is on the critical path of the request.

Real-time entity resolution, like identity matching, deduplication, lookup against large catalogs at request time. Query rate here is an operational SLA, not a burst you can buffer.

If you're seeing any of the following, you probably need this:

429 (Too Many Requests) errors showing up in production logs

P95 latency spiking as traffic ramps, even at moderate average utilization

Workarounds like duplicate endpoints or client-side load balancing

Deep dive into how it works

Declare. You set a QPS target on the endpoint. Databricks calculates and provisions the compute capacity to deliver that target. No replica counts, no node sizing, no capacity planning.

Works with existing endpoints. Update any Standard endpoint via the Python SDK, REST API, or the UI. New capacity takes effect the next time an index on the endpoint is created or synced.

Monitor scaling state. The scaling_info field on the endpoint tracks progress as it transitions from SCALING_CHANGE_IN_PROGRESS to SCALING_CHANGE_APPLIED.

Production observability. Operating a retrieval system in production needs visibility into requests per second, request latency, and endpoint health. The endpoint UI now surfaces all three for every endpoint.

Use service principal auth to maximize performance. Service principal traffic is routed through performance-optimized networks built for high-QPS production workloads. Personal access token (PAT) traffic is capped at a few tens of QPS, making it suitable for prototyping but not production. See the performance guide for the full walkthrough.

Sizing. Use the endpoint observability UI and native Genie integration to understand your traffic patterns and set target_qps with sufficient headroom for spikes.

Start building today

The difference between prototype and production is now a configuration parameter. High-QPS scaling is generally available today, no opt-in required. Two paths to start:

Spin up a new endpoint with target_qps set to your initial target

Update an existing endpoint with target_qps to scale up the index already serving your application

What’s coming next

We’re continuing to make search easier to operate at scale. Automatic scaling for traffic spikes—without manual capacity planning or sizing—and support for Storage-Optimized endpoints are planned for later this year.

To go deeper:

High QPS documentation

Databricks AI Search overview

Performance best practices

Building Real-Time Product Search on Databricks

Get the latest posts in your inbox

Subscribe to our blog and get the latest posts delivered to your inbox.

Sign up

View all blogs