Real-Time Retail Intelligence: Building E-Commerce Recommendations with Lakebase and AI Search on Databricks | Databricks Blog
Skip to main content
A complete reference architecture for building a multi-stage recommendation and ranking engine on Databricks — from clickstream ingestion to real-time personalized serving — replacing fragmented ML infrastructure with a single unified platform.
The system delivers low latency personalized product recommendations by combining pre-computed batch recommendations with a real-time scoring path, using Databricks AI Search for candidate retrieval, Lakebase for online feature serving, and Model Serving for low-latency inference.
By unifying data engineering, feature management, model training, and real-time serving on one governed platform, e-commerce teams eliminate glue code, accelerate iteration cycles, and gain end-to-end lineage from raw clickstream to production predictions.
The opportunity: Personalization as a revenue engine
Every second a shopper spends on a fashion e-commerce app generates a stream of intent signals — searches, product views, wishlist additions, cart interactions. The platforms that convert those signals into relevant product recommendations in real time are the ones that win. Industry benchmarks show that effective personalization can lift conversion rates by 10–30% and increase average order value significantly.
Yet building a production-grade recommendation system remains one of the hardest ML engineering challenges. It demands real-time data ingestion, complex feature engineering, multiple ML models working in concert, and serving infrastructure that responds in milliseconds — all while keeping inventory, location, and business rules in sync.
This blog presents a complete reference architecture for building such a system on Databricks, based on a real-world implementation for a leading fashion e-commerce platform in Asia serving over 1 million monthly active users across a catalog of 100,000+ SKUs.
System overview: The architecture at a glance
The architecture follows a unified platform approach where every component — from ingestion to serving — runs on Databricks, governed by Unity Catalog.
Overall System Architecture:
Data ingestion: From clickstream to lakehouse in seconds
The platform ingests approximately 1,000 events per second — product views, searches, add-to-cart actions, purchases, and session metadata. Lakeflow Connect’s Zerobus Ingest provides the ingestion backbone, landing events directly into Unity Catalog Delta tables without requiring a self-managed message broker.
An important architectural distinction: clickstream data flows through Zerobus into the lakehouse for offline feature computation and model training, but during real-time inference (Path B), in-session user signals — what the shopper is browsing right now — are sent directly as part of the API request payload to the Model Serving endpoint. This bypasses lakehouse storage entirely during the inference path, ensuring that real-time context is available without incurring ingestion latency.
Zerobus accepts data from any standard Kafka producer client (Java, Python, Go) via a simple configuration change — point the bootstrap server at the Zerobus endpoint, and records land in the target Delta table. For teams already running Kafka infrastructure, Structured Streaming with Declarative Pipelines offers an alternative path with the same downstream architecture.
The data flows into a medallion architecture:
Bronze layer — Raw, append-only event streams plus reference data:
User signals:
Behavioral: browsing sequences, search queries, dwell time patterns, click-through history
Transactional: purchase frequency, average order value, return rate, category spend distribution
Preference: wishlisted items, saved searches, brand affinities, size preferences
Demographic: age cohort, location cluster, device type, app engagement tier
Item signals:
Catalog: category hierarchy, brand, price tier, material, color, seasonality
Performance: conversion rate, view-to-cart ratio, return rate, average rating
Visual: image embeddings (capturing style similarity beyond text attributes)
Freshness: days since listing, trending velocity, stock trajectory
Contextual signals:
Temporal: time of day, day of week, proximity to payday, festival/sale periods
Geographic: user city, nearest fulfillment center, regional style preferences
Environmental: weather (drives category preferences), local events
Session: current browsing intent (inferred from in-session behavior)
Silver layer — Cleaned, sessionized, and enriched:
Sessionized user behavior (browsing sequences, dwell times, interaction patterns)
Cleaned product features (normalized attributes, category hierarchies)
Aggregated engagement signals (7-day view counts, trending scores, popularity indices)
User-product interaction matrices (implicit feedback signals)
Gold layer — Model-ready feature tables and training datasets:
User feature vectors (behavioral embeddings, preference profiles, demographic clusters)
Item feature vectors (product embeddings, attribute encodings)
Training datasets (labeled interaction pairs for model training)
Pre-computed recommendation lists (batch-generated, stored for fast lookup)
Features are refreshed on different cadences: behavioral aggregates update daily through scheduled Databricks Workflows, while the full product catalog syncs weekly. Embeddings for both users and items are recomputed daily to capture evolving preferences and new inventory. The Databricks Feature Store manages both offline features (for training) and online features (for serving), ensuring training-serving consistency — the same feature definitions used during model training are automatically available at inference time via Lakebase online tables.
Unity Catalog governs every layer — providing lineage from raw clickstream event to the final prediction served to the app, with fine-grained access control ensuring PII stays protected while aggregated features flow freely to model training.
Real-time serving: Two paths to low latency responses
The serving architecture provides two complementary paths, each optimized for different interaction patterns. Path A handles the high-volume, predictable surfaces where pre-computation is both feasible and optimal. Path B handles the dynamic, session-aware surfaces where the user's immediate intent must shape the response in real time.
Path A — Pre-computed batch recommendations (< 2 digit ms)
Path A serves the majority of recommendation surfaces — homepage carousels, category page rankings, email campaigns, and push notifications. These surfaces share a common trait: the user's identity and surface type are known in advance, so results can be computed ahead of time.
A nightly batch job, orchestrated by Databricks Workflows, runs the full 3-stage funnel offline for every active user. It retrieves the latest user embeddings, executes batch ANN queries against the AI Search item index to generate candidates, scores them with the LightGBM model using features from the Gold layer, and applies business rules (inventory scoring, delivery proximity, diversity, promotional boosting). The output — a top-N ranked product list per user (typically 50–100 items per surface) — is written to Lakebase online tables, keyed by user ID and surface type.
At serving time, the app performs a simple key-value lookup: user_id + surface → ranked product list. No model inference, no vector search, no feature assembly — just a direct read from Lakebase.
Because the batch job runs nightly, Path A reflects the previous day's signals and inventory state. For most surfaces this freshness is more than sufficient — long-term preferences and brand affinities evolve over days, not minutes — and new products that received their initial embeddings will surface in recommendations within 24 hours.
Path B — Real-time session-aware scoring (< 2-digit ms)
Path B activates when the recommendation context only exists at request time — "Similar items" on a product detail page, "Complete the look" suggestions, or dynamically re-ranked search results that adapt as the user browses.
The e-commerce app sends current session signals — items viewed in the last few minutes, active search queries, cart contents, and dwell-time patterns — directly as request payload to the Model Serving endpoint via REST API. The endpoint executes the full 3-stage funnel synchronously within a single request-response cycle:
Stage 1 — Real-time candidate retrieval The endpoint transforms session signals into a query embedding — either the item embedding of the product being viewed, or a blended user embedding that fuses the long-term preference profile (from Lakebase) with in-session behavior, weighting recent signals more heavily. This embedding is sent to Databricks AI Search, which performs hybrid retrieval: ANN similarity search combined with hard metadata filters (category eligibility, regional inventory, minimum stock thresholds). Items that are out of stock or ineligible never enter the candidate set. AI Search returns 200–500 candidates in a single network round-trip. For session-aware surfaces, the endpoint computes a blended query embedding — combining the user's pre-computed long-term preference embedding (retrieved from Lakebase) with a session embedding derived from recently viewed items, weighted toward recent signals to capture immediate intent. This blending is a lightweight computation performed within the endpoint, not a separate model call
Stage 2 — Feature assembly and scoring Candidate item IDs trigger point lookups against Lakebase online tables, retrieving pre-computed user features (behavioral aggregates, demographic segment, price sensitivity) and item features (conversion rate, trending score, days since listing). Real-time context features (time of day, device, location, active promotions) are derived directly from the request payload. The endpoint constructs feature vectors for each user-candidate pair — including cross features such as user-category affinity and brand preference overlap — and passes them to the LightGBM scoring model, which predicts conversion probability in a single batch inference call without GPU resources.
Stage 3 — Business rules and re-ranking The scored list passes through a deterministic re-ranking layer: inventory-aware scoring deprioritizes declining stock, delivery proximity scoring favors closer warehouses for faster fulfillment, diversity injection prevents brand or category dominance in the final list, and promotional boosting surfaces items aligned with active campaigns. The re-ranked list is truncated to the requested size (typically 10–20 items) and returned as the API response.
Business rules are configuration-driven — promotional weights, diversity thresholds, and inventory cutoffs are read from a managed configuration table at serving time, allowing commercial teams to adjust rules without redeploying the model.
The endpoint is implemented as a custom MLflow PyFunc model that orchestrates the multi-stage pipeline internally — querying AI Search, performing Lakebase lookups, running LightGBM inference, and applying business rules within a single predict() call.
A fallback strategy ensures resilience: if the real-time path exceeds its latency budget, the system gracefully degrades to serving cached popular items or the user's pre-computed recommendations from Path A.
Handling the cold start problem
Every recommendation system must address two cold-start scenarios:
New users (no browsing history): When a user first arrives, the system constructs a default user embedding from available demographic signals — location, device type, sign-up context, and any stated preferences. This embedding is used for ANN search against the item index, effectively placing the new user within a behavioral cluster of similar demographics. As the user interacts, their embedding rapidly converges toward their true preferences.
New products (no interaction data): When a new SKU enters the catalog, the system generates an item embedding from its attributes — title, category, brand, price point, and visual features extracted from product images. This embedding is used to find similar existing items in the vector space, and the new product inherits initial recommendation scores from its nearest neighbors. New products surface in recommendations at the next daily batch cycle.
Model lifecycle and continuous improvement
Models are retrained weekly using Databricks Workflows, with experiment tracking and versioning managed through MLflow. The platform supports champion/challenger deployment — new model versions are deployed alongside the production model, with traffic gradually shifted based on online performance metrics.
Key ML metrics monitored include:
AUC-ROC — measures the scoring model's ability to distinguish between items a user will and won't engage with
NDCG@K (Normalized Discounted Cumulative Gain) — evaluates whether the most relevant items appear at the top of the ranked list
Recall@K — measures what fraction of relevant items the candidate generation stage successfully retrieves
Log Loss — tracks prediction calibration, ensuring conversion probability scores remain well-calibrated over time
These model metrics are complemented by business KPIs — click-through rate, conversion rate, and revenue per session — which serve as the ultimate validation that model improvements translate to real-world impact.
Automated drift detection flags when feature distributions or prediction score distributions deviate from baselines, triggering investigation or accelerated retraining. Serving logs are correlated back to training pipelines via request-level identifiers, ensuring the feedback loop produces clean, leakage-free training data for the next model iteration. Position-aware training techniques ensure the model learns true user preference rather than display-position artifacts.
Expected outcomes
This architecture enables:
low 2-digit ms personalized responses at scale, matching the performance expectations of modern mobile-first shoppers
Daily-fresh recommendations that reflect the latest inventory, trending products, and evolving user preferences
Unified governance from raw clickstream to production predictions, with full lineage and access control via Unity Catalog
Rapid experimentation — new model variants can be trained, evaluated, and deployed within days rather than months
Operational simplicity — a single platform replaces the patchwork of separate systems for ingestion, feature engineering, training, and serving
Graceful scaling — from thousands to millions of users without re-architecting, leveraging serverless compute and managed infrastructure
Conclusion
Building a production-grade recommendation and ranking engine no longer requires stitching together a dozen specialized systems. By unifying real-time ingestion (Zerobus), feature management (Feature Store + Lakebase), model training (MLflow + Workflows), vector retrieval (AI Search), and low-latency serving (Model Serving) on a single governed platform, e-commerce teams can focus on what matters: understanding their customers and delivering the right product at the right moment.
The result is not just a recommendation engine — it is a complete, production-ready personalization platform that can serve as the intelligent backbone of any e-commerce experience.
Ready to build your own? Explore the Databricks Recommendation Engine Solution Accelerators, dive into AI Search documentation, or contact your Databricks account team for an architecture workshop.
Get the latest posts in your inbox
Subscribe to our blog and get the latest posts delivered to your inbox.
Sign up
View all blogs