Build durable agents with Temporal and Lakebase | Databricks Blog
Skip to main content
Preserve agent progress with Temporal: recover recorded work after worker failures, retry failed operations, and wait durably for human review.
Serve live application state with Lakebase Postgres: make evidence, recommendations, review decisions, and operational metrics queryable throughout each run.
Connect execution to governed data: read Unity Catalog policy through synced tables and, with Lakebase Change Data Feed enabled, publish operational changes back to Delta history tables.
A personal-loan underwriting agent gathers evidence, applies policy, and may wait days for a reviewer. During that time, workers can restart and tool calls can fail. The application must preserve completed work, resume execution, and keep the evidence available to the reviewer.
This reference implementation uses Temporal for durable execution and Lakebase Postgres for queryable operational state. A synced table makes underwriting policy from Unity Catalog available in Lakebase. Temporal Activities write evidence, decisions, and metrics to Lakebase; once enabled, Lakebase Change Data Feed can publish those changes to Unity Catalog-managed Delta history tables. This combination is especially useful when Databricks already manages the agent’s inputs and downstream analysis.
The Challenge with Long-Running Cloud Agents
A cloud agent may outlive the request, worker, container, or deployment that started it. A user can begin a session, return tomorrow, and continue on another worker. Deployments and process failures are routine, so the agent’s progress must survive independently of the process executing it. Recovery requires both the results of completed operations and the control-flow state needed to determine what happens next.
For this underwriting agent, this creates six requirements:
Recovery: A replacement worker must resume from the last completed step.
Retries: Tool calls and database operations must tolerate repeated execution without duplicating side effects.
Long waits: The agent must wait for people or external systems without holding a worker open.
Operational visibility: Applications and operators need the current status, evidence, retry state, and failure details.
Runtime governance: Policy updates must become available without a code deployment, and the application must define when an open case adopts them.
Audit: The system must retain the evidence, policy, recommendation, and human decision associated with each run.
A conversation transcript covers only part of this state. Recovery also requires the control-flow history: which operations were scheduled, which results were recorded, what the agent is waiting for, and which commands it has accepted.
Temporal simplifies the management of distributed systems. When building with Temporal, a Workflow is the durable control flow for one agent run. An Activity is a call to a model, tool, or database whose result is recorded in the Workflow's Event History; Activities can be retried. A Signal is an asynchronous command sent to a running Workflow, such as an underwriter’s decision. The Temporal Lakebase AgentWorkflow reference implementation is a runnable personal-loan underwriting agent. It calls several tools, reads governed policy, produces a recommendation, and waits for an underwriter.
Lakebase Postgres also helps developers manage these problems, but Temporal and Lakebase store different state for different consumers. Temporal’s Event History drives replay. Lakebase stores the application-facing view: current run status, messages, evidence, review state, and metrics. Unity Catalog remains the policy source; a synced table makes that policy queryable in Postgres, and Change Data Feed provides the return path for operational history. The systems do not share a transaction. Lakebase writes run as Temporal Activities under at-least-once execution. Deterministic identifiers, constraints, guarded updates, and Postgres upserts ensure that repeated Activity attempts target the same logical record.
This architecture adds two managed systems and a projection contract between them. Together, they improve the agent’s resilience and scalability while keeping operational overhead low. Temporal plus Lakebase is most useful when an agentic session must survive worker replacement, accept input after long waits, expose relational state to an application, and apply governed data while it remains open.
The Underwriting Use Case
I chose loan underwriting because the same run must gather evidence, apply policy, produce a recommendation, and wait for a person. A worker can fail between any of those steps. Policy can change without an application deployment, and the UI needs the current evidence before the Workflow closes.
Mocked applicants replace real credit bureaus and income providers, and the tool sequence is deterministic for simplicity. Each request contains a user ID, applicant ID, amount, purpose, model choice, and turn limit. FastAPI assigns run_id, starts LoanUnderwritingWorkflow, and uses that same ID across the API, Temporal execution, and Lakebase rows.
On the first turn, credit_check returns score, trade lines, delinquencies, and current debt. income_verification returns income and employment evidence. debt_to_income_calc calculates debt-to-income ratio. policy_lookup loads the policy for the loan purpose and evaluates the evidence against approval, referral, and hard-decline thresholds.
The sample’s borderline applicant has a 665 credit score, $76,000 in verified annual income, $2,400 in monthly debt, and one non-material delinquency flag. The policy result records every rule, threshold, actual value, pass/fail result, source, recommendation, and rationale. The model can recommend but cannot decide. An underwriter approves, denies, or requests more information. A request for more information becomes another user message and another agent turn. The case exercises a worker crash after completed tool calls, a committed Lakebase write whose Activity completion is lost, a review left open for days, a stale browser decision, and a policy change during execution.
Architecture
Figure 1. The execution, operational-state, and governance paths in the reference implementation.
To implement the underwriting agent, React and FastAPI handle HTTP and UI work: starting runs, rendering evidence, listing cases, and submitting review decisions. Temporal Cloud stores Event History and dispatches Tasks. Workers replay Workflow code and execute model, tool, and Lakebase Activities; network and database I/O remain outside deterministic Workflow code.
A run begins when FastAPI starts a Workflow. The worker schedules Activities, Temporal records their results, and the agent eventually reaches AWAITING_REVIEW. The underwriter’s response returns through a Signal. Approval or denial closes the run; a request for more information resumes the agent loop.
Lakebase holds two operational schemas. agent_ops contains run status, messages, tool calls, review records, events, and metrics that FastAPI can query with SQL. agent_policy contains the read-only synced policy used by policy_lookup. Each Activity writes records keyed by the same deterministic identifiers used by the Workflow, so the projection can catch up after a retry without making Lakebase part of Temporal’s replay mechanism.
Unity Catalog is the source for underwriting thresholds. A continuous synced table makes them available to the running agent. The applied thresholds, evidence, and subsequent human decision are written to agent_ops. Change Data Feed can publish those changes to Unity Catalog–managed history tables for audit and analysis.
Recover Completed Work After a Worker Failure
Temporal keeps the ordered Event History required to rebuild Workflow state on another Worker. That history includes Activity scheduling and results, timers, and Signals. Replay runs the Workflow code against those recorded Events and reconstructs variables such as the current turn, accepted review decisions, token usage, and collected evidence.
A recorded Activity result is returned during replay instead of running the Activity again. A completed credit check remains completed, and a recorded model response remains the response for that execution. If an Activity was in flight when the Worker failed and Temporal never recorded its completion, Temporal can schedule another attempt. For an agent, this preserves model responses already recorded in Event History. A model call whose completion was not recorded may still run again, even if the provider finished processing it.
Retry Policies are assigned at the granularity of individual operations and can be reused in code. In the example, model-calling Activities allow up to four attempts within a three-minute schedule-to-close timeout. The tool calling Activities allow up to three attempts and have a 60-second start-to-close timeout. The Lakebase Activities allow up to five attempts with a 15-second start-to-close timeout.
Make External Effects Safe to Repeat
One risk is that a Lakebase tool-result write can commit before the Worker reports Activity completion. If the connection drops in that gap, Temporal has no recorded result and schedules another attempt. Both attempts represent the same logical write.
Each Lakebase record has a stable identity. run_id anchors the operational schema. message_id identifies a message, tool_call_id a tool invocation, event_id a milestone, review_id a review round, and decision_id a reviewer command. Postgres primary keys and unique constraints enforce those identities.
The tool-start write shows both the stable identity and the terminal-state guard:
A retry targets the same tool_call_id. The final predicate only allows an existing nonterminal row to be written back to started. If the row is already succeeded or failed, PostgreSQL affects zero rows. It does not raise an error.
The caller must inspect a zero-row result. LakebaseWriteResult returns the affected row count, but the current Activity wrapper does not turn zero into a failure. Production code should classify zero as an expected no-op only after confirming the stored terminal state; otherwise it should raise or record a conflict. The same rule applies to guarded run and review transitions.
Similar upserts cover messages, tool results, and Events. Deterministic IDs make retries converge on the same logical row, while each guarded write defines which state transitions are legal. The API can briefly show older state while a write retries. After the Activity succeeds, the accepted row is queryable.
Every side-effecting tool needs an equivalent contract. A payment API may accept an idempotency key, an email service a caller-supplied message ID, and a database a unique constraint. If the external system provides no deduplication mechanism, the Activity needs its own record or a reconciliation process. Temporal determines when to retry. The Activity determines how the external system handles that retry.
Expose Current State and Operational Metrics
Event History supplies execution semantics and debugging detail. The application needs indexed relational queries over the current run: list cases by user and status, load one transcript with its evidence, find reviews waiting for a person, and aggregate measurements across executions.
Lakebase stores that application view in a normalized Postgres schema. agent_runs holds current status, Workflow ID, request, token totals, timestamps, and recommendation metadata. agent_messages preserves the transcript. agent_tool_calls records arguments, status, structured result, error, and timing. agent_review_decisions connects the recommendation to a stable review ID, reviewer command, rationale, and decision time.
The schema also records named Events and metrics at Workflow, turn, and Activity-attempt levels. FastAPI exposes run-detail, workflow-metrics, and retry-metrics endpoints backed by these tables. The UI can show one run gathering evidence, another waiting for review, and a third retrying a failed tool. Operators can query the same rows with SQL.
Evidence is available before the Workflow completes. After policy_lookup finishes, its structured result is stored with the tool call. When the run reaches AWAITING_REVIEW, the underwriter can see the credit score, DTI, thresholds, rule results, rationale, and policy source that produced the recommendation.
Keep Human Review Durable and Reject Stale Commands
When the model returns a recommendation, the Workflow derives review_id from run_id and the current turn. It writes the pending review to Lakebase, records an agent.review_pending event, sets the projection to AWAITING_REVIEW, and calls workflow.wait_condition. Temporal retains the open Workflow without keeping a Worker process occupied.
The API sends the underwriter’s action as a Signal. Before sending it, the API checks that Lakebase shows the run awaiting review and that the submitted review_id matches the current round. If either check fails, the API returns a conflict. The Workflow independently validates the command against its own state and ignores stale or duplicate decisions, protecting the execution even when the Lakebase projection lags.
After accepting the Signal, the Workflow persists the decision through an idempotent Lakebase Activity. Approval or denial completes the run. A request for more information changes the projection back to RUNNING, appends the reviewer’s rationale as a user message, and starts the next turn. Because the turn changed, the next recommendation receives a new review_id.
The API’s 202 response confirms that Temporal received the Signal. Business acceptance happens asynchronously in the Workflow, so a command can pass the API precheck and still be ignored if the review state has changed. The client refreshes the Lakebase projection to observe the resulting state.
Serve Governed Policy without Redeploying Workers
Underwriting thresholds change independently of Worker code. The source table in Unity Catalog contains purpose-specific values such as minimum credit score, automatic-approval DTI, hard decline thresholds, and policy name.
The setup script creates a continuous Lakebase synced table named agent_policy.underwriting_policy_limits. policy_lookup queries this read-only Postgres copy by normalized loan purpose. Policy owners update the Unity Catalog source; the sync pipeline propagates the change, and a later run reads it without a Worker or API deployment.
The policy result contains the applied thresholds, every rule’s actual value and pass/fail result, and the source. The demo can fall back to fixture policy when Lakebase is disabled or the row is unavailable, and records that path as fixture_fallback. A regulated Workflow may instead fail closed. The application has to make that fallback decision explicitly.
Return Operational Changes to Unity Catalog
The repository prepares each agent_ops table for Lakebase Change Data Feed by setting REPLICA IDENTITY FULL. An administrator still has to enable the feature for the schema. Lakebase then captures inserts, updates, and deletes from the Postgres write-ahead log and writes them in batches to Unity Catalog-managed Delta history tables named with the lb__history pattern.
Change Data Feed is currently in Public Preview and flushes changes roughly every 15 seconds. That interval fits audit and analysis, while the UI queries Lakebase directly for current operational state.
The history tables can reconstruct a run’s policy source, tool evidence, Activity attempts, review wait, recommendation, and human decision. The repository configures the source schema for this path but does not include an observed end-to-end Change Data Feed run. Enabling the feed and verifying the destination tables remain deployment steps.
Operate the System
The deployment separates React/FastAPI from the Temporal Worker. API replicas scale with request load; Workers scale with Workflow and Activity Task backlog and configured concurrency. Lakebase autoscaling adjusts database compute within project bounds.
Temporal Cloud pricing is based on Actions plus active and retained Event History storage, so retry frequency and long-open histories also affect cost. The team still sets Kubernetes replicas, Task Queues, connection-pool limits, and database bounds for its workload. Alternatively, you can stand up your own open-source Temporal Service using the latest open-source release.
The Lakebase client uses OAuth machine-to-machine authentication. Databricks OAuth tokens and generated database credentials expire, so the client refreshes its SQLAlchemy connection pool before the one-hour database credential expires. Connections use TLS. Without rotation, a long-running Worker would encounter database failures on a predictable schedule.
Operators use Temporal to inspect Workflow and Activity history, Lakebase to query application state and metrics, and Kubernetes to check process and deployment health. An operator can then distinguish a deliberate review wait from an Activity retry, a database access failure, or a failed tool.
Evidence and Limits
The test suite contains 21 passing tests for Workflow sequencing, review behavior, OAuth connection construction, idempotent persistence, metrics contracts, API Workflow start, and Worker settings. The crash-recovery script adds a process-failure exercise with the deterministic provider.
Applicant and provider data are fixtures. The repository does not validate lending models, regulatory compliance, production security controls, regional availability, or performance at scale. The local crash exercise ran with Lakebase disabled, so it isolates Temporal recovery. Change Data Feed still requires enablement and verification in the target Databricks environment.
Next Steps
Want to learn more? Try running the demo for yourself and see durable execution in action. Run the reference implementation, stop a worker mid-run, and watch the agent recover. Connect Lakebase to explore the evidence, policy, and human-review state behind each decision.
Get the latest posts in your inbox
Subscribe to our blog and get the latest posts delivered to your inbox.
Sign up
View all blogs