Online vs. Offline AI Evals: When to Use Each

AI agents excel at breaking things. They're non-deterministic: the same input can produce different outputs, some with bad consequences.

Evaluations are how we monitor and measure the performance of AI systems, but how and when do you test and score your agent? In this guide we'll show the two patterns for checking that your agent works the way you think it does in production.

The short version: offline and online evals differ in when they run and what they measure.

An offline eval scores your agent against a fixed dataset before you ship: a unit test for AI, run in CI on inputs you choose in advance.

An online eval scores each interaction as it happens in production, against real traffic and real outcomes. Use offline as a pre-deploy gate on a known set of cases; use online to see how the agent actually behaves once real users hit it. Most teams run both, offline to catch regressions before release, online to measure the truth that offline can only approximate.

The Common Components of a Good Eval

What we call "AI evaluation" is made up of a few parts. Together they give a complete view of an agent's performance.

Dataset

This can be a live stream of data as it happens, or a static file. Either way, the input is the first variable: the common source of truth every eval runs against.

Split Testing / Experiments

Given the same input, did GPT or Claude produce the better result? What about prompt A vs. B? Experiments let you test multiple theories at once, or skip them and just test the deployed agent.

Scoring

Scoring is where it all comes together. Given the combination of variables (the input, the model, the prompt), we can create a rubric that measures them the same way every time, based on what and how we care to measure:

LLM-as-Judge: A scoring function can call a second LLM with the agent output and an evaluation prompt. Use it for evaluations that are highly nuanced and hard to fit into something more deterministic.

Algorithmic: A code-only scoring function; measure the agent on token cost, number of words, speed of output generation, etc.

Signal-based: This could be a scoring function that waits for human feedback, or an event from somewhere else in the system; an order successfully created indicates a successful tool call, for example.

Defining your scoring rubrics is the most critical piece of a good eval system. Aim to capture a few dimensions to get the fullest view of your agent.

Those pieces combine into two patterns. Here's how they compare before we dig into each:

DimensionOffline evalOnline eval

Data sourceA fixed "golden" dataset you assembleLive production traffic

When it runsBefore deploy, in CI, or exported data separate from the runtimeContinuously, as each interaction happens

SignalNarrow but controlled: only the cases you pickedHigh-volume and real: whatever users actually do

Catches regressions before users see them?YesNo, it measures what's already deployed

Real-outcome signals (did the user accept it? did the PR merge?)NoYes, including deferred scoring when the outcome lands later

MaintenanceYou curate and refresh the dataset as the product driftsLow-performing variants fall out of execution as experiments are run

Offline Evals: Point-in-Time Peace of Mind

The key definition of an offline eval is a static dataset. Given static input, scorer, and any split-test experiment, we get a defined evaluation.

It's popular because you can test a change to the agent before deployment. Think unit tests, but for AI. You can run a static evaluation locally or as part of a CI pipeline on a known input.

The downside: the test's scope is inherently limited. A handful of examples in a static dataset is a weak signal for what the agent will actually hit in production, and it's easy to miss cases you didn't think of when you built the test.

There's a second cost. Someone has to build that golden dataset, keep it representative, and re-curate it every time the product changes.

Skip the upkeep and the set goes stale: a green suite starts certifying behavior the agent hasn't shown in months. Offline evals also run single-shot, on the inputs you chose in advance, so they tell you about the cases you thought to include and nothing about the ones you didn't.

Online Evals: Scoring Powered by Live Data

Online evals take a different approach: score and evaluate each agent interaction as it happens, on live data.

This has two big advantages. First, you're scoring the agent on real data, not synthesized tests. Second, the sheer volume of scores is higher, which means a more accurate signal.

This is where running evals inside Inngest pays off, and the reason is structural. Inngest already runs and orchestrates your agent durably: every step, retry, and outcome is persisted as it executes. So the data an eval needs is already there. Scoring it means reading runs the platform is already keeping, not standing up a second system to watch the first. The load-bearing part, durable execution, is already GA and proven; evals are that same machinery pointed at a new problem. Two things follow that offline setups can't do.

You can run split-test experiments against live data. In the GPT-vs-Claude example, once one model outperforms the other, it's trivial to route traffic to the winner.