What Is an Online Eval? Monitoring AI Quality in Production

An online eval runs asynchronously: traces flow from your production AI system into a scoring pipeline, each trace is evaluated against a rubric, and results accumulate into a live quality signal the team can monitor, segment, and alert on. The scoring happens after the response has already been returned to the user, so it adds no latency to the product experience.

The pipeline looks roughly like this in practice. A user sends a message to your AI feature. The system returns a response. That interaction is captured as a trace and routed to the eval pipeline. A scoring function evaluates it, passes or fails it against defined criteria, and logs the result. The pass rate across all traces updates in near-real time.

Teams typically set up dashboards showing pass rate over time, segmented by intent category, user cohort, or feature version. Alerts fire when the pass rate for a segment drops below a defined threshold. A sudden drop in pass rate on high-value user sessions is as actionable as a spike in error rate: it means something changed and needs investigation.

The key difference from running an offline eval is the input source. Offline evals use a curated dataset you control. Online evals use whatever real users actually send, which is always a larger and more varied distribution than any curated dataset captures.

Online evals surface failure modes that no pre-deployment test suite can anticipate, because they come from real user behavior at scale.

The clearest example is phrasing variation. An offline dataset of 50 carefully constructed test cases captures the failure modes a team knows to look for. It cannot capture the range of ways 10,000 users will phrase the same underlying intent. A phrasing variant that affects 0.3% of sessions looks negligible in a test environment; at production volume, 0.3% is 30 failed sessions per day. Online evals catch it because they run against every session.

Distribution shift is a second failure mode offline evals reliably miss. User behavior changes over time. Seasonal patterns, new user cohorts, and product changes all alter the distribution of intents hitting the system. An offline eval suite is a snapshot of what the product team expected users to do. Online evals track what users actually do as that evolves.

Novel failure modes come third. Some failure patterns only emerge at scale, when combinations of inputs and system states occur that low-volume testing never produces. Online evals surface these by running continuously rather than against a fixed set.

The two types cover different parts of the AI product lifecycle. Most production AI teams run both.

Offline evals gate releases. Online evals monitor live behavior. A team running only offline evals ships with confidence but flies blind after launch. A team running only online evals catches problems after users have already seen them, with no pre-deployment safety net.

The right question for any AI product team isn't which type to run. It's how to connect them so each one improves the other over time.

Online evals use the same two scoring approaches as offline evals, applied differently at production scale.

Code-based scoring

Code-based scoring runs on every trace. A regex match, a JSON schema check, a verified tool call, a required phrase present or absent: these checks are fast and cheap enough to apply to 100% of production traffic without meaningful infrastructure cost. Code-based scoring gives the team a constant baseline signal across the full volume of real sessions.

LLM judges

LLM judges don't run on every trace. Calling a separate language model to evaluate each production interaction adds cost that scales directly with traffic volume. Most teams handle this by sampling: score 5-20% of traces with an LLM judge, with higher sample rates for the segments that matter most (enterprise users, high-value intent types, sessions that followed an unusual path). This keeps cost manageable while still covering the cases where nuanced quality evaluation matters.

Human review

Human review plays a smaller but critical role. When online evals surface a novel failure mode that neither code-based scoring nor an existing LLM judge captures clearly, a human reviewer names it, writes the scoring criteria, and either creates a new code-based check or updates the judge rubric. Human review is the mechanism by which new failure modes get turned into repeatable eval cases.

A practical starting point: deploy code-based checks on everything from day one, sample 10% for LLM judge scoring, and reserve human review for the failures automated scoring can't characterize.

Online evals are how an offline eval dataset stays current as a product grows.

A production AI system encounters failure modes no pre-deployment test suite anticipated. Online evals surface those failures. A PM or engineer reviews the trace, names the failure mode, writes a scoring criterion for it, and adds it to the offline dataset. The next CI run tests against a tighter set of cases. The pre-deployment gate improves.

This loop is what eval-driven development looks like in a mature AI product team. The offline dataset is not a fixed artifact written at launch. It grows from production behavior over time, with each class of real failure becoming a permanent eval case that prevents the same failure from shipping again in a future version.

Teams that skip online evals lose access to this loop. Their offline datasets reflect what the team expected users to do at the time the system shipped, not what users actually do now. Over time, the eval suite drifts from reality, and quality problems surface only when users report them.

The feedback loop also applies to the scoring rubric. When human reviewers calibrate an LLM judge against online eval samples, the judge improves. Rubric updates propagate back to both offline and online scoring, raising the quality bar across the full eval system.

Running online evals generates a quality signal. The question PM teams then face is what that signal means for the business.

A pass rate tells you the system handled X% of production traffic correctly. It doesn't tell you whether the sessions that failed are also the sessions that drive retention, or whether failure concentrates in your most valuable user segments. Answering those questions requires the eval signal to live next to the product engagement data.

Amplitude's Agent Analytics treats AI interactions as events in the same product event stream where retention, conversion, and feature adoption are already tracked. Online eval scores can live in that same stream, which means a PM can ask: are the users whose sessions fail evals more likely to churn? Which intent categories are failing and how much of my core retention do they represent? Which model versions have better quality scores and better downstream conversion?

Session Replay adds a second layer: when an online eval fails, the team can pull the session recording and watch exactly what the user experienced. The eval score names the failure. The replay shows the context. Together they produce the information needed to prioritize which failure modes to fix first.

If you're building out your AI eval system from scratch, the AI Evals for Product Managers overview covers the full picture: traces, eval types, LLM judges, and how to connect quality to engagement metrics.

Online evals are how AI product teams find out what actually breaks in production. Connecting those failures to the product metrics that matter is how teams decide which ones to fix first.