Agent Observability Reaches 94% but Online Evals Only 45%

Agent observability is the practice of recording what an AI agent did, including its model calls, tool calls and intermediate steps, so that engineers can inspect a run afterwards. It answers the question of what happened, and it is now close to standard for teams with agents in production.

We think the more important question is the next one, which asks if what happened was right. That question needs a different instrument, and the survey data suggests fewer teams have it.

In this post we cover what the LangChain survey measured, how offline and online evaluation differ, what goes unseen when there are traces and no online scoring, why quality has overtaken cost as the barrier, five questions to ask before an agent touches production, when online evaluation is the wrong investment and what to add first.

What the LangChain Survey Actually Measured

We start with the numbers, because they are quoted loosely. The State of Agent Engineering survey collected 1,340 responses between 18 November and 2 December 2025. Of those respondents, 57.3% said they had agents running in production.

The headline observability figure is 89% across all respondents. Among those with agents in production it rises to 94%, and 71.5% of that group have full step-level tracing. Both numbers describe the ability to see what an agent did.

The evaluation figures are lower and easy to misread. The survey reports 52.4% running offline evaluations on test sets, 37.3% running online evaluations overall and 44.8% running them among teams with production agents. The 52% figure is offline only, and it should not be read as general evaluation coverage.

Two cautions apply. The sample leans towards early adopters, since 63% of respondents work in the technology industry and 49% are at companies with fewer than 100 employees. And every figure is self-reported, so a team that ticks a box for evaluation may run it rarely or on a small slice.

Offline and Online Evaluation Answer Different Questions

Evaluation is not a single checkbox. LangChain's evaluation concepts documentation separates the two kinds by what they run on and when.

Property	Offline evaluation	Online evaluation
Data	Curated test cases with reference outputs	Real production traces without reference outputs
When it runs	Before deployment	During production monitoring
What it catches	Regressions you already anticipated	Drift, rot and behaviour you did not anticipate
What it misses	Anything outside the test set	Problems with no usable signal on live data

The documentation puts it as "Use offline evaluations for pre-deployment testing" and "Use online evaluations for production monitoring", and adds that the two work as a loop, in which "Online evaluations surface issues that become offline test cases."

That is why we read the gap between 52.4% and 44.8% with care. A team can pass every offline test and still have no way to notice that live inputs have moved away from the test set.

What Goes Unseen When You Have Traces and No Online Evaluation

A trace is a faithful record of a run, and a faithful record of a wrong run is still wrong. OpenTelemetry describes a span as a unit of work that paints "a picture of what happened", and a trace as the path taken by a request, which is exactly what observability promises and nothing more.

Language models are built to produce plausible text. A trace can therefore look entirely logical, with each step following from the last, while the conclusion is disconnected from the systems it describes. Nothing in the record says the answer was incorrect.

We separate three needs that are often merged under one word, and each has a different instrument.

1. Observability Shows the Work

This is the record of reasoning, tool calls and data touched. It lets an engineer reconstruct a run, and it is necessary for any investigation of an agent's behaviour.

Its limit is that it is passive. It does not tell anyone to look, and a run that looks normal produces no signal to look at.

2. Verification Checks a Conclusion

Verification tests one specific conclusion against evidence, ideally by something independent of the system that produced it. Research on self-correction in language models found that "LLMs struggle to self-correct their responses without external feedback", which is why we want the checker to be a different model or a deterministic test.

This is per-case assurance. It says that this answer holds up, and it says nothing about the direction the agent is moving overall.

3. Evaluation Measures Quality Over Time

Evaluation scores outputs against a standard, repeatedly, so a decline shows as a trend. Online evaluation does this on live traffic, which is the only place a changed input distribution, a stale retrieval index or a model swap upstream becomes visible.

The failure it addresses is slow and quiet. Latency stays in bounds and error rates stay at zero, while the quality of answers erodes until a person notices a pattern. A crash would have paged someone, and a quality regression does not.

We cover the same class of problem for pipelines in our guide to silent failures in AI agent pipelines, where a successful status code hides a wrong result.

Quality Has Replaced Cost as the Barrier to Production

The survey reports quality as the top barrier to production, cited by 32% of respondents, with latency second at 20%. It describes cost as less frequently cited than in previous years, and it puts security second among respondents at companies with more than two thousand employees, at 24.9%.

The shift matters for how teams buy, and for what we would ask a vendor. If the constraint is the correctness of outputs, then a vendor comparison based on token prices and inference speed asks the wrong question. We would ask instead how a platform lets the team measure correctness on live traffic.

It also changes where the work lands. Engineering managers inherit the problem of showing that an agent is good, and a dashboard of traces does not demonstrate it.

The practical consequence is a change in what counts as done. An agent is not finished when it runs without errors, only when someone has defined what a good answer looks like, scored a sample of live outputs against that definition and decided what happens when the score drops. Those are product and engineering decisions, and they are cheaper to make before launch than after the first complaint.

For operations agents we add one more requirement. Because their conclusions can lead to changes in production, the check on a conclusion has to be independent of the reasoning that produced it, which is the argument we make in our piece on verifying AI root cause analysis.

Five Questions to Ask Before an Agent Touches Production

The same gap exists in tools that operate on infrastructure. Some show a detailed trace of what the agent looked at without saying how its conclusions are checked. We use these questions when we evaluate any such tool, including our own.

• What share of the agent's conclusions is checked by something other than its own reasoning?
• Is the checker a different model or a deterministic test, not the same model version?
• Does anything score outputs against real traffic, and not only a fixed test set?
• What is the workflow when the agent is confidently wrong?
• What human decision point comes before any system state changes?

The last question is the easiest to answer when the agent is limited to read-only access to production systems, because a wrong conclusion then costs an engineer's time and not an outage. For the wider architecture argument, see our summary of five constraints for AI operations.

When Online Evaluation Is the Wrong Investment

Online evaluation is not free. It needs a way to score outputs without reference answers, which often means another model acting as a judge, and a judge from the same family can share the blind spots of the agent it grades.

It is also premature for a prototype with a handful of users and no stable behaviour to measure. In that case a good offline test set and manual review of traces give more information per hour than an automated score nobody trusts yet.

And it does not replace verification of individual cases. A rising average says nothing about the single confident wrong answer that is about to change a configuration, which is why we ask for both kinds of check where the stakes justify them.

Add Online Evaluation Before Adding More Agent Observability

If your traces are already detailed, the next dollar is better spent on scoring a sample of live traffic, with an alert when the score drifts. More spans will not show that the answers are good.

We would set that up first and add richer tracing second. A per-incident check is verification, not evaluation over time, and the two should be built together. Operate brings evidence from logs, databases and code together through read-only adapters, and a separate agent on a different model checks each proposed root cause against that evidence.