MESH FLOW
Menu
← All insights
· Filippo Pietrantonio

AI Implementation

How Do You Monitor AI Agents in Production? (The Checks That Catch Silent Failures)

Uptime dashboards tell you an agent ran. They don't tell you it was right. Here's the monitoring stack that catches confidently wrong outputs, drift, and compounding errors before your customers do.

The short answer

Monitor outcomes, not uptime. A broken AI agent usually returns a clean response with a wrong answer inside it, so infrastructure metrics catch almost nothing. Track six things: task completion rate, human override rate, output quality via evals on sampled live traffic, step-level traces, input and output drift, and cost per task. Override rate is the earliest warning you will get.

Seventy-four percent of leaders call inaccuracy a highly relevant AI risk. About 8% report having actually had an AI-related incident (McKinsey, March 2026).

Those two numbers can't both be comfortable. Either inaccuracy is far rarer than almost everyone fears, or most companies have no reliable way to tell when it happens. Having built and run agents that touch real tickets and real invoices, we'd bet heavily on the second.

If you're the person accountable for an agent now sitting inside a live workflow, downtime isn't your real risk. Confident wrongness is — and it doesn't page anyone. This is what to watch instead.

Why do AI agents fail silently?

Because the failure looks identical to success at the infrastructure layer. A traditional service fails loudly: it times out, throws a 500, or stops. An agent fails by returning a well-formed, fluent, plausible answer that happens to be wrong — with a 200 status code and a 1.2-second response time.

Your monitoring was built for the wrong failure mode. Latency, error rate, and uptime all measure whether the agent ran. None of them measure whether it was right. If that's your whole dashboard, your detection system is your customers.

Reliability is task-specific, not model-specific. Stanford's 2026 AI Index shows how wide the spread is: on the HHEM document-summarization leaderboard, the top 15 models hallucinate between 1.8% and 5.4% of the time. On the harder AA-Omniscience benchmark, across 26 models the rate ranges from 22% to 94%. "We're using a good model" tells you nothing about your workflow.

Agents degrade exactly where business users push them. On the KaBLE benchmark, GPT-4o scored 98.2% on true beliefs but fell to 64.4% when a user asserted something false in the first person. That is a precise description of a support ticket, a vendor dispute, or a salesperson's optimistic CRM note.

And the oversight gap is widening, not closing. The AI Incident Database logged 362 incidents in 2025, up from 233 in 2024 — roughly a 55% jump — while the share of organizations rating their own incident response "excellent" fell from 28% to 18%.

What should you actually monitor?

Six signals, in rough order of how fast they'll save you. The first two are business metrics, not engineering ones — which is why they usually have no owner.

1. Task completion rate. Did the agent finish the job end to end, without a human finishing it for them? Not "did the API return," but "did the invoice get coded," "did the ticket get resolved." This is the only number your CFO cares about and the one most teams never instrument.

2. Human override rate. What percentage of agent outputs does a human change, reject, or redo? This is the single best leading indicator in agent operations: it moves before your quality metrics do, because your team notices problems before your dashboard does. A climbing override rate is a quality regression with a two-week head start.

3. Output quality via evals on sampled live traffic. Build a graded test set from real production cases, then run it continuously — not once before launch. Sample a slice of live traffic daily, score it with a model-as-judge, and route a smaller human-reviewed sample underneath. In Dynatrace's Pulse of Agentic AI 2026 (a vendor survey, so read it with that in mind), the top validation methods were data quality checks at 50%, human review of outputs at 47%, and drift or anomaly monitoring at 41%.

4. Step-level traces. When a multi-step agent fails, "the agent was wrong" is not a diagnosis. You need to see which tool call returned garbage, which retrieval pulled the wrong document, which step the reasoning went sideways. Without traces, every incident is an archaeology project.

5. Input and output drift. Your agent didn't change; your world did. New product SKUs, a renamed field, a customer segment that writes differently, a vendor who changed their invoice template. Watch the distribution of what's coming in and what's going out, and alert on the shift.

6. Cost per completed task. Not cost per token — cost per outcome. An agent that retries five times to succeed once is a budget problem disguised as a success metric. And most companies can't see this at all: KPMG's Global AI Pulse found 42% of organizations have only partial visibility into their AI spend (via ITPro). Same discipline we've argued for in controlling AI costs before token bills compound.

How do you catch errors that compound across steps?

You instrument every step, and you set your per-step bar far higher than feels necessary. Multi-step agents don't degrade linearly — they degrade multiplicatively, which is why a demo that looked flawless falls apart on real work.

The arithmetic is unforgiving and it's worth doing on a whiteboard. An agent that is 98% reliable on each individual step, run across a 20-step workflow, completes the whole task correctly about 67% of the time (0.98^20). At 95% per step, it's 36%. Nobody shipped a "36% accurate" agent — they shipped a 95% accurate one and never measured the end-to-end outcome.

This is why task completion rate and step traces have to be instrumented together. Step metrics tell you where it broke; the outcome metric tells you how often it actually mattered.

And it's why checkpoints beat longer chains. If a workflow has a step where a wrong answer is expensive and hard to reverse — issuing a credit, sending an external email, closing a ticket — put a human or a hard validation rule there and stop treating full autonomy as the goal. Dynatrace found 69% of agent-powered decisions are still verified by humans and 87% of organizations are building agents that require human supervision. That isn't immaturity. For most mid-market workflows, it's correct design.

Who owns agent monitoring?

Someone named, with the authority to roll back. Monitoring without a clear owner is theater, and the data backs that up hard: McKinsey found organizations with clear responsible-AI ownership score 2.6 on a four-point maturity scale, versus 1.8 for those with no accountable function.

The structural gap is real. Only about a third of organizations reach level three or higher in governance maturity, and the share of companies with AI-specific governance roles has crept from just 14% to 17% (Stanford HAI, 2026). Meanwhile 40% of large organizations now report scaling AI agents, up from 27% a year earlier (McKinsey, August 2026). Deployment is accelerating faster than accountability.

In practice, ownership splits cleanly: the process owner owns the outcome metrics and the override threshold, engineering owns traces and drift alerting, and one named person has a standing mandate to turn the agent off. If no one can disable it without a meeting, you don't have monitoring — you have logging. We go deeper on this in who should own AI automation inside a mid-market company.

The monitoring nobody budgets for is the reason pilots die

Here's the pattern we see repeatedly: attention peaks at launch and collapses the moment the agent starts doing real work. Dynatrace captured it almost too neatly — 69% of organizations use observability during implementation, but only 57% during operationalization. Monitoring effort drops exactly when the stakes rise.

Then the slow death. Quality degrades invisibly, the team quietly routes around the agent, usage decays, and nine months later someone cancels the project citing "unclear business value." Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, driven by surging costs, vague business outcomes, and immature risk management frameworks (via Forbes, 2025).

"Vague business outcomes" is mostly a measurement failure, not a value failure. You cannot defend an agent's ROI in a budget review if the only evidence you have is that it was up 99.9% of the time. MIT's Project NANDA research found roughly 95% of enterprise GenAI pilots deliver no measurable P&L impact (via Fortune, 2025) — and you cannot demonstrate P&L impact you never instrumented. It's the same root cause behind why most AI pilots never reach production and the pilot-to-production handoff most companies skip.

The honest version: monitoring is roughly 15–25% of the build effort on any agent we put into production at Mesh Flow, and we price it in from day one rather than selling it later as a phase two nobody approves. An unmonitored agent isn't a cheaper agent. It's an undated liability.

And the gap between deploying and capturing is still enormous. Deloitte's State of AI in the Enterprise 2026 found 74% of organizations hope to grow revenue through AI while only 20% already do, and 37% are still using AI at a surface level with little or no process change. McKinsey's 2026 survey puts the number attributing any EBIT impact to AI at 37%, essentially flat year over year. Measurement is what separates the two groups.

Frequently Asked Questions

How do you monitor an AI agent if you didn't build it in-house? Demand three things from the vendor before you sign: per-task traces you can export, an override or correction rate you can see, and an audit log of agent actions. If a vendor can't show you why a specific output happened, you've outsourced the work and kept the liability. Gartner estimated only around 130 of the thousands of self-described agentic vendors meet the real definition of autonomous agency — so ask.

What's the single most useful metric to start with? Human override rate. It requires no new infrastructure — just a thumbs-down or an "edited" flag in the interface your team already uses — and it moves before anything else does. Pair it with task completion rate and you have 80% of the value of a full observability stack in about a week of work.

How often should evals run? Continuously on sampled live traffic, plus a full regression run on every model, prompt, or tool change. Point-in-time pre-launch testing tells you the agent worked on launch day against the world as it was that morning. Models get updated underneath you and your data drifts; a one-time eval has a short shelf life.

Isn't this just normal observability with a new label? Partly — traces, logs, and drift detection all carry over. What's genuinely new is that correctness is probabilistic and task-specific rather than binary. Hallucination rates across 26 models span 22% to 94% depending on the task (Stanford HAI, 2026), so you have to measure quality on your workflow, with your data. No vendor benchmark substitutes for that.

Do we need to monitor an agent that only makes recommendations a human approves? Yes, and the metric is whether the human is actually reviewing. Rubber-stamping is the common failure: approval rates near 100% usually mean nobody is really reading. Track override rate and review time; if both collapse toward zero, you have an autonomous agent with a human-shaped alibi.

The bottom line

  • Uptime and latency measure whether your agent ran, not whether it was right. A confidently wrong answer is a successful request.
  • Instrument six signals: task completion rate, human override rate, evals on sampled live traffic, step traces, drift, and cost per completed task.
  • Start with override rate. It's nearly free to collect and it moves earlier than anything else.
  • Multiply, don't add: 98% per-step reliability across 20 steps is roughly 67% end-to-end. Put checkpoints where a wrong answer is expensive to reverse.
  • Name an owner who can switch the agent off without convening a meeting.

If you're putting an agent into a workflow that touches customers or money and you're not sure what you'd actually watch, that's the conversation worth having before launch rather than after. You can see how we approach it at mesh-flow.com.

Sources

Filippo Pietrantonio

Founder of Mesh Flow. Builds and ships AI automation systems for mid-market companies and founders.