{"id":24491,"date":"2026-11-20T18:33:26","date_gmt":"2026-11-20T18:33:26","guid":{"rendered":"https:\/\/dianapps.com\/blog\/?p=24491"},"modified":"2026-11-22T18:34:32","modified_gmt":"2026-11-22T18:34:32","slug":"how-to-measure-ai-agent-roi","status":"publish","type":"post","link":"https:\/\/prod-wp.dianapps.com\/blog\/how-to-measure-ai-agent-roi\/","title":{"rendered":"How to Measure AI Agent ROI: Containment Rate, Hallucination Benchmarks &#038; Evaluation Tools"},"content":{"rendered":"<p><b>Key Takeaways<\/b><span style=\"font-weight: 400;\">:\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">AI agent ROI must account for verified task completion, recovered revenue, operational savings, infrastructure costs, human reviews and expected error costs.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Mature AI support deployments typically achieve 55% to 60% containment, while best-in-class programmes can exceed 80%.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">True containment requires verified resolution, repeat-contact checks and human audits rather than counting every deflected conversation as a success.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Hallucination evaluation combines public benchmarks, custom golden datasets, faithfulness scoring, tool output checks and regular human reviews.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Tools such as LangSmith, Langfuse, Arize Phoenix, Braintrust, DeepEval and promptfoo help track agent performance, detect regressions and measure production quality.<\/span><\/li>\n<\/ul>\n<blockquote><p><b>Quick Answer<\/b><span style=\"font-weight: 400;\">: Measure AI agent ROI by comparing the value of successfully completed tasks, recovered revenue and time saved against total costs, including tokens, infrastructure, tooling, human reviews and expected errors. Track true containment, task success, escalation, repeat contacts, hallucination rates and cost per resolution to verify performance. Observability and evaluation tools such as LangSmith, Langfuse, Braintrust and DeepEval help monitor quality and catch regressions in production.<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Somewhere in most companies right now there is a slide claiming an AI agent saved four hundred thousand dollars, and nobody in the room can explain how that number was calculated. Learning how to measure AI agent ROI properly is the difference between a pilot that quietly expands and one that gets cancelled the first time finance asks a hard question, and the maths is more subtle than counting deflected tickets.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you\u2019re still deciding whether the thing you built is a copilot assisting people or an agent acting on its own, our comparison of an<\/span><a href=\"https:\/\/dianapps.com\/blog\/ai-copilot-vs-ai-agent\/\"> <span style=\"font-weight: 400;\">AI copilot vs an AI agent<\/span><\/a><span style=\"font-weight: 400;\"> is settling that first, because the two are measured differently.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Why Measuring AI Agent ROI Is Harder Than Measuring a Chatbot?<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">A chatbot is answering questions, while an agent is completing tasks, calling tools and changing records, and that shift is turning a simple deflection count into a question about outcomes, quality and risk. The measurement problem is growing with autonomy, and when we\u2019re scoping an ROI model with a client, these are the complications that keep appearing:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">An agent can \u201chandle\u201d a conversation without resolving it, so handled counts are overstating value unless resolution is verified afterwards.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A single hallucinated refund or a wrong tool call can wipe out the savings of a thousand correct ones, so error cost belongs in the model.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Agents are consuming variable tokens per task, which is making cost per resolution a distribution rather than a fixed price.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Many benefits are landing outside the support budget, such as revenue recovered after hours or engineering time returned, and they\u2019re easy to miss.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">None of this is making ROI unmeasurable, but it is meaning that the model needs more than one metric, and the reasons are not hard to see once you\u2019ve seen a containment number collapse under a repeat contact audit, so let\u2019s start with the formula.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">The ROI Formula: Value Created Minus Fully Loaded Cost<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Every credible agent ROI calculation is reducing to one line, which is value created minus fully loaded cost, divided by that cost, and the discipline is in refusing to leave anything out of either side. Value is including the cost of the human work the agent is replacing, measured as resolved conversations multiplied by your cost per human contact, plus any revenue the agent is capturing that would otherwise be lost, plus time returned to people who are now doing higher value work.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Cost is including model tokens, hosting, the observability and evaluation tooling you\u2019ll read about below, the human review hours needed to keep quality honest, the amortised build cost, and the expected cost of errors, which is the probability of a damaging mistake multiplied by what it costs when it happens. Written out, the formula is reading as ROI equals (resolved tasks \u00d7 human cost per task + recovered revenue + time value) minus (tokens + infrastructure + tooling + review hours + amortised build + expected error cost), all divided by that cost total. The first term is where containment rate comes in, which is the next topic.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">AI Agent Containment Rate Benchmark<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Containment rate is the share of conversations an agent is resolving end to end without a human, and it is the single number most leaders are asking for, so it is worth knowing where the published bands sit before you present your own.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">A 2026 benchmark report from<\/span><a href=\"https:\/\/jugl.com\/resources\/ai-agent-benchmark-report\/\"> <span style=\"font-weight: 400;\">Jugl<\/span><\/a><span style=\"font-weight: 400;\">, drawing on Zendesk, Gartner and Salesforce data, is placing mature deployments at a median of fifty five to sixty percent, with strong programmes reaching the high sixties and best in class deployments passing eighty, and it is pairing those figures with escalation and repeat contact bands that keep them honest. The table below is summarising the AI agent containment rate benchmark we\u2019re using with clients this year.<\/span><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Deployment Stage or Capability<\/b><\/td>\n<td><b>Weak<\/b><\/td>\n<td><b>Median<\/b><\/td>\n<td><b>Strong<\/b><\/td>\n<td><b>Best in Class<\/b><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Launch band (first 90 days)<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Below 35%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">40% to 50%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">50% to 55%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Above 55%<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Mature deployment (6 to 12 months)<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Below 45%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">55% to 60%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">65% to 72%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">80% and above<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Scripted FAQ bot (for comparison)<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Below 25%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">25% to 35%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">35% to 45%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Rare above 45%<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Agent with real system actions<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Below 50%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">55% to 65%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">65% to 75%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">75% and above<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Escalation rate (healthy range)<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Above 50%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">25% to 35%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">20% to 25%<\/span><\/td>\n<td><span style=\"font-weight: 400;\">15% to 22%<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-weight: 400;\">A containment figure sitting above the best in class band with no escalation data beside it is almost always a deflection number wearing a containment label, and that distinction is the next section.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">True Containment vs Deflection: The Vanity Metric Trap<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Deflection is counting conversations kept away from the human queue, while true containment is counting conversations that were actually resolved, and the gap between the two is where most inflated ROI claims are hiding. A bot that posts a help centre link and closes the chat is deflecting beautifully and resolving nothing, so the measurement has to look past the moment the conversation ends, and these are the checks we\u2019re insisting on:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Count a conversation as contained only if the customer isn\u2019t contacting again about the same issue within forty eight hours or seven days.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Exclude conversations the customer abandoned mid flow, because an abandoned chat is a failure that is dressed as a resolution.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Verify a sample of contained conversations with human reviewers each week, scoring whether the stated resolution actually matched the customer\u2019s need.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Split containment by intent, since a ninety percent order tracking rate is hiding a thirty percent billing dispute rate inside one average.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Once containment is defined this strictly, the number is usually dropping by ten to twenty points, and that lower figure is the one that will survive an audit and the one your ROI model should use.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Escalation, Repeat Contact and CSAT: The Metrics That Keep Containment Honest<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Three companion metrics are deciding whether a containment rate is worth celebrating, and any ROI report without them is incomplete. Escalation rate is the share of conversations handed to a human, and a healthy agent is sitting between twenty and thirty five percent, because a figure below fifteen is usually meaning the agent is refusing to hand off cases it should, and the escalation reasons are telling you exactly which intents to improve next.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Repeat contact rate is measuring customers who came back within a window, and the published median for AI resolved conversations is around eleven percent against under nine for human resolved ones, so a repeat rate climbing above fifteen is a signal that containment is being bought with incomplete answers. Customer satisfaction should be split between contained and escalated conversations rather than averaged, because an agent with a four out of five score on contained chats and a two on escalated ones is telling two very different stories. Tracked together, these four numbers are forming the operational core of every honest support agent ROI model.<\/span><\/p>\n<div style=\"background: #EEF2FE; border: 1px solid #DBE2FB; border-radius: 14px; padding: 28px 32px; margin: 38px 0;\">\n<p style=\"color: #1b3fae; font-size: 22px; line-height: 1.3; font-weight: bold; margin: 0 0 10px;\"><span style=\"font-weight: 400;\">Not Sure Your Containment Number Would Survive an Audit?<\/span><\/p>\n<p style=\"color: #4b5563; font-size: 16px; line-height: 1.6; margin: 0 0 22px;\"><span style=\"font-weight: 400;\">We\u2019re reviewing agent metrics against published benchmarks and rebuilding ROI models leadership can trust.<\/span><\/p>\n<p><a style=\"display: inline-block; background: #2563EB; color: #ffffff; text-decoration: none; font-size: 15px; font-weight: 600; padding: 13px 26px; border-radius: 8px;\" href=\"https:\/\/dianapps.com\/contact\">Get a Metrics Review<\/a><\/p>\n<\/div>\n<h2><span style=\"font-weight: 400;\">How to Measure Autonomous AI Agent Performance Beyond Support?<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Containment is a support metric, and a growing share of agents are doing something else entirely, such as processing invoices, triaging alerts, writing code or researching accounts, so the question of how to measure autonomous AI agent performance is needing a broader toolkit. For task oriented agents we\u2019re replacing containment with a small set of execution metrics that translate across domains:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Task success rate, judged against a defined outcome rather than the agent\u2019s own claim of completion, with a human verified sample every week.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Tool call accuracy, meaning the share of tool invocations that used the right tool with correct arguments, which is catching failures before they reach outcomes.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Steps and tokens per completed task, since an agent that succeeds in twelve steps where three would do is quietly burning the ROI.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Recovery rate, which is how often the agent is correcting its own error after a failed tool call rather than looping or giving up.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Human intervention rate, tracked separately from escalation, because every intervention is costing reviewer time that belongs in the model.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">These metrics are also the fairest way to compare an agent against the alternative it is replacing, and our comparison of<\/span><a href=\"https:\/\/dianapps.com\/blog\/ai-agents-vs-rpa-vs-traditional-automation\/\"> <span style=\"font-weight: 400;\">AI agents vs RPA vs traditional automation<\/span><\/a><span style=\"font-weight: 400;\"> is showing why a deterministic script sometimes wins that contest on every number.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Hallucination Benchmarks: From Public Leaderboards to Your Own Golden Set<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Hallucination is the risk that is turning a positive ROI negative overnight, and measuring it is beginning with public benchmarks and ending with a dataset nobody else can build for you. Public leaderboards such as Vectara\u2019s hallucination evaluation model rankings are giving you a model level view of how often a given model is inventing content when summarising a document, and task oriented benchmarks such as tau bench are measuring whether an agent follows policy and uses tools correctly across simulated customer scenarios, which is far closer to production behaviour than a question answering score.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Those numbers are useful for choosing a model and useless for proving your agent is safe, because your prompts, your tools, your retrieval corpus and your customers are all different from the benchmark\u2019s. The asset that matters is a golden set, which is a few hundred real tasks drawn from production, each with a verified correct outcome and the sources the answer should rest on, refreshed monthly as new failure modes appear. Every evaluation in the next section is running against that set.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Building a Hallucination Evaluation Pipeline<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">A hallucination pipeline is scoring every agent response against evidence, and in production that evaluation has to run continuously on a sample rather than once before launch, because prompts, models and data are all drifting. The pipeline we\u2019re building for clients is combining automated scoring with human calibration, and the layers are these:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Faithfulness scoring, where a judge model is checking each claim in the response against the retrieved context and flagging anything unsupported.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Groundedness and citation checks, confirming that every referenced policy, price or record actually exists in the source system the agent queried.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Tool output consistency, comparing what a tool returned with what the agent told the user, which is catching the subtle misreadings that pure text checks miss.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A weekly human review of a stratified sample, used both to catch what the judge missed and to recalibrate the judge\u2019s threshold.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Hallucination rate reported per intent and per model version, so a regression after a model upgrade is visible within a day.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Report the hallucination rate alongside its financial cost, meaning rate multiplied by volume multiplied by average damage per incident, because that product is the number finance will understand, and it belongs on the cost side of the ROI line.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Cost Metrics: Tokens, Latency and Cost Per Resolution<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The cost side of the formula is where agent economics are diverging from chatbot economics, because an agent is making several model calls per task and the count is varying with difficulty.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The unit that is tying everything together is cost per resolution, which is total agent cost divided by truly contained or successfully completed tasks, and the table below is showing the components we\u2019re tracking and the ranges typical of 2026 deployments.<\/span><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Cost Metric<\/b><\/td>\n<td><b>How to Measure It<\/b><\/td>\n<td><b>Typical 2026 Range<\/b><\/td>\n<td><b>Why It Matters for ROI<\/b><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Tokens per task<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Input plus output tokens across all calls in one task<\/span><\/td>\n<td><span style=\"font-weight: 400;\">3,000 to 15,000 for support, far higher for research agents<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Drives model spend directly<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Model cost per task<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Tokens multiplied by blended per million token price<\/span><\/td>\n<td><span style=\"font-weight: 400;\">$0.01 to $0.10 for support, $0.50 and above for multi step research<\/span><\/td>\n<td><span style=\"font-weight: 400;\">The variable cost line<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Cost per resolution<\/span><\/td>\n<td><span style=\"font-weight: 400;\">All agent costs divided by truly resolved tasks<\/span><\/td>\n<td><span style=\"font-weight: 400;\">$0.30 to $2.00 including tooling and review<\/span><\/td>\n<td><span style=\"font-weight: 400;\">The number to compare with human cost per contact<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Latency per turn<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Time from user message to agent response<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Under 3 seconds for chat, under 1.5 for voice<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Slow agents are losing containment to abandonment<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Human review cost<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Reviewer hours multiplied by loaded hourly rate<\/span><\/td>\n<td><span style=\"font-weight: 400;\">5 to 20 hours monthly per 10,000 tasks<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Often forgotten and always real<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Error cost<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Incident rate multiplied by average damage<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Highly variable, must be estimated per intent<\/span><\/td>\n<td><span style=\"font-weight: 400;\">The line that separates honest models from optimistic ones<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-weight: 400;\">Cost per resolution sitting at a tenth of your human cost per contact is the usual sign of a healthy programme, and tracking it weekly is catching token drift long before the invoice does.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Best AI Agent Observability and Evaluation Tools in 2026<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">None of the metrics above are measurable without tooling that is capturing every step of every run, and the market for the best AI agent observability and evaluation tools has matured quickly, with clear leaders for different needs.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The table below summarises the platforms we\u2019re seeing most often in production this year, including what each one is strongest at and where it is fitting best.<\/span><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Tool<\/b><\/td>\n<td><b>Category<\/b><\/td>\n<td><b>Strongest At<\/b><\/td>\n<td><b>Best Fit<\/b><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">LangSmith<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Observability and evaluation<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Per node tracing, cost attribution, datasets and online evaluators<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Teams on LangGraph or LangChain<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Langfuse<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Open source observability<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Self hosted tracing, prompt management, scoring APIs<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Teams needing data to stay in house<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Arize Phoenix<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Open source evaluation and tracing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Embedding analysis, drift detection, OpenTelemetry native traces<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Data science led teams<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Braintrust<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Evaluation platform<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Experiment comparison, scorers, CI integration<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Product teams iterating on prompts weekly<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Weights and Biases Weave<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Tracing and evaluation<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Experiment tracking lineage, model comparison<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Teams already using W&amp;B for ML<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Datadog LLM Observability<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Enterprise monitoring<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Unified infra, app and LLM telemetry with alerting<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Enterprises standardised on Datadog<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Galileo and Patronus<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Guardrails and evaluation<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Hallucination detection, safety scoring, policy checks<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Regulated and high stakes deployments<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">DeepEval, Ragas and promptfoo<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Open source test frameworks<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Unit style tests for faithfulness, relevance and regressions<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Engineering teams wanting evaluation in CI<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-weight: 400;\">Most mature stacks are combining one tracing platform with one open source test framework, and if you\u2019re building directly on a model provider\u2019s SDK, the patterns in our guide to<\/span><a href=\"https:\/\/dianapps.com\/blog\/how-to-build-ai-agents-with-openai\/\"> <span style=\"font-weight: 400;\">building AI agents with OpenAI<\/span><\/a><span style=\"font-weight: 400;\"> are plugging into any of these through OpenTelemetry.<\/span><\/p>\n<div style=\"background: #EEF2FE; border: 1px solid #DBE2FB; border-radius: 14px; padding: 28px 32px; margin: 38px 0;\">\n<p style=\"color: #1b3fae; font-size: 22px; line-height: 1.3; font-weight: bold; margin: 0 0 10px;\"><span style=\"font-weight: 400;\">Want Tracing and Evaluation Set Up Before Launch?<\/span><\/p>\n<p style=\"color: #4b5563; font-size: 16px; line-height: 1.6; margin: 0 0 22px;\"><span style=\"font-weight: 400;\">We\u2019re instrumenting agent systems with observability, golden sets and automated evaluation in under two weeks.<\/span><\/p>\n<p><a style=\"display: inline-block; background: #2563EB; color: #ffffff; text-decoration: none; font-size: 15px; font-weight: 600; padding: 13px 26px; border-radius: 8px;\" href=\"https:\/\/dianapps.com\/contact\">Talk to Our AI Engineers<\/a><\/p>\n<\/div>\n<h2><span style=\"font-weight: 400;\">Choosing Between Observability, Evaluation and Guardrail Tools<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The tool table is hiding a distinction that confuses many buyers, which is that observability, evaluation and guardrails are three different jobs that some products bundle and others don\u2019t, and buying one while expecting the others is a common and expensive mistake. When a client is asking which to prioritise, we\u2019re explaining the three like this:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Observability is recording what happened, meaning every call, tool invocation, token and latency, and it is the foundation nothing else works without.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Evaluation is scoring whether what happened was good, offline against a golden set before deploy and online on sampled production traffic afterwards.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Guardrails are preventing bad outputs in real time, such as blocking an unsupported claim or a disallowed action before it reaches the user.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Budget for all three, but sequence them, since tracing on day one, evaluation before the first prompt change, and guardrails before any high stakes action is the order that is working.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">A team with excellent guardrails and no evaluation is flying blind on quality, and a team with excellent evaluation and no observability can\u2019t explain a single failure, so the stack needs to be complete before ROI claims mean anything.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Setting Up Evaluation as a CI Gate<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The most valuable thing an evaluation tool is doing isn\u2019t producing a dashboard, it is blocking a deploy, and treating evaluation as a continuous integration gate is what is turning quality from a periodic review into an engineering guarantee. The pattern is storing the golden set alongside the code, running the full evaluation suite on every change to a prompt, a tool definition, a retrieval index or a model version, and failing the build when task success, faithfulness or tool accuracy is dropping below an agreed threshold or when cost per task is rising beyond a tolerance.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Frameworks such as promptfoo and DeepEval are making this a few hundred lines of configuration, Braintrust and LangSmith are offering hosted experiment comparison for the same purpose, and the judge models inside them should themselves be pinned to a version so that a judge upgrade can\u2019t silently move the goalposts. Teams running this gate are catching the majority of regressions before customers do, and they\u2019re also generating a continuous record of quality over time, which is exactly the evidence an ROI report needs when leadership asks whether the agent is getting better or worse.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Security and Risk Metrics for High Stakes Agents<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Some agents are operating where a single wrong action is carrying consequences far beyond a poor customer experience, such as agents with write access to financial systems, infrastructure or security tooling, and for those the ROI model is needing a risk section that is as rigorous as the value section. The metrics that belong there are prompt injection attempt and success rates measured through regular red team runs, the share of actions that passed through a human approval gate, the rate of privilege or scope violations caught by policy checks, data leakage incidents from logs and outputs, and the mean time to detect and contain an agent misbehaviour once it starts.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Each of these is feeding the expected error cost line in the formula, and for security operations in particular the sums are large, which is why our guide to<\/span><a href=\"https:\/\/dianapps.com\/blog\/ai-agents-for-cybersecurity\/\"> <span style=\"font-weight: 400;\">AI agents for cybersecurity<\/span><\/a><span style=\"font-weight: 400;\"> is treating measurement and containment as inseparable. An agent that is saving two hundred analyst hours a month while creating a five percent chance of a serious incident isn\u2019t delivering positive ROI until that risk is priced and reduced.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">A Worked ROI Example: A Support Agent at 10,000 Conversations a Month<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Numbers are making all of this concrete, so consider a support agent handling ten thousand conversations a month at a company whose loaded cost per human contact is eight dollars. True containment after excluding repeat contacts and abandonments is landing at fifty three percent, which is five thousand three hundred resolved conversations, and multiplying by eight dollars is giving forty two thousand four hundred dollars of avoided human cost, with after hours revenue recovery adding a further three thousand that the sales team is verifying through order data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">On the cost side, model spend at roughly six thousand tokens per conversation and a blended price of three dollars per million tokens is around one hundred eighty dollars, observability and evaluation tooling is costing two thousand, twenty hours of human review at forty dollars is eight hundred, the one hundred fifty thousand dollar build amortised over twenty four months is six thousand two hundred fifty, and expected error cost at a two percent damaging error rate on billing intents is estimated at one thousand two hundred. Total cost is roughly ten thousand four hundred against forty five thousand four hundred of value, which is a monthly ROI near three hundred thirty percent, and every input is traceable.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Reporting ROI to Leadership Without Overclaiming<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The worked example is only persuasive because every line is defensible, and the way results are presented is deciding whether the programme earns trust or suspicion. Having sat through a lot of these reviews, we\u2019re recommending the following habits for every agent ROI report:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Lead with true containment or task success and show the stricter definition you used, because an executive who later discovers a looser number is losing faith in all of them.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Show cost per resolution against human cost per contact on the same chart, since that ratio is the most intuitive expression of value.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Present the hallucination and error cost line openly, with the trend, because hiding it is the fastest way to lose credibility when an incident happens.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Separate hard savings from soft benefits such as time returned, and never add the two into one headline figure.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Include the human review hours, because a programme that is claiming full automation while quietly consuming a reviewer\u2019s week isn\u2019t being honest with itself.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Agents producing content are tempting teams toward the loosest metrics of all, and our comparison of<\/span><a href=\"https:\/\/dianapps.com\/blog\/ai-writing-software-vs-autonomous-ai-agent\/\"> <span style=\"font-weight: 400;\">AI writing software vs an autonomous AI agent<\/span><\/a><span style=\"font-weight: 400;\"> is explaining why output volume is never the number to report.<\/span><\/p>\n<div style=\"background: #EEF2FE; border: 1px solid #DBE2FB; border-radius: 14px; padding: 28px 32px; margin: 38px 0;\">\n<p style=\"color: #1b3fae; font-size: 22px; line-height: 1.3; font-weight: bold; margin: 0 0 10px;\"><span style=\"font-weight: 400;\">Ready to Prove Your Agent Is Paying for Itself?<\/span><\/p>\n<p style=\"color: #4b5563; font-size: 16px; line-height: 1.6; margin: 0 0 22px;\"><span style=\"font-weight: 400;\">We\u2019re building ROI models, evaluation pipelines and reporting that stand up to finance and compliance review.<\/span><\/p>\n<p><a style=\"display: inline-block; background: #2563EB; color: #ffffff; text-decoration: none; font-size: 15px; font-weight: 600; padding: 13px 26px; border-radius: 8px;\" href=\"https:\/\/dianapps.com\/contact\">Contact Us Today<\/a><\/p>\n<\/div>\n<h2><span style=\"font-weight: 400;\">Common Mistakes That Inflate or Hide Agent ROI<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">After reviewing dozens of agent programmes, we\u2019re seeing the same measurement errors in nearly every one, and most are cutting both ways, overstating value in the pitch and understating it later when the programme is defending its budget.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Check your own model against the following before it goes anywhere near a board deck:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Counting deflection as containment, which is the single most common inflation and is usually worth ten to twenty points.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Using a human cost per contact that includes nothing but salary, which is understating the value the agent is creating.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Ignoring token drift, where prompts and context windows grow month over month and cost per resolution doubles without anyone noticing.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Evaluating once before launch and never again, so a model upgrade or a retrieval change is silently degrading quality for weeks.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Averaging CSAT across contained and escalated conversations, hiding a problem in one group behind the other.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Leaving errors cost out entirely, which is making the model look generous until the first incident makes it look dishonest.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">None of these are hard to fix, but all of them are easier to fix before the first ROI number has been promised, so build the model properly the first time.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Final Thoughts<\/span><\/h2>\n<p>Measuring AI agent ROI goes beyond tracking automation rates or the number of tasks completed. Businesses need to evaluate how effectively agents resolve user queries through containment rate, how reliably they generate accurate responses through hallucination benchmarks, and how consistently they perform using structured evaluation tools. These metrics help organizations identify performance gaps, reduce operational costs, improve customer experiences, and make informed decisions about scaling AI automation.<\/p>\n<p><span style=\"font-weight: 400;\">Teams that are defining containment as true resolution, tracking escalation and repeat contact beside it, scoring hallucinations against their own golden set, running evaluation as a deploy gate, and pricing error cost into the formula are producing ROI figures that grow more credible every month rather than less, and that credibility is what is unlocking the budget to expand.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The tooling to do this is mature in 2026 and mostly affordable, so the remaining work is organisational rather than technical, and if you\u2019d rather partner with a team that has built these models and pipelines before, our guide on<\/span><a href=\"https:\/\/dianapps.com\/blog\/how-to-choose-an-ai-agent-development-company\/\"> <span style=\"font-weight: 400;\">how to choose an AI agent development company<\/span><\/a><span style=\"font-weight: 400;\"> is the place to start, and we\u2019ve got you covered.<\/span><\/p>\n<div>\n<p>However, achieving measurable ROI requires more than deploying an AI agent. It involves choosing the right use cases, designing reliable workflows, integrating relevant business systems, and continuously monitoring agent performance. With the right strategy and implementation, AI agents can become valuable business assets rather than just another automation experiment.<\/p>\n<p>If your business is planning to adopt intelligent automation, explore <a href=\"https:\/\/dianapps.com\/ai-agent-development-services\"><strong>AI Agent Development Services<\/strong><\/a> to build, evaluate, and scale AI-powered solutions aligned with your business goals.<\/p>\n<style>.elementor-24495 .elementor-element.elementor-element-2932a52{text-align:left;}.elementor-24495 .elementor-element.elementor-element-2932a52 > .elementor-widget-container{margin:0px 0px 0px 0px;}.elementor-24495 .elementor-element.elementor-element-0b767d1 .elementor-tab-title{border-width:1px;border-color:#00000014;}.elementor-24495 .elementor-element.elementor-element-0b767d1 .elementor-tab-content{border-width:1px;border-bottom-color:#00000014;}.elementor-24495 .elementor-element.elementor-element-0b767d1 > .elementor-widget-container{margin:0px 0px 0px 0px;}<\/style><div class=\"porto-block elementor elementor-24495\">\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-27707ca elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"27707ca\" data-element_type=\"section\">\r\n\t\t\t\r\n\t\t\t\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\r\n\t\t\t\t\t\t\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-0163611\" data-id=\"0163611\" data-element_type=\"column\">\r\n\r\n\t\t\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\r\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-03a2969 elementor-widget elementor-widget-text-editor\" data-id=\"03a2969\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.14.0 - 26-06-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2932a52 elementor-widget elementor-widget-heading\" data-id=\"2932a52\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.14.0 - 26-06-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-large\">FAQs <\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0b767d1 elementor-widget elementor-widget-toggle\" data-id=\"0b767d1\" data-element_type=\"widget\" data-widget_type=\"toggle.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.14.0 - 26-06-2023 *\/\n.elementor-toggle{text-align:left}.elementor-toggle .elementor-tab-title{font-weight:700;line-height:1;margin:0;padding:15px;border-bottom:1px solid #d5d8dc;cursor:pointer;outline:none}.elementor-toggle .elementor-tab-title .elementor-toggle-icon{display:inline-block;width:1em}.elementor-toggle .elementor-tab-title .elementor-toggle-icon svg{-webkit-margin-start:-5px;margin-inline-start:-5px;width:1em;height:1em}.elementor-toggle .elementor-tab-title .elementor-toggle-icon.elementor-toggle-icon-right{float:right;text-align:right}.elementor-toggle .elementor-tab-title .elementor-toggle-icon.elementor-toggle-icon-left{float:left;text-align:left}.elementor-toggle .elementor-tab-title .elementor-toggle-icon .elementor-toggle-icon-closed{display:block}.elementor-toggle .elementor-tab-title .elementor-toggle-icon .elementor-toggle-icon-opened{display:none}.elementor-toggle .elementor-tab-title.elementor-active{border-bottom:none}.elementor-toggle .elementor-tab-title.elementor-active .elementor-toggle-icon-closed{display:none}.elementor-toggle .elementor-tab-title.elementor-active .elementor-toggle-icon-opened{display:block}.elementor-toggle .elementor-tab-content{padding:15px;border-bottom:1px solid #d5d8dc;display:none}@media (max-width:767px){.elementor-toggle .elementor-tab-title{padding:12px}.elementor-toggle .elementor-tab-content{padding:12px 10px}}.e-con-inner>.elementor-widget-toggle,.e-con>.elementor-widget-toggle{width:var(--container-widget-width);--flex-grow:var(--container-widget-flex-grow)}<\/style>\t\t<div class=\"elementor-toggle\">\n\t\t\t\t\t\t\t<div class=\"elementor-toggle-item\">\n\t\t\t\t\t<h3 id=\"elementor-tab-title-1201\" class=\"elementor-tab-title\" data-tab=\"1\" role=\"button\" aria-controls=\"elementor-tab-content-1201\" aria-expanded=\"false\">\n\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon elementor-toggle-icon-left\" aria-hidden=\"true\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-closed\"><i class=\"fas fa-caret-right\"><\/i><\/span>\n\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-opened\"><i class=\"elementor-toggle-icon-opened fas fa-caret-up\"><\/i><\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-toggle-title\" tabindex=\"0\">What is a good AI agent containment rate in 2026?<\/a>\n\t\t\t\t\t<\/h3>\n\n\t\t\t\t\t<div id=\"elementor-tab-content-1201\" class=\"elementor-tab-content elementor-clearfix\" data-tab=\"1\" role=\"region\" aria-labelledby=\"elementor-tab-title-1201\"><p>Mature deployments are sitting at a median of fifty five to sixty percent, with best in class programmes passing eighty.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<div class=\"elementor-toggle-item\">\n\t\t\t\t\t<h3 id=\"elementor-tab-title-1202\" class=\"elementor-tab-title\" data-tab=\"2\" role=\"button\" aria-controls=\"elementor-tab-content-1202\" aria-expanded=\"false\">\n\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon elementor-toggle-icon-left\" aria-hidden=\"true\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-closed\"><i class=\"fas fa-caret-right\"><\/i><\/span>\n\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-opened\"><i class=\"elementor-toggle-icon-opened fas fa-caret-up\"><\/i><\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-toggle-title\" tabindex=\"0\">What is the difference between containment and deflection?<\/a>\n\t\t\t\t\t<\/h3>\n\n\t\t\t\t\t<div id=\"elementor-tab-content-1202\" class=\"elementor-tab-content elementor-clearfix\" data-tab=\"2\" role=\"region\" aria-labelledby=\"elementor-tab-title-1202\"><p>Deflection counts conversations kept from humans, while containment counts conversations that were actually resolved end to end.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<div class=\"elementor-toggle-item\">\n\t\t\t\t\t<h3 id=\"elementor-tab-title-1203\" class=\"elementor-tab-title\" data-tab=\"3\" role=\"button\" aria-controls=\"elementor-tab-content-1203\" aria-expanded=\"false\">\n\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon elementor-toggle-icon-left\" aria-hidden=\"true\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-closed\"><i class=\"fas fa-caret-right\"><\/i><\/span>\n\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-opened\"><i class=\"elementor-toggle-icon-opened fas fa-caret-up\"><\/i><\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-toggle-title\" tabindex=\"0\">How do you measure hallucinations in a production agent?<\/a>\n\t\t\t\t\t<\/h3>\n\n\t\t\t\t\t<div id=\"elementor-tab-content-1203\" class=\"elementor-tab-content elementor-clearfix\" data-tab=\"3\" role=\"region\" aria-labelledby=\"elementor-tab-title-1203\"><p>Score sampled responses for faithfulness against retrieved sources and tool outputs, calibrated with weekly human review.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<div class=\"elementor-toggle-item\">\n\t\t\t\t\t<h3 id=\"elementor-tab-title-1204\" class=\"elementor-tab-title\" data-tab=\"4\" role=\"button\" aria-controls=\"elementor-tab-content-1204\" aria-expanded=\"false\">\n\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon elementor-toggle-icon-left\" aria-hidden=\"true\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-closed\"><i class=\"fas fa-caret-right\"><\/i><\/span>\n\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-opened\"><i class=\"elementor-toggle-icon-opened fas fa-caret-up\"><\/i><\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-toggle-title\" tabindex=\"0\">Which tools are best for AI agent observability and evaluation?<\/a>\n\t\t\t\t\t<\/h3>\n\n\t\t\t\t\t<div id=\"elementor-tab-content-1204\" class=\"elementor-tab-content elementor-clearfix\" data-tab=\"4\" role=\"region\" aria-labelledby=\"elementor-tab-title-1204\"><p>LangSmith, Langfuse, Arize Phoenix and Braintrust lead tracing and evaluation, with DeepEval and promptfoo for CI tests.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<div class=\"elementor-toggle-item\">\n\t\t\t\t\t<h3 id=\"elementor-tab-title-1205\" class=\"elementor-tab-title\" data-tab=\"5\" role=\"button\" aria-controls=\"elementor-tab-content-1205\" aria-expanded=\"false\">\n\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon elementor-toggle-icon-left\" aria-hidden=\"true\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-closed\"><i class=\"fas fa-caret-right\"><\/i><\/span>\n\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-opened\"><i class=\"elementor-toggle-icon-opened fas fa-caret-up\"><\/i><\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-toggle-title\" tabindex=\"0\">How do you calculate cost per resolution for an AI agent?<\/a>\n\t\t\t\t\t<\/h3>\n\n\t\t\t\t\t<div id=\"elementor-tab-content-1205\" class=\"elementor-tab-content elementor-clearfix\" data-tab=\"5\" role=\"region\" aria-labelledby=\"elementor-tab-title-1205\"><p>Divide all agent costs, including tokens, tooling and review hours, by the number of truly resolved tasks.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<div class=\"elementor-toggle-item\">\n\t\t\t\t\t<h3 id=\"elementor-tab-title-1206\" class=\"elementor-tab-title\" data-tab=\"6\" role=\"button\" aria-controls=\"elementor-tab-content-1206\" aria-expanded=\"false\">\n\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon elementor-toggle-icon-left\" aria-hidden=\"true\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-closed\"><i class=\"fas fa-caret-right\"><\/i><\/span>\n\t\t\t\t\t\t\t\t<span class=\"elementor-toggle-icon-opened\"><i class=\"elementor-toggle-icon-opened fas fa-caret-up\"><\/i><\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-toggle-title\" tabindex=\"0\">Should error cost be included in AI agent ROI?<\/a>\n\t\t\t\t\t<\/h3>\n\n\t\t\t\t\t<div id=\"elementor-tab-content-1206\" class=\"elementor-tab-content elementor-clearfix\" data-tab=\"6\" role=\"region\" aria-labelledby=\"elementor-tab-title-1206\"><p>Yes, multiply the damaging error rate by volume and average damage and put it on the cost side.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t<script type=\"application\/ld+json\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@type\":\"FAQPage\",\"mainEntity\":[{\"@type\":\"Question\",\"name\":\"What is a good AI agent containment rate in 2026?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"<p>Mature deployments are sitting at a median of fifty five to sixty percent, with best in class programmes passing eighty.<\\\/p>\"}},{\"@type\":\"Question\",\"name\":\"What is the difference between containment and deflection?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"<p>Deflection counts conversations kept from humans, while containment counts conversations that were actually resolved end to end.<\\\/p>\"}},{\"@type\":\"Question\",\"name\":\"How do you measure hallucinations in a production agent?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"<p>Score sampled responses for faithfulness against retrieved sources and tool outputs, calibrated with weekly human review.<\\\/p>\"}},{\"@type\":\"Question\",\"name\":\"Which tools are best for AI agent observability and evaluation?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"<p>LangSmith, Langfuse, Arize Phoenix and Braintrust lead tracing and evaluation, with DeepEval and promptfoo for CI tests.<\\\/p>\"}},{\"@type\":\"Question\",\"name\":\"How do you calculate cost per resolution for an AI agent?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"<p>Divide all agent costs, including tokens, tooling and review hours, by the number of truly resolved tasks.<\\\/p>\"}},{\"@type\":\"Question\",\"name\":\"Should error cost be included in AI agent ROI?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"<p>Yes, multiply the damaging error rate by volume and average damage and put it on the cost side.<\\\/p>\"}}]}<\/script>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\r\n\t\t\t\t<\/div>\r\n\t\t\t\t\t\t<\/div>\r\n\t\t\t\t<\/section>\r\n\t\t<\/div>\n<\/div>\n<div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Key Takeaways:\u00a0 AI agent ROI must account for verified task completion, recovered revenue, operational savings, infrastructure costs, human reviews and expected error costs. Mature AI support deployments typically achieve 55% to 60% containment, while best-in-class programmes can exceed 80%. True containment requires verified resolution, repeat-contact checks and human audits rather than counting every deflected conversation [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":24494,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_yoast_wpseo_meta-robots-noindex":"","_yoast_wpseo_meta-robots-nofollow":"","_yoast_wpseo_canonical":"","_yoast_wpseo_opengraph-title":"","_yoast_wpseo_opengraph-description":"","_yoast_wpseo_opengraph-image":"","_yoast_wpseo_twitter-title":"","_yoast_wpseo_twitter-description":"","_yoast_wpseo_twitter-image":"","_wp_applaud_exclude":false,"footnotes":""},"categories":[1622],"tags":[2956],"class_list":["post-24491","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","tag-how-to-measure-ai-agent-roi"],"featured_image_src":{"landsacpe":["https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi-1140x445.webp",1140,445,true],"list":["https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi-463x348.webp",463,348,true],"medium":["https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi-300x169.webp",300,169,true],"full":["https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp",1536,864,false]},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How to Measure AI Agent ROI: Metrics, Benchmarks &amp; Tools<\/title>\n<meta name=\"description\" content=\"How to measure AI agent ROI using containment rates, hallucination benchmarks, cost per resolution and leading observability and evaluation tools.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Measure AI Agent ROI: Metrics, Benchmarks &amp; Tools\" \/>\n<meta property=\"og:description\" content=\"How to measure AI agent ROI using containment rates, hallucination benchmarks, cost per resolution and leading observability and evaluation tools.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/\" \/>\n<meta property=\"og:site_name\" content=\"Learn About Digital Transformation &amp; Development | DianApps Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-20T18:33:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-22T18:34:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1536\" \/>\n\t<meta property=\"og:image:height\" content=\"864\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Vikash Soni\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Vikash Soni\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"20 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Measure AI Agent ROI: Metrics, Benchmarks & Tools","description":"How to measure AI agent ROI using containment rates, hallucination benchmarks, cost per resolution and leading observability and evaluation tools.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/","og_locale":"en_US","og_type":"article","og_title":"How to Measure AI Agent ROI: Metrics, Benchmarks & Tools","og_description":"How to measure AI agent ROI using containment rates, hallucination benchmarks, cost per resolution and leading observability and evaluation tools.","og_url":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/","og_site_name":"Learn About Digital Transformation &amp; Development | DianApps Blog","article_published_time":"2026-11-20T18:33:26+00:00","article_modified_time":"2026-11-22T18:34:32+00:00","og_image":[{"width":1536,"height":864,"url":"https:\/\/dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp","type":"image\/webp"}],"author":"Vikash Soni","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Vikash Soni","Est. reading time":"20 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#article","isPartOf":{"@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/"},"author":{"name":"Vikash Soni","@id":"https:\/\/dianapps.com\/blog\/#\/schema\/person\/0126fafc83e42bece2acbfe92f7d0f4f"},"headline":"How to Measure AI Agent ROI: Containment Rate, Hallucination Benchmarks &#038; Evaluation Tools","datePublished":"2026-11-20T18:33:26+00:00","dateModified":"2026-11-22T18:34:32+00:00","mainEntityOfPage":{"@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/"},"wordCount":4036,"commentCount":0,"image":{"@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#primaryimage"},"thumbnailUrl":"https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp","keywords":["How to Measure AI Agent ROI"],"articleSection":["Artificial Intelligence"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/","url":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/","name":"How to Measure AI Agent ROI: Metrics, Benchmarks & Tools","isPartOf":{"@id":"https:\/\/dianapps.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#primaryimage"},"image":{"@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#primaryimage"},"thumbnailUrl":"https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp","datePublished":"2026-11-20T18:33:26+00:00","dateModified":"2026-11-22T18:34:32+00:00","author":{"@id":"https:\/\/dianapps.com\/blog\/#\/schema\/person\/0126fafc83e42bece2acbfe92f7d0f4f"},"description":"How to measure AI agent ROI using containment rates, hallucination benchmarks, cost per resolution and leading observability and evaluation tools.","breadcrumb":{"@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#primaryimage","url":"https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp","contentUrl":"https:\/\/prod-wp.dianapps.com\/blog\/wp-content\/uploads\/2026\/11\/measure-ai-agent-roi.webp","width":1536,"height":864,"caption":"measure ai agent roi"},{"@type":"BreadcrumbList","@id":"https:\/\/dianapps.com\/blog\/how-to-measure-ai-agent-roi\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dianapps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How to Measure AI Agent ROI: Containment Rate, Hallucination Benchmarks &#038; Evaluation Tools"}]},{"@type":"WebSite","@id":"https:\/\/dianapps.com\/blog\/#website","url":"https:\/\/dianapps.com\/blog\/","name":"Learn About Digital Transformation &amp; Development | DianApps Blog","description":"Dianapps","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dianapps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/dianapps.com\/blog\/#\/schema\/person\/0126fafc83e42bece2acbfe92f7d0f4f","name":"Vikash Soni","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/dianapps.com\/blog\/wp-content\/uploads\/2026\/08\/vikash-soni-400-96x96.jpg","url":"https:\/\/dianapps.com\/blog\/wp-content\/uploads\/2026\/08\/vikash-soni-400-96x96.jpg","contentUrl":"https:\/\/dianapps.com\/blog\/wp-content\/uploads\/2026\/08\/vikash-soni-400-96x96.jpg","caption":"Vikash Soni"},"description":"Vikash Soni (CTO &amp; Co-founder, DianApps) leads engineering at DianApps, where he has spent over 10 years building AI and machine learning systems, alongside earlier work in AR\/VR and blockchain. He has delivered 250+ AI and machine learning systems across various industries, e.g. healthcare, fintech, and retail. His work centers on the parts of AI development that decide whether a project ships: retrieval architecture, evaluation design, and the data preparation most teams underestimate. He advises founders and enterprise technology leaders on where AI genuinely fits a problem, and where a simpler system would serve better.","sameAs":["https:\/\/dianapps.com\/","https:\/\/www.instagram.com\/_ai_4everyone","https:\/\/www.linkedin.com\/in\/reachvikashsoni\/"],"url":"https:\/\/prod-wp.dianapps.com\/blog\/author\/infodianapps-com\/"}]}},"_links":{"self":[{"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/posts\/24491","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/comments?post=24491"}],"version-history":[{"count":3,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/posts\/24491\/revisions"}],"predecessor-version":[{"id":24500,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/posts\/24491\/revisions\/24500"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/media\/24494"}],"wp:attachment":[{"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/media?parent=24491"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/categories?post=24491"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prod-wp.dianapps.com\/blog\/wp-json\/wp\/v2\/tags?post=24491"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}