IBM says only a quarter of AI initiatives deliver the expected return. The fix starts before the build: decide what the agent is for and how to measure it.
The number that should worry you
IBM published a short guide to agentic AI for technology leaders this year, and one figure in it deserves more attention than the rest. In the IBM Institute for Business Value research it draws on, only 25% of AI initiatives had delivered the return their sponsors expected, and only 16% had been scaled across the enterprise. Three quarters of executives said they were building or piloting autonomous agents. Most of those pilots will not pay back.
That gap is not a technology problem. The models are good enough. The agent frameworks in Copilot Studio, Azure AI Foundry and the Power Platform are good enough. The gap is that most organisations start building an agent before they have decided what it is for and how they will know whether it worked. This post is about fixing that, and it is the first of three drawing on IBM's guide. The next two cover why a pile of agents delivers less than one good workflow and why your people decide whether any of it pays back.
ROI is a decision, not a discovery
The most useful idea in IBM's guide is also the simplest. Before anything is built, define what return means for this agent. Not in general. For this one.
IBM groups the ways organisations measure the return from agents into four. Productivity gains, which is time given back to people. Revenue growth, where the agent changes what customers buy or how quickly. Employee and customer satisfaction, which is real but hard to put a number on. And cost reduction, where the agent removes work that was being paid for. Their research found 67% of leaders name cost reduction as the top benefit and that the AI-first companies in their sample saw an 18% return against a much lower average.
The four are not interchangeable. An agent that drafts supplier correspondence saves time. It does not, on its own, reduce cost unless someone decides what to do with the time. An agent that answers customer queries overnight may lift satisfaction and still cost more to run than the queries were worth. If you let the agent's sponsor pick a measure after the fact, they will pick the one that looks best. Pick it first.
How we do it with clients
When a client asks us about an agent, the first session is not about the agent. It is about the decision or the process the agent sits inside. Three questions do most of the work.
What happens today, and what does it cost? Not an estimate from the person who wants the agent. A real baseline: how many of these cases a month, how long each takes, who does them, what it costs when one goes wrong. If nobody can answer that, the baseline is the first piece of work, and it is cheap. Our AI readiness checker is a reasonable place to start if you want a rough view before we talk.
Which one measure will we stand on? One. Hours returned per month, cost per case, days from enquiry to quote, tickets resolved without a person. The measure should be something finance will accept and something the business already reports, so the before and after are comparable. If the measure needs a new dashboard to exist, build the dashboard before the agent. Our post on what an AI analytics project costs covers what that budget usually looks like.
What would make us switch it off? Every agent should have a kill condition agreed before go-live. If after eight weeks the measure has not moved by the amount we agreed, we stop and ask why. That sounds harsh. It is the opposite. It gives the sponsor permission to try something without betting their credibility on it.
Why the mid-market has an advantage here
IBM's guide is written for enterprises with a portfolio of agents and an orchestration layer to manage them. A UK or Irish business with two hundred people does not have that problem yet, and that is an advantage. You can pick one process, baseline it properly, build one agent against one number, and know within a quarter whether it paid back. Enterprises are trying to do that forty times at once.
What you do need is the same thing they need: data the agent can trust. IBM lists unstructured and scattered data as the first barrier to return, and it matches what we see. An agent reasoning over a Business Central instance, a CRM and a shared mailbox will give confident answers built on three different versions of the customer. We wrote about that failure mode in why 85% of AI projects fail before the model does anything wrong, and it is why most of our agent work starts with the data foundations rather than the agent.
The short version
Decide the measure before the build. Baseline it. Agree the number that counts as success and the number that means stop. Then build the smallest agent that could move it. The companies in IBM's 25% did not have better models. They knew what they were measuring.
Related reading
Why 85% of AI projects fail before the model does anything wrong
What an AI analytics project costs, and how to budget for one
An AI agent needs decision rights, not just intelligence
Start with the decision, then choose the AI framework
Simon Devine
Managing Director
Part of the Hopton Analytics team, delivering governed analytics programmes for UK mid-market organisations.
