Most marketers say AI is improving things, but few of them can prove it.
According to a new Metric Sherpa study produced with Sprinklr, 92% of organizations with customer-facing AI in regular use said it met or exceeded expectations. Only 27% said they could show proof of that.
Similarly, 67% of the people who said AI made work faster also said it reduced costs. However, most of those had no baseline to base it on: Only 32% of that group said they tracked costs before launch.
Measurement needs to start before the AI does
This is likely because most teams know what they want AI to improve, but few define the metric, record the starting point, or set a clear threshold for success. So when the results arrive, even if there’s a success story, there’s no way to prove it.
The study found that while 99% could name an intended outcome for their AI use, only 26% defined success metrics, and just 10% also recorded a baseline.
Without the last two, it’s impossible to prove AI’s ROI.
If the goal is lower acquisition cost, the team needs a cost baseline before the AI starts making media or campaign decisions. If the goal is better customer service, containment alone won’t tell you much. The team needs data on resolution, repeat contacts, escalations, and handoffs.
These measurements must be built into the workflow before launch.
Don’t let the platform define success
If they aren’t, teams are forced to rely on whatever numbers the platform gives them.
A paid media platform can optimize conversion. A service platform can optimize containment. A content system can optimize engagement. All three can show improvement inside their own dashboards while the business outcome stays flat.
Eighty-five percent of those surveyed reported improvement in at least one metric category they hadn’t tracked before launch. The researchers couldn’t tell whether those teams added measurement later or relied on another source, but the disconnect is hard to ignore.
Marketing ops needs to connect platform activity to CRM, finance, customer, or experiment data. Otherwise, the tool doing the work also gets to decide whether the work succeeded.
Someone still has to own the result
Even with the right metrics in place, somebody has to be responsible for the outcome.
Only 17% said their organization names an owner for AI business outcomes. It was the least common of the eight practices measured in the study.
That role is different from owning the platform or the workflow. The platform owner makes sure the tool works. The workflow owner makes sure the process runs. The outcome owner answers for whether the AI produced the desired result.
Organizations using a named outcome owner were able to quantify results 46% of the time, versus 23% of those without one. They also used controlled tests and financial reporting more often. The study doesn’t establish that ownership caused better measurement, but the association is substantial.
Better measurement can make AI look worse
Teams that measure more carefully may also find more things going wrong.
Organizations using three or more of the study’s measurement practices reported more AI-to-employee handoff problems, adoption issues, and data-quality friction than those using zero or one. It isn’t clear if the increased measurement exposed more problems or if more complicated deployments had more problems to find. Either way, the operational lesson is useful.
10X your SEO with Semrush for Enterprise.
The world’s most powerful SEO platform, purpose-built for Enterprise.
Request demo
A mature AI program can look messier because somebody is finally checking what happens after the automation fires. That means reviews need to cover more than output and speed. Look at where handoffs break, where employees ignore recommendations, where customers repeat themselves, and where a metric moves in the wrong direction.
Decide what would make you stop
Only 28% said their organization uses agreed criteria to decide whether to scale, change, or stop an AI application. Just 37% review AI results on a regular schedule.
That leaves plenty of AI running without a clear rule for what happens if the direct metric improves but the business result doesn’t.
For one application, define the outcome, baseline, success threshold, and comparison before the next review. Then write down what result would justify scaling it, what would require a change, and what would cause the team to stop.
AI can make a martech stack much more capable. It can also make weak measurement much harder to unwind once the system is running.
The full report can be downloaded here. (No registration required)
