To get value from AI, fix the leaky pipe
The numbers are stark: 90% of companies are investing in AI for customer service, yet less than a quarter have scaled beyond a pilot, and only 4% are creating “substantial value” with AI (source).
On top of everything that customer service leaders have to deal with, they now need to manage the cognitive dissonance of seeing incredible, beautiful demos of advanced AI chatbots and voicebots, but finding it extremely hard to deliver the same results at scale in their own operations. The problem is not the technology, the problem is how to deploy that technology in the messy, complicated real world of customers and customer service.
It’s a very leaky pipe, and a distinct effort will be needed to fix it.
How value leaks from concept to scale - and how to fix it
There are many ways that value leaks when deploying AI in customer service, but most fit into one of these three categories.
1. Concept to pilot: the pilot does not demonstrate the envisaged KPI improvements
Some projects fall at the first hurdle. The shiny demo loses its lustre when it is piloted on real customers or Customer Service Representatives (CSRs). One extreme example: a company found that handling time consistently increased when testing an AI knowledgebase tool. On investigation, they found that calls were taking longer because CSRs were using the new tool, then reverting back to the old tool to double-check the answer.
Here are some key watch-outs at the pilot stage:
Be clear on what can feasibly be piloted and what results can be expected - need to set out in advance what the pilot is actually going to test, and then set up the pilot in the right way. For example: will the sample size be big enough to be able to show a statistically significant change?
When piloting with CSRs: overcommunicate context, and communicate again - it’s brilliant to involve CSRs in pilots, but we need to set up the pilot so that they are comfortable trying something different. For example, It’s not uncommon for CSRs to disregard the pilot instructions because they are worried it will negatively impact their quality score.
Continuously improve, don’t just observe results - the biggest mistake I see with customer service pilots is to treat them like passive experiments: companies set up a pilot and measure whether it passes or fails the success criteria. By contrast, the best pilots are set up as learning experiences, with teams working together to maximise performance week by week.
2. Pilot to scale: the full scale implementation is substantially less effective than what was observed in the pilot
One additional issue to contend with in pilots is the observer effect. A group of CSRs will naturally improve performance during a pilot because they are being managed in a different way, and they know their results are being watched more closely than usual. It’s a strong implicit incentive to perform better - which is great to see, but it isn’t representative of how well the AI tool will work at scale.
Getting results from AI at scale is a different skill to running a pilot. Here are the major watch-outs at this stage:
Line managers are critical, always brief them first - the key people who can help to land change in a contact centre are the direct supervisors of the CSRs. They can become great allies in delivering change. To win them over, we need to give them specialist training, before the general CSR population, so that they can lead from the front as experts
Don’t report on average performance, monitor cohorts of behaviours - segment the CSR population in terms of usage (how often each CSR is using the new tool) and performance (how have their KPIs changed since the tool was launched) and plan relevant interventions for each segment
Don’t stop overcommunicating context - context is even more important when implementing at scale. CSRs need to know why we are implementing this change (in a way that is meaningful to them), what we want them to do differently, and what is the outcome we are expecting
3. Scale to value: when scaled, AI delivers measurable improvements to some KPIs, but this is not reflected in an overall productivity improvement
Even now we are not in the clear. It feels great to look at a chart which shows a consistent improvement in First Contact Resolution or Turnaround Time, but it is another challenge entirely to translate those improvements into higher level productivity improvements, like cost to serve per customer, or revenue generated per employee.
These are the critical things to get right at this stage:
Redesign processes and customer journeys to seamlessly integrate the new AI solution - so that the automated step of the process adds up to a more efficient process overall
Set up a strong interlock between the project team and Workforce Management (WFM) - so that KPI improvements are reflected in the capacity plan
Reserve project delivery resource to make further post-live improvements to the AI solution - to be able to react to on-the-ground feedback about how the AI tool can better support value creation
In summary, what any AI implementation needs is a sustained focus on value. Starting from project inception (start with “what is the problem we are trying to solve” rather than “where can we use this new AI tool?”) through to business adoption (change management needs to continue for 3+ months beyond any major rollout). It’s critical to have someone on point for value realisation throughout the programme, and for that person to have sufficient authority and resource to make the needed interventions.
Recommended news articles
Forbes: Can customer service solve the global consumer confidence crisis?
ZDNet: A million customer conversations with AI agents yielded this surprising lesson
Inc.: Dear bosses, please stop calling it “The year of the customer”
Reuters: Capgemini to buy outsourcing firm WNS for $3.3 billion in AI push
Latest perspectives from BCG
European consumers brace for more uncertainty
Europeans are feeling gloomy, according to a survey of 16,000 consumers in nine countries. More than half, 54%, are pessimistic about the economy of their home country, a 7 percentage point increase since July 2024
Customers are looking for bargains; companies must meet this demand while preserving profit margins
But economic growth will eventually return, so consumer companies and retailers should prepare to address future demand through strong innovation and marketing
We're seeing one of the quickest hype-to-reality cycles ever. The buzz is driving billions in investment, mostly out of FOMO. But the fundamentals of business haven't changed. Making AI work at scale takes more than plugging in a new tool. It means cleaning up data, breaking down silos, updating knowledge and processes, and, most importantly, managing human change. These are the same challenges companies have always faced with new tech. Wild times, Nick.
What I see is that the reality around AI in customer service is sobering. Only a small number have managed to get beyond limited pilots and are realising value. This disconnect is the core issue: it isn’t the technology, but the ability to successfully embed it in complex, real-world service environments.
Your "leaky pipe" analogy is spot on as value is dissipating at each stage of AI implementation. Each step exposes gaps, often not technical but organisational and behavioural. We underestimate the effort needed to engage stakeholders, redesign processes, and drive adoption across the broader workforce.
We need persistent, expert-led value realisation: leaders with the authority and resources to bridge the gaps between proof of concept and widespread value. Without rethinking processes and ensuring change management continues well beyond rollout, even the best AI tools can fall short of expectations. I'm an optimist in the benefits of AI, a pessimist in our ability to use it effectively, a strategist in how it can change our future. So here's a question for you: How do you see the role of leadership evolving in an era where AI value realisation depends less on technical prowess and more on cross-functional, human-centred transformation? And what will it take to ensure that “value realisation” isn't just a project phase, but a core organisational competency for the age of AI? I'm interested to see if your thinking aligns with mine!
Michael Cooper (uncx.substack.com)