You’ve said that speech recognition is the foundation everything else in conversational AI sits on. For an executive evaluating AI voice agents, what should they be asking about that layer before they look at anything else?
Most voice agents run on a pipeline of speech to text (STT), a large language model (LLM), and text to speech (TTS). STT captures what the caller actually said, and everything downstream depends on it. If the transcript is wrong, the LLM works from bad input and the agent gives a wrong answer. For executives building voice agents, I’d look at six things in the STT layer.
First, accuracy. Word error rate (WER) tells you how well the model captures speech, but don’t get carried away by the headline numbers model providers publish. An STT system that reports 95% accuracy overall can still be failing 30% of callers with strong accents, or getting numbers wrong. Ask how it holds up on the hard cases: several speakers on one line, accents and dialects, background noise or music. Companies should build a bench of 30-50 tasks that reflect their actual use cases and evaluate every new model against it.
Second, latency, meaning the time to first response and the speed of the streaming output. In a cascaded pipeline, latency adds up fast: roughly 150 to 200 milliseconds for STT, 300 to 600 for the model, and another 150 to 200 for TTS. A real time conversation needs the whole loop under about 800-900 milliseconds, or it starts to feel like a walkie talkie.
Third, cost. STT is typically priced per minute of audio, and for a contact center processing billions of minutes a year, a small difference in per minute price compounds into a very large number. Real-time audio is also priced much higher than recorded files, so enterprises can save by sorting out which use cases truly need real-time processing and which can run offline.
Fourth, customization. Most models can’t accurately pick up company or product names, so the STT model needs to be tunable on company specific vocabulary. Building that list is iterative. You have to separate words that fail because they’re unique to the company from words that fail because of poor enunciation. The enterprises I’ve seen do this well keep a corpus of no more than 1,000 words that they update by analyzing where the voice model fails.
Fifth, redaction. Because the transcript goes to an LLM, the STT layer should automatically remove card numbers, government identifiers, and addresses before anything reaches downstream AI systems. One enterprise I worked with built an in-house generative AI model on top of the voice model’s redaction to strip sensitive data before it reached the LLM or got written to log databases.
Sixth, hallucinations. Speech models are prone to them, which is a real problem in healthcare, financial services, and public safety. Every model generated response should run through hallucination checks and toxicity filters before it reaches the caller or another AI system.
You have worked on expanding capabilities of voice models including language support. What did that expansion involve beyond training more models, and why did it matter so much to the large contact-center platforms and the platforms that consolidated onto the service?
Training the models on new languages was only part of it. I also had to manage availability, pricing, and validation in each region.
First, the models had to be available in the cloud regions where major customers operate, so they could use them while complying with local laws. That matters most in the EU, where the first prohibitions under the EU AI Act took effect in February 2025. Local availability also gives customers the low latency they need for use cases such as autonomous voice agents. Voice models run on GPUs, which are in short supply, so I had to make sure each region had enough capacity or enable cross-region inferencing.
Second, pricing had to be competitive in each region while holding margins. A single global price would have made us uncompetitive in markets such as APAC because of currency differences, and we were up against leading AI labs releasing new models quickly, plus free open source models from Asian labs that only cost compute to run. I set regional prices based on marginal cost of operations, competitor pricing, and target margins.
Third, I ran limited private betas with customers in each region before general availability, to find gaps and fine tune. Number formatting, for example, varies a lot between languages such as Mandarin and Japanese, and real-world testing gave me the feedback to retrain with more data or adjust the algorithms.
This mattered to contact center platforms because speech capabilities such as transcription have become commoditized. Platforms that used to build them in-house now buy them and put their own engineering into things like voice agents. Consolidating basic speech processing onto one vendor gives them pricing leverage, since they can use their global minute volume to negotiate a single low price. It also lets them scale globally without a different vendor in each region, secure inference capacity everywhere they operate, and give their global customers a consistent experience.
Data residency rules in the EU, the Gulf and parts of Asia Pacific kept a whole segment of enterprises from using cloud speech AI at all. How did regional expansion change who could adopt the product, and what does that tell leaders about where AI adoption actually gets blocked?
Many governments don’t want their citizens’ voice data leaving their borders, so enterprises need AI models hosted and running inference in-country. Product leaders who want to sell into these regions need to address two things.
First, training and data security. Enterprises in APAC and the EU are careful about how models are trained and don’t want their proprietary data used for training. We addressed this by documenting, as part of the product documentation, that customer data isn’t used. Voice data also has to be secured in transit, with end-to-end encryption from legacy contact center platforms to the cloud.
Second, inferencing and storage. Voice processing should happen locally, through endpoints in each of the cloud provider’s global regions, so nothing crosses international borders. For storage, we gave customers clear architecture documentation showing that no data is stored internally, and audio stays in the selected region. Output from batch processing goes to the customer’s own cloud storage account, and real-time audio is processed in memory without being stored.
Once these changes were in place, we could sell to customers in regulated industries such as healthcare and finance, and in the public sector.
In your previous roles, you have moved from supporting a single input modality for AI models to a more multimodal approach. Walk us through the decision to bet on a modality the product wasn’t designed for, and what you’d tell a product leader facing a similar call.
AI models used to be purpose-built for each modality, whether voice, documents, or video. With generative AI, many models from the leading labs are multimodal by default, and a siloed approach no longer serves customers well. I expanded into audio because voice is the most natural way people communicate, and it opens up use cases such as real-time translation in 911 call centers, better contact center service, and productivity tools for doctors. Models also do better with more context. If an investment advisor meets a client over video with an AI live assistant, that assistant is more useful when it can analyze the documents exchanged, the conversation, and the client’s facial expressions together.
So I built support for customers to send any type of data, whether audio, video, or text, with the system detecting the modality and routing it to the right model. Customers could also specify the modality upfront and describe in natural language what they wanted extracted. I helped architect it so decisions like language detection and scene detection happen automatically, without the user having to manage them.
For a product leader facing a similar call, I’d suggest looking at three things. First, the customer’s problems. Don’t build a complex product with high costs and latency when most use cases are simple, and make sure there’s enough demand that you aren’t chasing a market that’s hard to serve. Second, the technical and model architecture, including training, data, hosting, and compute needs. A key early decision is whether to build one consolidated model that handles all capabilities or a separate model for each. And third, pricing and go-to-market. A multimodal model needs far more training data and compute than a voice-only model, so it costs much more to build and run, and the pricing has to reflect that customers are getting more value.
You led the shift from a sprawl of specialized speech models to a small consolidated set, which changed both cost and accuracy. What did that consolidation reveal about the economics of running AI as a platform business?
Consolidation was critical to accelerating growth for the products I supported. Moving from many models to a few lowered inference costs and improved margins, and it made it easier to build, test, and respond to competitors.
The lesson was that you need a clear reason for consolidating before you work out how to do it. In my case, I was moving models from CPUs to consolidated models on GPUs, which let me rearchitect them as larger models that lowered cost and made new functionality easier to add.
GPUs are built for large models because they process voice data in parallel. An ensemble of many small models has two problems: the audio has to pass through each model in sequence, and each model needs its own dedicated compute. Running that ensemble on GPUs gets expensive, while CPUs are cheaper but can’t support larger models and slow down processing. So the benefit only comes from rearchitecting. I retrained the models on new data, increased model size significantly, enabled parallel processing, and used the latest advances in generative AI architecture.
Fewer models also made testing far simpler. If you have a separate model for each major language, every update means testing 100+ language models, and once you add capability models such as speaker identification, that becomes hundreds of combinations. With a small set of models, you can check new functionality quickly. Data and training costs drop too, since you can use mixed data sets that cover multiple languages, speakers, and number formats, and train on a smaller number of large GPUs.
In the model development business, model sprawl is a hidden tax. Every extra model carries its own compute, training, testing, and maintenance. That said, consolidation doesn’t mean always reaching for one big model. Over provisioning a large model on simple tasks burns margin, and under provisioning on complex ones burns trust, so the real skill is matching model size to the job. Done well, consolidation is one of the few moves where accuracy and unit economics improve at the same time.
From the inside, where does executive thinking about AI in customer experience most often go wrong, and what’s the corrective?
Customer experience has become one of the hero use cases for generative AI, with enterprises using it to reduce support costs, improve satisfaction, and drive revenue. Because of the hype, executives often expect quick results, and then run into problems that delay large scale deployment. Across the enterprises I’ve worked with, three issues come up most.
The first is going straight to autonomous agents. Executives assume an off-the-shelf agent can go to production without grounding in company information, rigorous edge case testing, or pilots with real customer data. In customer support, AI should be rolled out in stages of increasing autonomy. One financial firm started by analyzing historical call recordings to learn what its best support staff did and used that to train underperformers. Then it deployed real-time assistants for human agents, and finally voice agents for low complexity calls, leaving humans to handle the complex ones.
The second is scaling before the architecture is stable. That means choosing the right models, cloud providers, and security and privacy controls, and mapping each step of the voice and data flow, from audio securely reaching the AI system, to how it holds up at high call volume, to how it responds to the customer. Model choice drives compute cost and latency. Smaller models (under 5B parameters) suit frequent, simple tasks like rescheduling appointments, while larger models (over 10B) suit complex reasoning across workflows. The system should also be tested with real customers to check that replies sound natural and empathetic.
A communications provider got this right by first routing its contact center traffic securely to a cloud provider through a virtual private cloud setup. It then tested 10 different models, encrypted everything, and worked with the cloud vendor on failover across regions to handle outages.
The third is launching without a clear business case or success metrics, which leaves executives unable to justify the investment. Productivity gains are a common choice but take time to show up. Customer experience metrics such as net promoter score or customer satisfaction are more tangible, as are contact center metrics like first call resolution (FCR) and average handle time (AHT). One healthcare enterprise set a single north star of improving CSAT by 20% in six months, and once it hit that, moved on to operational metrics.
