I’ve been in the trenches with AI voice for years now. Not just as an investor in over 200 companies, including some of the biggest names in AI like Anthropic and OpenAI, but as a founder who has built and sold companies. I’ve seen the hype, and I’ve seen the reality. And let me tell you, the reality is a lot messier and more interesting than the press releases.
I’m not here to give you the same old “AI is the future” speech. You’ve heard that a million times. I’m here to share what I’ve learned from the front lines – the painful, expensive, and sometimes hilarious lessons that you only get from building and investing in this stuff for real.
1. The “Human-Like” Trap
Everyone is chasing “human-like” voice AI. And I get it. The idea of a perfect digital replica of a human voice is seductive. But here’s the thing: most of the time, you don’t actually need it. In fact, sometimes it can even be a disadvantage.
I remember one of my early investments, a company I’ll call “Echo Labs.” They were brilliant, some of the smartest PhDs I’ve ever met. They spent millions and two years of runway trying to create a voice that was indistinguishable from a human. They got incredibly close, to the point where even their own mothers couldn’t tell the difference in a blind test. But when they finally launched their customer service bot, the reaction was… not what they expected. Customers were creeped out. They felt like they were being tricked. The uncanny valley is a real thing, and Echo Labs fell right into it. Their super-advanced, “human-like” AI was so good it was bad.
What they learned, and what I’ve seen time and time again, is that for most applications, “good enough” is all you need. A clear, pleasant, and easily understandable voice is far more important than one that can pass the Turing Test. Don’t get so caught up in the ‘human-like’ trap that you forget what the user actually needs. Authenticity is more important than mimicry.
2. Your Data is Your Moat, Not Your Model
I’ve seen so many startups obsess over having the latest and greatest AI model. They’ll spend a fortune on R&D, trying to eke out a few more percentage points of accuracy. But here’s a secret: the model is not your defensible advantage. Everyone has access to the same open-source models from Google, Meta, and OpenAI. Your real moat is your data.
Think about it. If you have a unique, proprietary dataset of, say, 100,000 hours of customer service calls in the insurance industry, you can train a model that will outperform any generic model, no matter how advanced it is. That’s your unfair advantage. That’s what investors like me are looking for. I’m not investing in your algorithm; I’m investing in your data acquisition strategy.
I once passed on a company that had a brilliant team of AI researchers from Stanford. They had developed a new architecture for speech recognition that was theoretically groundbreaking. But they had no data. They were trying to build a skyscraper on a foundation of sand. Meanwhile, I invested in a company with a much less sophisticated model, but they had an exclusive partnership with a major call center network. They had the data. Guess which one is now a unicorn with a multi-billion dollar valuation? It’s not the one with the fancy algorithm.
3. The Long Tail of Accents is a Killer
When you’re building a voice product for a global audience, you can’t just train it on standard American English and call it a day. The world is a beautiful, messy, and diverse place, and that means a lot of different accents. And let me tell you, the long tail of accents is a killer for speech recognition models.
We had a portfolio company that was building a voice-controlled assistant for cars. It worked great in our demos in Silicon Valley. But when they launched it in the UK, it was a complete disaster. It couldn’t understand a word of what people were saying, from the poshest London accent to the thickest Glaswegian brogue. The same thing happened when they launched in Australia, and then again in India. Each new market was a new fire drill, a new scramble to collect data and retrain their models.
They ended up having to spend millions of dollars and years of work building out a dedicated data collection team for each new region. It was a painful and expensive lesson. So if you’re building a voice product, don’t underestimate the challenge of accents. It’s not a bug, it’s a feature of the real world. You need a strategy for the long tail from day one.
4. Latency is the Enemy of a Good User Experience
You can have the most accurate, human-like voice AI in the world, but if it takes two seconds to respond, it’s useless. Latency is the silent killer of user experience in voice applications. People expect conversations to be in real-time. Any noticeable delay, and the illusion is shattered. The conversation feels stilted and unnatural.
I’ve seen this happen so many times. A team will be so focused on accuracy that they’ll build a model that’s too big and slow to run in real-time. They’ll have a great demo in a controlled environment, but when they try to put it into a real product, it’s a clunky and frustrating experience.
One of my companies, which was building an AI sales agent, learned this the hard way. Their initial model was incredibly smart. It could understand complex questions and give detailed, nuanced answers. But it took a full three seconds to respond. In a sales call, that’s an eternity. It just killed the conversation. They had to go back to the drawing board and build a much smaller, faster model, even if it meant sacrificing some accuracy. The trade-off was worth it. A slightly less accurate answer that comes in 300 milliseconds is infinitely better than a perfect answer that takes 3 seconds.
5. The Unsexy Plumbing Matters More Than You Think
Everyone wants to work on the cool stuff – the AI models, the new features. But in the world of AI voice, the unsexy plumbing matters more than you think. I’m talking about things like data pipelines, infrastructure, and monitoring. This is the stuff that will make or break your product.
I’ve seen so many promising voice startups fail because they didn’t invest in their infrastructure. They had a great model, but their data pipeline was a mess. They were using a patchwork of scripts and manual processes to move data around. They couldn’t retrain their model quickly, so they couldn’t adapt to new data. Or their infrastructure was unreliable, so their service was always going down. It doesn’t matter how great your model is if your customers can’t access it.
It’s not glamorous work, but it’s essential. If you’re building a voice product, make sure you have a solid foundation. Invest in your plumbing. Use tools like Kubernetes for orchestration, Kafka for data streaming, and Prometheus for monitoring. It will pay off in the long run. A solid infrastructure is what allows you to iterate quickly and build a reliable product.
6. Voice is Not a Monolith
People talk about “voice” as if it’s one thing. But it’s not. There are so many different aspects to voice AI – speech-to-text, text-to-speech, voice cloning, speaker identification, sentiment analysis, and more. And each one is a deep and complex field in its own right.
I’ve seen a lot of startups try to do everything at once. They want to build the best speech recognition, the best text-to-speech, and the best voice cloning, all at the same time. And they almost always fail. It’s just too much to take on. You can’t be the best at everything.
The successful voice startups I’ve seen are the ones that focus on one thing and do it really well. They pick one part of the voice stack and they become the best in the world at it. Look at companies like Descript for editing or ElevenLabs for TTS. They didn’t try to build an entire voice platform from day one. They started with a single, focused product and they nailed it. So if you’re starting a voice company, don’t try to boil the ocean. Pick your niche and own it.
7. The Ethical Tightrope is Real
AI voice is a powerful technology, and with great power comes great responsibility. The ethical implications of what we’re building are enormous. We’re talking about the potential for mass-scale disinformation, fraud, and manipulation. It’s scary stuff, and it’s not some far-off dystopian future; it’s happening right now.
I’ve had to have some tough conversations with founders about this. I’ve had to push them to think about the potential for misuse of their technology. I’ve had to ask them what safeguards they’re putting in place to prevent it. Are they watermarking their audio? Are they implementing know-your-customer (KYC) protocols? Are they prepared to deal with the fallout when someone uses their technology to impersonate a CEO and steal millions of dollars?
It’s not an easy conversation to have. But it’s a necessary one. We can’t just be blindly optimistic about this technology. We have to be clear-eyed about the risks. And we have to be proactive about mitigating them. As builders and investors, we have a moral obligation to do so.
8. We’re Still in the First Inning
For all the hype and all the progress, we’re still in the very early days of AI voice. The technology is still immature. The use cases are still being discovered. The market is still taking shape.
I know it can feel like you’re late to the party. It can feel like all the big problems have already been solved by the big tech companies. But that’s just not true. There are still so many opportunities to build amazing things in this space. The next generation of voice applications will be more specialized, more personalized, and more deeply integrated into our lives.
So if you’re a founder who’s passionate about voice, don’t be discouraged. Don’t be intimidated by the big players. There’s still plenty of room for you to make your mark. The game is just getting started. I’m putting my money where my mouth is, with over a dozen investments in the space in the last year alone. I’m still as excited about the future of AI voice as I was when I made my first investment in the space. It’s going to be a wild ride, but I can’t wait to see what we build together.
Frequently Asked Questions
Can I implement all of these at once?
I'd strongly recommend against it. Pick the 2-3 items that resonate most with your current situation and focus there. Trying to do everything simultaneously is a recipe for doing nothing well.
How do I know which items apply to my situation?
Start by honestly assessing where your biggest bottleneck is right now. The items that address that specific constraint will give you the highest return on your time and energy.
How were these items selected?
Each item on this list comes from direct experience, either from building my own companies or from patterns I've observed across the 200+ startups I've invested in. I prioritize practical, actionable items over theoretical concepts.
Which item on this list has the highest impact?
It depends on your stage and context, but in my experience, the items near the top of the list tend to have the broadest applicability. That said, sometimes the less obvious items create the biggest breakthroughs for specific situations.