I still remember the smell of burning plastic. We were in the early days of MovieLaLa, my second startup, and we were trying to build a recommendation engine that could predict what movie you'd want to watch before you even knew it yourself. This was long before every company had "AI" in their pitch deck. We were a small team, crammed into a tiny office, with a server rack that sounded like a jet engine taking off.
One night, I was working late, pushing a new model to production. Suddenly, the lights flickered and a plume of smoke started rising from our server rack. One of our GPUs had just fried itself. That was the moment I realized a hard truth: AI is not just about fancy algorithms and clever code. It's about raw, brutal, and often unforgiving hardware.
For the last 13 years, I’ve been in the trenches of Silicon Valley, building and investing in AI companies. I’ve seen two of my own companies get acquired, and I’ve written checks to over 200 startups, including some of the biggest names in AI today like Anthropic, OpenAI, and Scale AI. And through it all, the one constant has been the critical importance of the hardware that powers everything.
Everyone loves to talk about the magic of large language models, but nobody wants to talk about the ugly reality of the infrastructure that makes them possible. I’m here to give you the unfiltered truth.
The GPU Chokehold
Let’s start with the elephant in the room: NVIDIA. To say they have a monopoly on AI hardware is an understatement. They have a chokehold. When the AI boom really started to take off, getting your hands on high-end NVIDIA GPUs was like trying to find a unicorn. Prices were insane, and the waiting lists were months long. I remember calling in every favor I had, just to get a few cards for a portfolio company that was on the verge of a breakthrough.
This GPU shortage was a wake-up call for the entire industry. It showed us how vulnerable we were, relying on a single supplier for the most critical component of our infrastructure. It’s a dangerous position to be in. We saw startups with brilliant ideas fail because they couldn’t get the hardware they needed to train their models. It was a bloodbath.
And it’s not just about availability. The cost of these GPUs is astronomical. A single H100 card can set you back tens of thousands of dollars. When you need thousands of them to train a foundational model, the numbers get staggering. We’re talking about hundreds of millions of dollars just for the hardware. This creates a huge barrier to entry, making it incredibly difficult for new players to compete.
The Data Center Dilemma
So you’ve managed to get your hands on a mountain of GPUs. Now what? You need a place to put them. And that brings us to the next major challenge: AI data centers.
These aren’t your traditional data centers. AI workloads are incredibly power-hungry and generate a massive amount of heat. You need specialized cooling systems, high-density power distribution, and a whole lot of space. Building one of these facilities is a multi-billion dollar undertaking.
I’ve walked through some of the most advanced AI data centers in the world, and they are truly mind-boggling. The scale of the operation is hard to comprehend. Rows upon rows of servers, humming with a collective power that could light up a small city. The engineering that goes into keeping these facilities running is nothing short of heroic.
But here’s the dirty secret: they are incredibly inefficient. A huge portion of the energy consumed by these data centers is used for cooling, not for computation. We are literally burning energy to fight the heat generated by our own hardware. It’s a vicious cycle, and it’s not sustainable in the long run.
We need a new approach to data center design. We need to think about how we can build more energy-efficient systems, from the chip level all the way up to the cooling infrastructure. This is a massive opportunity for innovation, and I’m starting to see some interesting ideas emerge, like liquid cooling and more distributed data center architectures.
The Custom Silicon Dream
Given the challenges with NVIDIA’s dominance and the limitations of current data center technology, it’s no surprise that many companies are starting to explore the idea of building their own custom silicon.
At RemoteTeam, my first startup, we hit a wall with off-the-shelf hardware. We were trying to do real-time analysis of team collaboration patterns, and the latency of existing solutions was just too high. We made the crazy decision to design our own chip. It was a massive undertaking, and we made a ton of mistakes along the way. But in the end, it was the right call. Our custom silicon gave us a huge competitive advantage and was a key factor in our acquisition by Gusto.
Building your own chip is not for the faint of heart. It’s a long, expensive, and incredibly difficult process. You need a team of world-class engineers, deep pockets, and a whole lot of patience. But if you can pull it off, the rewards can be immense.
You get to design a chip that is perfectly tailored to your specific workload. This can lead to massive improvements in performance, power efficiency, and cost. You also get to control your own destiny, freeing yourself from the whims of a single supplier.
We’re seeing this play out across the industry. Google has its TPUs, Amazon has its Inferentia and Trainium chips, and even Microsoft is getting into the game with its own custom silicon. This is a trend that I believe will only accelerate in the coming years.
A Glimpse into the Quantum Future
Looking ahead, there’s one technology that has the potential to change everything: quantum computing.
Now, I know what you’re thinking. Quantum computing has been “just around the corner” for decades. But I believe we are finally starting to see real progress. The machines are still small and noisy, but they are getting better every year.
I’ve had the opportunity to meet with some of the leading researchers in the field, and the work they are doing is truly groundbreaking. They are tackling problems that are simply intractable for even the most powerful classical computers.
Quantum computing is not going to replace classical computing overnight. But for certain types of problems, like drug discovery, materials science, and financial modeling, it has the potential to be a complete game-changer.
I’m not a quantum physicist, but I’m a big believer in the power of this technology. I’ve made a few early-stage investments in quantum computing startups, and I’m excited to see what they can achieve. It’s a long-term bet, but it’s one that I believe will pay off in a big way.
My Advice to Founders
So, what does all of this mean for you, the founder who is trying to build the next great AI company? Here are a few pieces of advice based on my own experience:
- Don’t underestimate the importance of hardware. It’s not as sexy as building a new model, but it’s just as important. Make sure you have a clear plan for how you are going to get the hardware you need, and don’t be afraid to think outside the box.
- Be scrappy. In the early days, you’re not going to have a massive budget for hardware. You need to be creative and find ways to get the most out of what you have. This might mean using older hardware, optimizing your code, or even building your own custom solutions.
- Think about the long term. The AI landscape is constantly changing. The hardware that is state-of-the-art today will be obsolete in a few years. You need to be constantly thinking about what’s next and how you can position your company for success in the long run.
Building an AI company is a marathon, not a sprint. It’s a tough, challenging, and often frustrating journey. But it’s also one of the most exciting and rewarding things you can do. The hardware is a critical piece of the puzzle, and if you can get it right, you’ll be well on your way to building something truly special.
Frequently Asked Questions
Can these results be replicated?
The specific numbers will vary, but the underlying patterns and principles are transferable. The key is understanding the context behind the results, not just copying the tactics. Every company has unique constraints that shape what works.
What would you do differently looking back?
I'd move faster on the things that were working and cut the things that weren't sooner. Most founders, myself included, hold onto failing strategies too long because of sunk cost. Speed of learning is everything.
What was the biggest challenge in this case?
Almost always, the biggest challenge is people and alignment, not technology or strategy. Getting the right team focused on the right problem is harder than any technical challenge I've encountered.
How long did it take to see results?
Most meaningful business results take 3-6 months to materialize. Anyone promising overnight success is selling something. The companies in my portfolio that grew fastest were the ones that stayed patient and consistent.