Everyone talks about AI models, but they ignore the brutal reality of the hardware. Here's what it really takes to build and scale AI infrastructure.
I’ve been in the Silicon Valley game for a while now. Two successful exits, over 200 angel investments in companies that are now household names like Anthropic, OpenAI, and Scale AI. I’ve seen a lot. I’ve seen trends come and go, and I’ve seen companies rise and fall. And I’m telling you, the biggest thing that will make or break an AI company in the next five years isn’t the model, it’s the metal.
We’re all obsessed with the latest and greatest large language models. GPT-4, Claude 3, whatever comes next. But we’re forgetting the simple fact that these models are useless without the hardware to run them. And right now, we’re in the middle of a hardware crisis.
The GPU Shortage Nightmare
I remember getting a frantic call at 2 AM. It was the CEO of a startup I’d backed, a brilliant team with a truly revolutionary idea in medical imaging. Their model was outperforming everything on the market, but they were completely stalled. They’d been trying for six months to get a big enough cluster of H100s. Their monthly burn was north of $300,000, and all of it was going down the drain while they waited in line behind the big guys. It was heartbreaking.
This wasn’t a unique story. All over the valley, startups were scrambling for computing power. Begging friends, trimming profits, delaying launches. The GPU shortage was a brutal wake-up call. It showed us just how dependent we’ve become on a single company, NVIDIA, for the hardware that powers the entire AI revolution.
And it’s not just about the shortage. It’s about the cost. The price of high-end GPUs has skyrocketed. For a startup, the cost of building out the necessary infrastructure can be crippling. It’s a huge barrier to entry, and it’s stifling innovation.
The "Build vs. Buy" Dilemma
So what’s the solution? For a growing number of companies, the answer is to build their own custom silicon. To design their own chips, optimized for their specific AI workloads.
This is a huge decision. It’s not for the faint of heart. Building custom silicon is a long, expensive, and risky process. But the potential payoff is enormous.
Look at what Google did with the TPU. I remember when they first started talking about it internally. People thought they were crazy. Why would a software company get into the chip game? It was a massive, expensive gamble. But they saw the writing on the wall. They knew that owning their hardware stack was the only way to win in the long run. Now, their TPUs are the engine behind their entire AI empire, a moat that’s almost impossible for competitors to cross.
But it’s a brutal path. I’ve seen it go wrong more times than it’s gone right. I once advised a company that poured $50 million into a custom chip project. They had a rockstar team, some of the best minds from Apple and Intel. Two years later, the project was a smoldering crater. The chip was a performance dog, the software was a buggy mess, and the market had already moved on. They ran out of money before they could even get to a second revision.
So how do you decide whether to build or buy? It comes down to a few key factors:
- Scale: How big is your model? How much data do you need to process? If you’re operating at a massive scale, the cost savings of custom silicon can be significant.
- Workload: What kind of AI workloads are you running? If you have a very specific and well-defined workload, you can design a chip that is highly optimized for that task.
- Resources: Do you have the team and the funding to pull it off? Building custom silicon is a multi-year, multi-million dollar investment.
For most startups, the answer is still to buy. To use off-the-shelf GPUs from NVIDIA. But as the AI market matures, I think we’re going to see more and more companies making the leap to custom silicon.
A Deep Dive into Custom Silicon
So you’ve decided to build your own chip. What’s next? The first thing you need to understand is that you’re not just building a piece of hardware. You’re building a whole ecosystem. You need to think about the software, the drivers, the compilers. It all has to work together seamlessly.
One of the biggest advantages of custom silicon is performance-per-watt. You can design a chip that is much more power-efficient than a general-purpose GPU. This is a huge deal in the world of AI data centers, where power and cooling costs can be a major expense.
But there are also some big challenges. The biggest one is the software. The CUDA ecosystem that NVIDIA has built is a massive moat. It’s the reason why so many developers are locked into the NVIDIA platform. If you’re building your own chip, you need to have a plan for how you’re going to compete with that.
Another challenge is the pace of innovation. The AI hardware space is moving at a breakneck speed. By the time you finish designing and building your chip, it might already be obsolete. You need to be constantly thinking about what’s next.
The Future of AI Chip Design
So what does the future of AI chip design look like? I think we’re going to see a few key trends emerge.
First, I think we’re going to see a move towards more specialized chips. Instead of one-size-fits-all GPUs, we’re going to see a proliferation of chips designed for specific AI workloads, like training, inference, and computer vision.
Second, I think we’re going to see a rise in new architectures. Things like 3D-IC stacking and chiplets are going to allow us to pack more and more computing power into a smaller and smaller space.
Third, I think we’re going to see a lot more innovation in the software layer. As more companies build their own custom silicon, we’re going to see a Cambrian explosion of new software and tools for AI development.
It’s an exciting time to be in the AI hardware space. The challenges are huge, but so are the opportunities. The companies that can figure out how to build the next generation of AI hardware are going to be the ones that shape the future of AI.
My Final Word
So if you’re a founder out there, my advice is simple: obsess over your hardware strategy as much as you obsess over your model architecture. It’s not the sexiest part of the job, I get it. But in the end, it’s the foundation everything else is built on. The decisions you make here will echo for years. They could be the difference between a footnote in history and the next big thing.
Don’t be afraid to think outside the box. Don’t be afraid to take risks. The future of AI is being built today, and it’s being built on a foundation of silicon. Make sure you’re building on solid ground.
Frequently Asked Questions
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.