I’ve seen the inside of hyperscale data centers that power the AI revolution. I’m not talking about a quick tour. I mean the kind of access you get when you’re trying to figure out why your models are crashing at 3 AM. The first time I walked into one, the sheer scale was staggering. It felt like stepping into a sci-fi movie—endless rows of humming servers, a city of machines thinking in unison. But what most people don’t get is that behind the sleek racks and blinking lights is a world of brutal trade-offs and logistical nightmares.
Everyone is obsessed with the latest AI models, the LLMs that can write poetry or generate photorealistic images. That’s the sexy part. But the real war, the one that determines who wins in the long run, is being fought in the trenches of hardware and infrastructure. I’ve been on the front lines of this war for years, both as a founder and an investor. I’ve seen companies with brilliant AI researchers fail because they couldn’t get their hands on enough GPUs. I’ve also seen teams with less-than-perfect models succeed because they had their infrastructure figured out. It’s the unglamorous, dirty secret of the AI industry.
The GPU Shortage is Real, and It’s Not Going Away
Remember the great toilet paper shortage of 2020? The GPU shortage is like that, but a thousand times worse and with no end in sight. When we were building RemoteTeam, we hit a wall. We needed to train our models, but we couldn’t get the GPUs we needed. We were a small startup, and the big players were buying up all the supply. We had to get creative. We ended up building a distributed network of smaller, less powerful GPUs, which was a massive engineering headache. It was a constant struggle to keep our models from crashing, and it slowed down our progress. It’s a story I’ve heard a hundred times since.
Today, the situation is even more dire. The demand for high-end GPUs from companies like NVIDIA is insatiable. I’ve talked to founders who have their entire business plan on hold because they’re waiting for a shipment of H100s. It’s a brutal game of who you know and how much you’re willing to pay. And it’s not just about the cost. It’s about the opportunity cost. Every day you’re not training your models, your competitors are. It’s a race, and the GPU shortage is the biggest bottleneck.
Custom Silicon: The New Arms Race
What do you do when you can’t buy the tools you need? You build them yourself. That’s what Google did with their Tensor Processing Units (TPUs), and it was a brilliant move. They saw the writing on the wall. They knew that relying on off-the-shelf hardware was a losing game in the long run. So they invested billions in developing their own custom chips, optimized for their specific AI workloads. It gave them a massive competitive advantage. They could train their models faster and more efficiently than anyone else.
Now, everyone is trying to catch up. Amazon has their Trainium and Inferentia chips. Microsoft is working on their own custom silicon. And it’s not just the big cloud providers. I’m seeing more and more startups that are designing their own chips from the ground up. It’s a huge undertaking, but the payoff can be enormous. It’s the ultimate form of vertical integration. When you control the hardware, you control your own destiny.
But let’s be clear: designing custom silicon is not for the faint of heart. It’s a long, expensive, and risky process. You need a team of world-class chip designers, and you need deep pockets. I’ve seen companies burn through hundreds of millions of dollars trying to develop their own chips, only to fail. It’s a high-stakes game, but it’s one that more and more companies are willing to play.
Beyond GPUs and TPUs: The Future of AI Hardware
As powerful as GPUs and TPUs are, they’re not the end of the story. The AI industry is constantly pushing the boundaries of what’s possible, and that requires new and innovative hardware. One of the most exciting areas of research is quantum computing. Quantum computers have the potential to solve problems that are intractable for even the most powerful classical computers. They could revolutionize drug discovery, materials science, and financial modeling. We’re still in the early days of quantum computing, but the progress is accelerating. I’ve invested in a few quantum computing startups, and I’m incredibly bullish on the long-term potential.
Another area that I’m watching closely is edge AI. The idea behind edge AI is to move the computation from the cloud to the devices themselves. This is important for applications that require low latency, such as self-driving cars and augmented reality. It’s also important for privacy, as it keeps the data on the device instead of sending it to the cloud. There are a number of startups that are developing specialized chips for edge AI, and I think we’re going to see explosive growth in this area over the next few years.
The Bottom Line
Building and scaling AI infrastructure is hard. It’s a constant battle against shortages, bottlenecks, and technical challenges. But it’s also where the real innovation is happening. The companies that are able to solve these problems are the ones that will win in the long run. It’s not just about having the best models. It’s about having the best hardware, the best infrastructure, and the best team. That’s the lesson I’ve learned over and over again in my career, and it’s the one I want to share with you.
Frequently Asked Questions
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.