Everyone talks about AI models, but nobody talks about the brutal reality of the hardware that runs them. Here's the unfiltered truth about what it really takes to build and scale AI infrastructure.
I’ve been in the Silicon Valley trenches for a long time. I’ve built and sold two companies, RemoteTeam to Gusto and MovieLaLa to Gfycat. I’ve also been fortunate enough to be an early investor in over 200 companies, including some of the biggest names in AI today like Anthropic, OpenAI, Scale AI, and Hugging Face. And I can tell you one thing for sure: most founders are thinking about AI infrastructure all wrong.
They’re so focused on the models, the algorithms, the software, that they completely forget about the hardware. It’s like trying to build a skyscraper without thinking about the foundation. It’s a recipe for disaster.
I’ve seen it happen time and time again. A promising startup with a brilliant idea, a great team, and a beautiful product. But they hit a wall, and they can’t figure out why. Their models are too slow, their costs are out of control, and they can’t scale. And it’s all because they didn’t pay enough attention to the hardware.
I’m writing this because I wish someone had told me all of this when I was starting out. I had to learn these lessons the hard way, through years of trial and error, and millions of dollars down the drain. I want to save you from making the same mistakes.
The GPU Shortage Nightmare I Lived Through
I remember it like it was yesterday. We were in the early days of MovieLaLa, and we were building a recommendation engine that was powered by deep learning. We needed a lot of GPUs to train our models, but we couldn't get our hands on them. The GPU shortage was in full swing, and the big cloud providers were hoarding all the supply.
We were a small startup, and we couldn't compete with the likes of Google and Amazon. We were stuck. We tried everything. We begged, we borrowed, we even tried to build our own GPUs from scratch. It was a nightmare.
We eventually managed to scrape together enough GPUs to get by, but it was a constant struggle. We were always worried about running out of capacity, and we could never train our models as fast as we wanted to. It was a huge bottleneck, and it slowed us down tremendously.
This experience taught me a valuable lesson: you can't rely on the big cloud providers for your AI infrastructure. They’re not your friends. They’re your competitors. They want to lock you into their ecosystem and squeeze every last penny out of you.
Big Cloud or Your Own Metal? It's Not Even a Question.
This is probably my most controversial opinion, but I believe that for any serious AI company, owning your own hardware is a massive competitive advantage. I know, I know, everyone is talking about the cloud. It’s flexible, it’s scalable, it’s easy to use. But it’s also a trap.
When you use the cloud, you’re at the mercy of the cloud provider. They can change their prices, their terms of service, their APIs, at any time. You have no control. You’re just a renter.
When you own your own hardware, you’re in control. You can optimize it for your specific workload, you can get better performance, and you can save a lot of money in the long run. It’s a bigger upfront investment, but it pays off.
At RemoteTeam, we made the decision to build our own infrastructure from day one. It was a bold move, but it was the right one. It allowed us to move faster, to iterate more quickly, and to build a better product. It was a key part of our success, and it’s one of the reasons why Gusto acquired us.
The Edge is Sharper Than You Think
Another area where I see a lot of founders making mistakes is with edge AI. They think of it as a niche market, something that’s only relevant for self-driving cars or drones. But the truth is, the edge is the future of AI.
As AI models get bigger and more powerful, it’s becoming increasingly impractical to run them in the cloud. The latency is too high, the bandwidth is too limited, and the costs are too prohibitive. The only way to make AI truly ubiquitous is to run it at the edge, on the devices themselves.
This is a huge opportunity for startups. The big cloud providers are not well-positioned to win at the edge. They’re too centralized, too slow, too expensive. Startups that can build a decentralized, edge-native AI platform will be the winners in this new paradigm.
Custom Silicon: The Ultimate Moat
This is where things get really interesting. I believe that the ultimate moat for an AI company is to build its own custom silicon. I know it sounds crazy. Building a chip is hard, it’s expensive, and it’s risky. But it’s also the only way to get the best possible performance and efficiency for your AI workload.
Look at what Google has done with the TPU, or what Tesla has done with their self-driving chip. They’re not doing it because it’s fun. They’re doing it because it gives them a massive competitive advantage. It allows them to build products that are years ahead of the competition.
I’m not saying that every startup should go out and build its own chip. But I am saying that you should be thinking about it. You should be thinking about what your long-term hardware strategy is. Because if you’re not, you’re going to get left behind.
This is why I’ve invested in companies like Anthropic, OpenAI, and Scale AI. They understand the importance of hardware. They’re thinking about the full stack, from the silicon all the way up to the application. They’re building the future of AI, and they’re doing it on their own terms.
Don't Even Bother With Quantum... For Now
I get asked about quantum computing a lot. It’s a fascinating field, and it has the potential to revolutionize computing as we know it. But it’s also a long way off. We’re still in the very early days of quantum computing, and it’s not something that founders need to be worrying about right now.
There are a lot of other, more pressing problems to solve. Like the GPU shortage, the tyranny of the big cloud providers, and the transition to the edge. These are the problems that will define the next decade of AI. And these are the problems that I’m focused on.
My Unfiltered Advice to You
So, what’s the bottom line? If you’re a founder in the AI space, you need to be thinking about hardware from day one. Don’t make the same mistakes I did. Don’t get locked into the big cloud providers. Don’t ignore the edge. And don’t be afraid to think big, to think about building your own custom silicon.
It’s not going to be easy. It’s going to be hard. But it’s also the only way to build a truly defensible, long-lasting AI company. The future of AI is not just about software. It’s about the full stack. And the founders who understand that are the ones who will win.
Frequently Asked Questions
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.