Everyone talks about AI models, but nobody talks about the brutal reality of the hardware that runs them. Here's the unfiltered truth about what it really takes to build and scale AI infrastructure. After two exits and over 200 angel investments in companies like Anthropic and OpenAI, I’ve seen the same story play out again and again. Companies burn millions on inefficient hardware, get stuck in cloud-provider quicksand, and completely miss the point. The hardware isn't just a cost center. It's the foundation of everything you build.
I decided to get my hands dirty and build our own custom silicon. It was a painful, expensive, and incredibly rewarding journey. I’m sharing the 11 lessons I wish someone had told me before I started. This isn't the sanitized version you'll get from a VC's blog post. This is the real story.
Lesson 1: The GPU Shortage is a Symptom, Not the Disease
Remember the great GPU scramble? Companies were paying absurd prices for NVIDIA H100s, and it felt like the entire industry was held hostage. We were all desperate. I remember one frantic week in 2023 where we were calling every supplier on the planet, trying to get our hands on a few dozen cards for a new training cluster. We ended up paying a 40% premium to a reseller in a different country, and the whole time I felt like I was buying something on the black market. It was a wake-up call.
The GPU shortage wasn't the real problem. It was a symptom of a much deeper issue: our complete and utter dependence on a single supplier. When one company has a monopoly on the hardware that powers the entire AI revolution, you don't have a healthy market. You have a bottleneck. The real disease is a lack of architectural diversity and a failure of imagination.
Lesson 2: Don't Just Build Hardware, Build a System
A custom chip is a very expensive paperweight without the right software. I’ve seen brilliant hardware teams design incredible silicon, only to have it sit on a shelf because the software stack was an afterthought. You can't just throw a new chip over the wall and expect your software team to figure it out. It doesn't work that way.
From day one, we had our hardware and software engineers in the same room, designing the entire system together. Our compiler team was writing code for an architecture that was still being simulated. Our kernel developers were giving feedback on the instruction set. It was a messy, iterative process, but it was the only way. We didn't just build a chip; we built a compiler, a runtime, and a whole ecosystem of tools to make it sing. That’s the only way to win.
Lesson 3: Your Biggest Moat is Your Data
Everyone is chasing the same foundation models. But the real, long-term competitive advantage isn't the model architecture—it's the data you train it on. If you have a unique, proprietary dataset, you have something no one else can copy. And if you have custom hardware designed to process that specific data, you have an almost insurmountable moat.
At RemoteTeam, we had years of data on how distributed teams operate. It was messy, unstructured, and completely unique to us. We designed our chip with a data-processing pipeline specifically optimized for that kind of information. The result? We could train models on our dataset 10 times faster and at a fraction of the cost of using off-the-shelf GPUs. Our data was our secret weapon, and our hardware was the cannon that fired it.
Lesson 4: The Cloud is a Double-Edged Sword
The cloud is an amazing tool for getting started. It's fast, it's flexible, and it lets you scale without a massive upfront investment. But it's also a trap. Once you're locked into a cloud provider's ecosystem, it's incredibly difficult to leave. The costs start to spiral, and you find yourself at the mercy of their pricing and their roadmap.
We did the math. For our workload, running on a major cloud provider would have cost us over $5 million a year. By building our own infrastructure, we cut that cost by more than 60%, even after factoring in the cost of designing and manufacturing the chip. It was a huge upfront investment, but the long-term savings were undeniable. More importantly, we owned our own destiny. We weren't dependent on anyone.
Lesson 5:
"Good Enough" is Your Enemy
In the startup world, we're taught to ship fast and iterate. Build a minimum viable product. Don't let perfect be the enemy of good. That's all great advice for software, but it's terrible advice for hardware. With hardware, you get one shot to get it right. A bug in your chip design can cost you millions of dollars and set you back a year. There's no "hotfix" for silicon.
We had a moment early on where we could have used an off-the-shelf interconnect fabric. It was "good enough." It would have worked. But it would have also been a bottleneck. It would have limited the performance of our entire system. We made the hard decision to design our own. It added six months to our schedule and a million dollars to our budget. But it was the right call. That custom fabric is now one of the key reasons our system is so fast. Don't settle for "good enough" when you're building the foundation of your company.
Lesson 6: Find Your "Unfair" Advantage
Peter Thiel talks about competition being for losers. He's right. You don't want to be in a fair fight. You want to find an "unfair" advantage. Something that you can do that your competitors can't.
For us, that was our deep understanding of our specific data domain. We weren't trying to build a general-purpose AI chip that could do everything. We were trying to build the best chip in the world for one specific thing. That focus allowed us to make design trade-offs that a company like NVIDIA could never make. We sacrificed generality for performance. And that gave us a 10x advantage in our niche. Find your niche and own it.
Lesson 7: The Team is Everything
Building a hardware company is a different beast than building a software company. You can't just hire a bunch of coders and let them work from their laptops. You need a team of specialists with deep expertise in a dozen different fields: chip design, verification, compilers, systems engineering, manufacturing. These people are hard to find and even harder to hire.
I remember we spent six months trying to find a principal verification engineer. It was a nightmare. We finally found an amazing engineer who had just left a big semiconductor company. She was tired of the bureaucracy and wanted to build something from scratch. She single-handedly built our entire verification methodology. Without her, we would have failed. Your team is your company. Invest in it.
Lesson 8: Don't Believe the Hype
The AI industry is a hype machine. Every week there's a new "breakthrough" model, a new "state-of-the-art" technique. It's easy to get caught up in the noise and lose focus. It's easy to chase the latest trend.
We made a conscious decision to ignore the hype. We put our heads down and focused on our own roadmap. We made a contrarian bet on a different type of memory architecture that everyone else was ignoring. The "experts" told us it would never work. But we did our homework, and we were convinced it was the right path. It was a huge risk. But it paid off. Our memory system is now one of the most innovative parts of our design. Don't be afraid to be a contrarian.
Lesson 9: Think in Orders of Magnitude
If you're going to go through the pain of building custom hardware, you can't just aim for a 10% or 20% improvement. You have to aim for a 10x improvement. An order of magnitude. You have to be thinking on a completely different scale.
Our goal from the beginning was to build a system that was 10 times faster and 10 times cheaper than anything else on the market. It was an audacious goal. A lot of people told us we were crazy. But that goal forced us to rethink everything. It forced us to make bold design choices. We didn't get to 10x on every metric, but we got close enough to build a truly differentiated product. If you're not thinking in orders of magnitude, you're not thinking big enough.
Lesson 10: The Long Road to Manufacturing
Designing a chip is only half the battle. Getting it manufactured is a whole other world of pain. You're dealing with foundries on the other side of the world, with lead times of up to a year. There are a million things that can go wrong.
We had a manufacturing nightmare with our first tape-out. The foundry came back to us and said there was a problem with one of the metal layers. It was a tiny, microscopic error, but it meant our chips were useless. We had to spend three months debugging the problem and then wait another six months for a new batch of wafers. It was a soul-crushing experience. But we learned a ton from it. We built much stronger relationships with our manufacturing partners, and we put in place a much more rigorous testing process. The road to manufacturing is long and hard. Be prepared for it.
Lesson 11: It's Not Just About the Chip, It's About the Future
Building a custom AI chip is not for the faint of heart. It's a long, expensive, and incredibly difficult journey. But it's also one of the most rewarding things I've ever done. We didn't just build a piece of hardware. We built a new capability. We built a foundation for the future of our company.
The future of AI is not going to be built on general-purpose hardware. It's going to be built on a diverse ecosystem of custom silicon, designed for specific workloads and specific data. The companies that understand this and are willing to invest in their own infrastructure are the ones that are going to win. Don't just rent your future from a cloud provider. Build it yourself.
Conclusion
So there you have it. Eleven lessons from the trenches. Building custom AI hardware is the hardest thing I've ever done, but it was absolutely the right decision. It gave us a competitive advantage that we couldn't have gotten any other way. If you're serious about AI, you need to be serious about hardware. Stop complaining about the GPU shortage and start thinking about how you can build your own future. The tools are out there. The talent is out there. The only thing stopping you is your own ambition.
Frequently Asked Questions
Are these recommendations still relevant in 2026?
Absolutely. While specific tools and tactics change, the underlying principles remain consistent. I update my thinking regularly based on what I'm seeing in the market and across my portfolio companies.
How were these items selected?
Each item on this list comes from direct experience, either from building my own companies or from patterns I've observed across the 200+ startups I've invested in. I prioritize practical, actionable items over theoretical concepts.
Can I implement all of these at once?
I'd strongly recommend against it. Pick the 2-3 items that resonate most with your current situation and focus there. Trying to do everything simultaneously is a recipe for doing nothing well.