What Building a Custom AI Chip Taught Me About Tech and Tough Choices

I remember the exact moment I knew we had to build our own silicon. We were staring at a server rack, lights blinking, fans screaming, and a power bill that looked more like a rounding error in a nation's budget. We were trying to train a new model, and the off-the-shelf GPUs just weren't cutting it. They were too slow, too power-hungry, and way too expensive to scale to the level we needed.

Most people in the AI world live and breathe software. They obsess over models, algorithms, and data. But I’m here to tell you a secret: the hardware is where the real magic, and the real pain, happens. Having built AI hardware in Silicon Valley, I’ve seen firsthand how critical it is to get it right. I’m going to break down the real challenges we faced, from dealing with GPU shortages to designing our own silicon—and what those experiences taught me about building technology that lasts.

Lesson 1: The "Buy vs. Build" Question is a Trap

Everyone in tech loves a good "buy vs. build" debate. When it comes to AI hardware, the conventional wisdom is to always buy. Why? Because designing and manufacturing your own chip is insanely hard. We're talking hundreds of millions of dollars in R&D, a team of the most expensive engineers on the planet, and a timeline that can stretch for years. And at the end of it all, you might have a very expensive paperweight.

I remember a specific project where we were trying to build a real-time object detection system for a new kind of autonomous drone. This wasn't your average consumer drone; it was a ruggedized industrial drone designed for inspecting miles of remote pipeline. The drone needed to fly at high speed while identifying tiny cracks and signs of corrosion, some only a few millimeters wide. The performance requirement was insane: we needed to process a high-resolution video stream in real-time, with a latency of under 10 milliseconds. Anything more, and the drone would have flown past the defect before it could be flagged.

We bought the latest and greatest GPUs, the ones that all the benchmarks said were the best. We spent weeks, then months, trying to optimize our models to run on them. We even flew to the GPU manufacturer's headquarters and begged their engineers for help. They were smart people, but they couldn’t change the fundamental limitations of their hardware. The latency was just too high. The drone would have been a hundred meters past the object it was supposed to be detecting.

That was the "aha" moment. We had a whiteboard covered in equations, diagrams, and desperation. We had calculated the cost of buying enough GPUs to get the performance we needed. The number was astronomical. It would have bankrupted the company. It was then that we realized the "buy vs. build" question was a false dichotomy. It wasn’t a choice. It was a necessity. We had to build. The conversation with our investors was not easy. They were skeptical, and rightly so. But we showed them the data, the simulations, and our unwavering conviction. We told them that this was the only way to build a defensible business, to create a product that was not just 10% better, but 10x better. They took a leap of faith with us, and for that, I will always be grateful.

Lesson 2: There's No Such Thing as a Free Lunch (Especially in Physics)

When you’re designing a chip, you’re constantly fighting the laws of physics. There are three things you want: more performance, less power, and a smaller size. The problem is, you can only pick two. This is the classic engineering tradeoff, and in AI hardware, it’s amplified by a factor of a thousand.

Think of it like trying to design a race car. You want it to be as fast as possible, as fuel-efficient as possible, and as small and light as possible. You can make it faster by putting in a bigger engine, but that will make it heavier and less fuel-efficient. You can make it lighter by using exotic materials, but that will make it more expensive. There are no easy answers.

We had one debate that went on for weeks. It was about the number of bits we should use for our calculations. The industry standard was 32-bit floating-point numbers. But some of our engineers argued that we could get a huge performance boost by using 16-bit or even 8-bit numbers. The problem was that lower precision could lead to less accurate results. We ran simulations, we built prototypes, we argued until we were blue in the face. In the end, we came up with a hybrid approach that used different precisions for different parts of the calculation. It was a complex solution, but it gave us the best of both worlds: high performance and high accuracy.

Another huge challenge was managing power consumption. In a chip with billions of transistors, a significant portion of the silicon is dedicated to just moving data around. This is known as the "von Neumann bottleneck," and it's a huge source of inefficiency. We spent a lot of time and energy designing a memory architecture that would minimize data movement. We used a combination of on-chip SRAM, which is very fast but expensive, and off-chip DRAM, which is slower but cheaper. We also developed a sophisticated caching system that would keep frequently used data close to the processing units. It was a delicate balancing act, but it was essential to achieving our power and performance goals.

Lesson 3: Edge AI is a Different Beast

Not all AI is created equal. There’s a world of difference between running a massive model in a data center and running a model on a device at the edge, like a smartphone or a car. Data centers have virtually unlimited power and cooling. Edge devices have a tiny battery and have to survive in the real world.

I’m talking about drones flying in the desert, cameras operating in sub-zero temperatures, and medical devices that have to be 100% reliable. These are not friendly environments for electronics. You have to worry about things like heat, vibration, and power spikes. You also have to worry about security. An edge device is out in the wild, where it can be tampered with. You need to build in security from the ground up.

Our edge AI chip was a marvel of engineering. We used a new type of non-volatile memory that was more resistant to temperature changes. We developed a novel power management system that could dynamically adjust the chip’s performance to match the workload. And we built in a hardware root of trust to ensure that the device could not be compromised. It was a completely different design philosophy from the data center world, and it required a completely different set of skills.

One of the biggest challenges with edge AI is the diversity of applications. In the data center, you're typically running a small number of large models. At the edge, you have thousands of different applications, each with its own unique requirements. You need a chip that is flexible enough to handle all of them. This is where the concept of a "Tensor Processing Unit" or TPU comes in. A TPU is a specialized processor that is designed to accelerate the matrix multiplication operations that are at the heart of most AI models. By creating a flexible and programmable TPU, we were able to build a chip that could be adapted to a wide range of edge AI applications.

Lesson 4: The Software Stack is Everything

You can have the best chip in the world, but if you don’’t have the software to run on it, you have nothing. This is a lesson that many hardware startups learn the hard way. They spend all their time and money on the silicon, and then they realize that no one knows how to program it.

Building a compiler is one of the most challenging tasks in computer science. It’s a mix of art and science. You have to understand the intricacies of the hardware, the nuances of the programming language, and the goals of the developer. Our compiler team was made up of some of the smartest people I’ve ever worked with. They were obsessed with performance, and they would spend weeks trying to shave a few percentage points off the execution time of a model.

But it wasn’t just about performance. It was also about the developer experience. We wanted to make it as easy as possible for developers to use our hardware. We created a rich set of documentation, tutorials, and examples. We built a community forum where developers could ask questions and share their work. We knew that if we could win the hearts and minds of developers, we would win the market. We spent almost as much on our software team as we did on our hardware team. It was a huge investment, but it was absolutely worth it. The software is what turns a piece of silicon into a platform.

Lesson 5: The Future is Custom

For a long time, the semiconductor industry was dominated by a few big players. But that’s starting to change. The rise of AI is creating a new wave of demand for custom silicon. Companies like Google, Amazon, and Meta are all designing their own chips, and for good reason. They understand that to win in AI, you need to control your own destiny. You can’t be dependent on a single supplier.

This is a huge opportunity for startups. The big players are focused on the mass market. They are building chips that are good enough for everyone. But there are a thousand niche markets that are being underserved. There are opportunities to build custom chips for everything from medical imaging to financial trading to scientific research.

Building our own AI chip was one of the hardest things I’ve ever done. It was a long, expensive, and often frustrating journey. But it was also one of the most rewarding. We pushed the boundaries of what was possible, and we built a product that is making a real difference in the world.

So, if you’re an entrepreneur out there with a crazy idea for a new piece of hardware, don’t let anyone tell you it can’t be done. It will be hard, but it might just be worth it. The future is custom, and it’s waiting for you to build it. Don’t be afraid to take the leap. The world needs your crazy ideas. The next great semiconductor company could be started in a garage, just like the last one. It could be you.

Frequently Asked Questions

What experience informs this perspective?

This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.

What's the most common pushback you get on this?

People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.

How can I apply this thinking to my own situation?

Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.

How has this view evolved over time?

My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.