I was on a board call last month. The CFO puts up a slide, and my eyes went straight to one line item: 'AI Compute - $100,000/month'. For a seed-stage company. The crazy part? After a 30-minute conversation, we found a way to cut that bill by 90% without crippling their product.
Everyone's talking about training costs. The headlines love the drama of nine-figure training runs. But that's a distraction. For 99% of companies, the thing that actually eats your runway isn't training, it's inference. It's the cost of every single API call your product makes. It's death by a thousand cuts.
As an investor in over 200 startups, including Anthropic, OpenAI, and Scale, I see the Stripe bills. I know what kills companies. It's not a lack of ambition or a bad idea. It's running out of money. And right now, too many founders are building on a fundamentally broken cost structure, completely blind to the monster hiding in their AWS bill.
I'm writing this because I'm tired of watching good founders make the same bad math. So I analyzed the top 20 LLMs to find the truth. Not the marketing fluff, but the real numbers. I looked at what actually drives costs up and what you can do about it. The results might surprise you.
The Sticker Price is a Lie
The first mistake everyone makes is looking at the price per million tokens and calling it a day. It’s a lie. Or at least, it’s only a tiny part of the story.
Most pricing pages show you two numbers: a cost for input tokens and a cost for output tokens. And guess what? Output is almost always more expensive. For some models, it’s 3-5x more expensive. Founders get seduced by a low input cost, thinking they’ve found a bargain. They build their entire application around it, only to realize that their specific use case—like generating long-form content or detailed code—is overwhelmingly driven by output tokens. Their bill is 300% higher than their spreadsheet predicted.
I saw this with a legal tech company I advise. They were using a model with a super cheap input price to summarize long legal documents. But the summaries themselves were also quite long, and the high output cost was eating their lunch. We switched them to a different model with a higher input cost but a much more balanced input/output ratio. Their monthly bill dropped by 40% overnight. No change to the user experience, just a smarter model choice.
You have to calculate your effective cost. Look at your application’s typical workflow. What is your ratio of input to output tokens? Is it 10:1, 1:1, 1:10? Do the math. Create a simple calculator that multiplies your expected token counts by the actual input and output costs. That’s the only number that matters.
Speed Kills (Your Margins)
Here’s another trap: picking the "cheapest" model without looking at throughput. Cost per token is meaningless if the model is so slow it makes your product unusable.
Latency is a product killer. Users will not wait 10 seconds for an answer from your AI chatbot. They’ll just leave. Throughput, the number of tokens you can process per second, is a direct constraint on your growth.
I had a portfolio company building a customer service bot. They chose a model that was, on paper, 50% cheaper than the next best alternative. But it was painfully slow. The user experience was terrible. Customers were getting frustrated and support ticket volume wasn’t going down. They were forced to switch to a pricier, high-throughput model. The cost per token doubled, but guess what happened? User satisfaction shot up. The bot could handle conversations so much faster that it resolved 3x more issues. The ROI from the improved customer experience completely dwarfed the increase in inference cost.
When you analyze models, don’t just look at the price. Benchmark the throughput for your specific task. How many concurrent users can you support with a single instance? A model that costs 2x more but has 5x the throughput is actually the cheaper option. It allows you to serve more users with less infrastructure.
The Hidden Costs Will Bankrupt You
This is the part that almost no one talks about. The advertised cost per token is just the tip of the iceberg. The real expenses are buried in the operational complexity of running these things at scale.
Self-Hosting vs. API: And then there's the self-hosting trap. The siren song of 'cutting out the middleman' and running models on your own iron. It sounds smart. It's a classic engineer's fantasy. But the reality is a special kind of hell. Suddenly your best engineers aren't building your product; they're wrestling with CUDA drivers and GPU memory allocation. Unless you're operating at massive scale, just use an API. The premium you pay is cheap insurance against distraction.
Model Distillation & Quantization: You don’t always need the 200-billion parameter monster model. For many, many tasks, a smaller, distilled model is good enough. Model distillation is the process of training a smaller, faster model to mimic the performance of a larger one. Quantization is a technique to run models with lower precision, which dramatically reduces memory and computational costs. One of my most successful investments in the AI space built their entire business on this. They use the biggest, most powerful models to generate high-quality synthetic data, and then use that data to train a tiny, specialized model that costs 1/100th as much to run. It’s fast, cheap, and perfectly tuned for their specific use case. They get the quality of a massive model for the price of a tiny one.
The Cost of "Good Enough": The final hidden cost is chasing perfection. Your model doesn’t need to be perfect. It needs to be good enough to solve the user’s problem. I see founders endlessly tweaking prompts and fine-tuning models to get from 95% accuracy to 96% accuracy. That last 1% can increase your costs by 10x. It might require a much larger model or a complex chain of prompts. Is it worth it? Almost never. Ship the 95% solution. Get user feedback. You’ll often find that users don’t even notice the difference. But they will notice if your product is too expensive or too slow.
A Framework for Sanity
So how do you avoid these traps? It’s not about finding the single "cheapest" model. It’s about building a smart, flexible AI stack.
Tier Your Models: Stop using a sledgehammer to crack a nut. Not every task needs GPT-4. Build a routing layer. Use a cheap, fast model for simple stuff (like classifying an email's sentiment). Use your expensive, state-of-the-art model only when you absolutely have to. This one change can cut costs by 50% or more.
Cache Everything: If two users ask the same question, you should only have to pay for the answer once. Aggressive caching is one of the most effective, yet most overlooked, ways to reduce inference costs.
Negotiate: Once you have significant volume, talk to the model providers. They want your business. You can often negotiate custom pricing, dedicated capacity, and better terms. Don’t just accept the public pricing.
At the end of the day, inference cost isn't a technical problem. It's a business model problem. It requires a product mindset, not a research one. Stop chasing the biggest, baddest model and start chasing the smartest, most economical implementation. The startups that get this right are the ones that will still be here in five years. The rest are just building very expensive, very cool science projects.
Frequently Asked Questions
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.