'''I’m going to say something that might get me in trouble with the AI purists. Most of what you’re measuring in your AI analytics is probably garbage. Vanity metrics. Stuff that looks good in a pitch deck but does absolutely nothing to move the needle on your business.
I’ve seen it a hundred times. Founders who are obsessed with model accuracy, who can talk for hours about their F1 scores and their ROC curves. But when I ask them how their AI is actually helping their users or making them money, they just stare at me blankly.
I get it. I’ve been there. When I was building my first AI startup, MovieLaLa, I was drowning in data. I had dashboards coming out of my ears. I was tracking everything. And I was getting nowhere.
It wasn’t until we were acquired by Gfycat that I started to see the light. I had the opportunity to work with some of the smartest AI minds in the world, and I realized that they were looking at a completely different set of metrics than I was. They were looking at the metrics that actually mattered.
The Three AI Metrics That Actually Matter
It took me years of painful trial and error to figure this out. I want to save you that time and money. So here they are, the three AI metrics that I live by, the ones that have helped me build successful companies and guide my portfolio startups to success.
1. Predictive Value: Are You Calling the Shots Correctly?
This is the big one. The one that separates the pretenders from the contenders. Predictive value tells you how often your AI’s predictions are actually correct and lead to a desired outcome. It’s not about accuracy in a vacuum; it’s about accuracy in the real world.
I learned this the hard way at MovieLaLa. We had a recommendation engine that was 95% accurate in our offline tests. We were ecstatic. We thought we had cracked the code. But when we launched it, our user engagement barely budged. Why? Because our model was recommending movies that people had already seen or weren't interested in. It was accurate, but it wasn't valuable.
We were measuring the wrong thing. We were measuring how well our model could predict if a user would click on a movie, not if they would actually watch it and enjoy it. We were optimizing for clicks, not for user satisfaction.
So we changed our approach. We started tracking a new metric: Successful Recommendation Rate. This measured the percentage of recommendations that led to a user watching at least 30 minutes of a movie. It was a much harder metric to track, but it was the right one. Our accuracy dropped to 70%, but our user engagement shot up by 40%. We were finally making a real impact.
Another example is from a company I invested in, a fintech startup that was using AI to predict loan defaults. Their model was incredibly accurate, with a 98% success rate in identifying who would default. The problem was, it was too conservative. It was flagging so many good borrowers as potential defaulters that the company was losing out on a ton of business. Their predictive value was low because their predictions, while accurate, were not leading to the desired outcome of a healthy loan portfolio. They were optimizing for low defaults, not for a profitable business. They had to go back and retrain their model to accept a slightly higher risk of default in order to approve more loans and actually grow their business.
How to track it:
- Define what a "successful" prediction looks like for your business. Is it a user making a purchase? A user completing a task? A user not churning?
- Track the entire user journey, from the prediction to the outcome.
- Don't be afraid of a lower accuracy number if it means a higher success rate.
2. Time to Value: How Fast Can Your AI Deliver?
In the startup world, speed is everything. You need to be able to iterate quickly, learn from your mistakes, and adapt to changing market conditions. The same is true for your AI. Time to value measures how long it takes for your AI to deliver a tangible benefit to your users or your business.
At RemoteTeam, we were building an AI-powered tool to help companies manage their remote employees. We had a feature that could predict when an employee was at risk of burnout. It was a great idea, but it took our model 24 hours to process the data and make a prediction. By the time we alerted the manager, it was often too late.
Our time to value was too high. We had a powerful tool, but it was too slow to be useful. So we went back to the drawing board. We simplified our model, we optimized our data pipeline, and we got the prediction time down to under an hour. Suddenly, our burnout prediction feature became one of our most valuable tools. Managers could intervene early, provide support, and prevent their employees from quitting.
I saw a similar situation with a health-tech startup in my portfolio. They had developed an AI that could detect early signs of a rare disease from medical scans. The technology was groundbreaking, but it took three days to get a result back to the doctor. In a life-or-death situation, three days is an eternity. They were so focused on the accuracy of their model that they completely neglected the speed of delivery. They had to re-engineer their entire system to deliver results in under an hour. It was a massive undertaking, but it was the only way to make their product viable.
How to track it:
- Measure the time it takes from when your AI receives the data to when it delivers a result.
- Constantly look for ways to reduce this time. Can you simplify your model? Can you optimize your code? Can you use a faster infrastructure?
- Remember that a good enough model today is better than a perfect model tomorrow.
3. Cost Per Insight: Are You Getting a Return on Your AI Investment?
AI can be expensive. You have to pay for the data, the infrastructure, the talent, and the tools. It's easy to spend a lot of money on AI without getting a lot in return. That's why you need to track your cost per insight. This metric tells you how much you're spending to get a valuable piece of information from your AI.
I see this all the time with my portfolio companies. They get so excited about the potential of AI that they throw a ton of money at it without a clear plan. They hire a team of data scientists, they buy the latest and greatest tools, and they build complex models that don't actually solve a real problem. They end up with a very expensive science project, not a business.
That's why I always push my founders to be ruthless about their AI spending. I ask them: "What is the one question you want to answer with your AI? And how much are you willing to spend to get that answer?"
This forces them to focus on the most important problems and to be creative about how they solve them. Sometimes, the best solution is not a complex deep learning model. Sometimes, it's a simple regression analysis that you can run on your laptop.
One of my portfolio companies, a marketing tech startup, was spending a fortune on a sophisticated AI to personalize their email campaigns. They had a team of five data scientists and a massive AWS bill. But when I dug into their numbers, I found that their fancy AI was only marginally better than a simple A/B test. They were spending millions of dollars for a 2% lift in open rates. Their cost per insight was through the roof. I advised them to ditch the complex AI and go back to basics. They were not happy about it, but they did it. And their marketing ROI went up by 30%.
How to track it:
- Calculate the total cost of your AI, including salaries, infrastructure, and tools.
- Define what a valuable "insight" looks like for your business.
- Divide your total cost by the number of insights you're generating.
- Constantly look for ways to reduce your cost per insight. Can you use a cheaper tool? Can you automate some of your data processing? Can you use a simpler model?
Stop Drowning in Data and Start Making an Impact
So there you have it. The three AI metrics that have helped me build two successful companies and invest in over 200 more. Predictive Value, Time to Value, and Cost Per Insight. These are the metrics that will help you separate the signal from the noise, the vanity from the value.
Stop chasing after a 99% accuracy rate. Stop building complex models that take forever to run. Stop throwing money at AI without a clear return on investment.
Instead, focus on what really matters: delivering value to your users and your business. If you can do that, you’ll be well on your way to building a successful AI-powered startup. '''
Frequently Asked Questions
Do I need technical skills to discovered the 3 ai metrics that actually impact your startup?
Not necessarily. While technical understanding helps, the most important skills are clear thinking and the ability to break problems into smaller pieces. Many successful founders I've invested in started with zero technical background and either learned enough to be dangerous or found the right technical partner.
How do I measure success with this approach?
Pick one or two metrics that directly tie to your goal and track them weekly. Vanity metrics like page views or follower counts rarely matter. Focus on metrics that reflect real engagement or revenue impact.
What tools do I need to get started?
Start with the basics. You don't need expensive software or fancy tools. A spreadsheet, a note-taking app, and direct access to your customers will get you further than any enterprise platform. Add tools only when you hit a specific bottleneck.