''' I’ve seen it a dozen times. A sharp founding team, a product that has real potential, and a dashboard full of metrics that look like a NASA control panel. They’re tracking everything. Daily actives, session duration, feature adoption, conversion funnels, cohort retention. You name it, they have a chart for it.
And yet, they’re flying blind.
They come to me for advice, and I ask a simple question: “What’s the most important user action in your product?” They can’t answer. Not with conviction, anyway. They’ll say, “Well, it depends…” or “We’re tracking a few key activation events…”
Stop. You’re committing the first deadly sin of AI product analytics.
For the past decade, I’ve been in the trenches building and investing in companies. I’ve had two exits—RemoteTeam to Gusto and MovieLaLa to Gfycat—and I’ve angel invested in over 200 companies, including giants like Anthropic, OpenAI, and Scale AI. I’ve seen what separates the products that take off like a rocket from the ones that fizzle out. It’s not about having more data. It’s about having the right data, and more importantly, the right mindset.
Most of the advice out there on product analytics is garbage. It’s written by data scientists who’ve never had to ship a product or make payroll. They talk about statistical significance and p-values, but they don’t talk about the gut-wrenching uncertainty of being a founder. This guide is different. This is for the founders in the arena.
Here are the seven deadly sins of AI product analytics, and how to confess, repent, and save your product.
Sin #1: Vanity Metrics Over Actionable Insights
This is the original sin. It’s the easiest one to fall for. You log in, see your daily active users are up 5%, and you get a little dopamine hit. Feels good, right? But what does it actually tell you? What are those users doing? Are they the right users? Are they getting value? Or did a botnet just discover your free trial?
I once worked with a startup that was obsessed with their registration numbers. They were growing 20% week-over-week. They were popping champagne. But their retention was a leaky bucket. Users signed up, poked around for 30 seconds, and never came back. The numbers looked great on a slide deck, but the business was dying.
The Repentance:
Stop tracking metrics that make you feel good and start tracking metrics that make you think. For every metric on your dashboard, ask yourself: “What decision will I make differently based on this number?” If you don’t have a good answer, kill the chart. It’s just noise.
Instead, focus on a “North Star” metric. This is the one number that best captures the core value you’re creating for your users. For Facebook, it was daily active users. For Airbnb, it was nights booked. For my company, RemoteTeam, it was the number of international employees paid successfully each month. Find your North Star and align your entire company around it.
Sin #2: Analysis Paralysis
The modern data stack is a miracle. With a few clicks, you can pipe every user event into a data warehouse, hook up a BI tool, and generate a thousand different charts. The problem is, you can spend all your time analyzing and no time building.
I’ve seen founders get so lost in the data that they forget to talk to their users. They’ll run A/B tests on button colors while ignoring the glaring feedback in their support tickets. Data is a powerful tool, but it’s not a substitute for intuition and qualitative feedback.
The Repentance:
Timebox your analysis. Give yourself a set amount of time each week—say, two hours—to dig into the data. Look for anomalies, trends, and opportunities. Form a hypothesis. Then, get out of the dashboard and go validate it. Talk to users. Ship a feature. Run a real-world experiment.
Remember, the goal is not to be right. The goal is to learn. And the fastest way to learn is by doing, not by analyzing.
Sin #3: Ignoring the “Why” Behind the “What”
Your analytics can tell you what your users are doing. They can’t tell you why. They can show you that 50% of users drop off during onboarding, but they can’t tell you it’s because your UI is confusing or your value proposition is unclear.
This is where so many AI-driven analytics tools fall short. They can find correlations, but they can’t explain causation. They might tell you that users who invite a teammate in their first session are 10x more likely to retain. That’s interesting, but is it because inviting is a magic feature, or is it because only the most motivated users bother to invite anyone in the first place? It’s a classic correlation vs. causation trap.
The Repentance:
Combine quantitative data with qualitative insights. When you see a surprising pattern in your analytics, go find the users behind the data points. Send them an email. Get them on a Zoom call. Ask them open-ended questions. “Walk me through the last time you used our product.” “What were you trying to accomplish?” “Was there anything you found frustrating?”
I can’t tell you how many times a 15-minute user interview has given me more insight than a week of data analysis. You’ll be amazed at what you learn when you just shut up and listen.
Sin #4: Worshipping at the Altar of A/B Testing
A/B testing is a powerful tool for optimization. It’s great for tweaking headlines, button colors, and checkout flows. But it’s a terrible tool for innovation.
You can’t A/B test your way to a breakthrough product. You can’t A/B test a vision. Steve Jobs didn’t A/B test the iPhone. Henry Ford didn’t A/B test the Model T. They had a point of view. They had conviction.
I see too many founders abdicating their responsibility to have a vision. They treat A/B testing as a magic 8-ball that will give them all the answers. They’re so afraid of being wrong that they only take tiny, incremental steps. They’re optimizing for local maxima while their competitors are finding a whole new mountain.
The Repentance:
Use A/B testing for what it’s good for: optimization, not ideation. Use it to refine your product, not to define it. And never, ever test something you don’t believe in. If you have a strong conviction about a feature, just ship it. Call it a “CEO test.” If it works, great. If it doesn’t, you’ll learn something and you can try again.
Your job as a founder is to have a vision and to lead your team towards it. Don’t let the false precision of A/B testing distract you from that.
Sin #5: The “Data-Driven” Charade
This one drives me crazy. People throw around the term “data-driven” as a way to sound smart and objective. But in reality, it’s often a cover for a lack of conviction. It’s a way to avoid making a tough call.
I was in a board meeting once where the CEO presented a 50-slide deck full of charts and graphs to justify a minor product change. It was a masterclass in data theater. After 30 minutes, I finally interrupted him. “Just tell me,” I said, “do you believe this is the right thing to do?” He was taken aback. He stammered for a bit and then admitted that he wasn’t sure. He was hoping the data would make the decision for him.
The Repentance:
Be data-informed, not data-driven. Use data to inform your intuition, not to replace it. Look at the numbers, listen to your users, and then make a decision. Own it. If you’re right, give your team the credit. If you’re wrong, take the blame. That’s leadership.
And for God’s sake, stop using data to win arguments. The goal is to find the truth, not to be right. If you’re using data to bludgeon your opponents into submission, you’re doing it wrong.
Sin #6: Building a Data Science Skunkworks
Every founder I know wants to hire a team of PhDs from Google and have them build some magical AI that will solve all their problems. They think that if they just had enough smart people and enough data, they could predict the future.
Here’s the reality: you don’t need a massive data science team. At least not at first. In the early days, your biggest risk is not the sophistication of your models; it’s the relevance of your product. You need to be focused on finding product-market fit, not on building a perfect churn prediction model.
I’ve seen startups burn through millions of dollars in venture capital building a data science team that was completely disconnected from the product team. They were working on “interesting” problems, but they weren’t shipping anything that moved the needle for the business.
The Repentance:
Democratize your data. Make it easy for everyone in your company—not just the data scientists—to access and understand your product analytics. Invest in self-serve tools like Amplitude or Mixpanel. Train your product managers and designers to answer their own questions.
Your goal should be to have a team of product-aware data people, not a team of data-aware product people. The insights should be coming from the people who are closest to the users and the product, not from an ivory tower.
Sin #7: Believing Your Own Hype
This is the most dangerous sin of all. It’s the one that can kill your company even when everything seems to be going right.
As your company grows, you’ll start to get positive reinforcement. Your metrics will go up and to the right. Your investors will tell you you’re a genius. You’ll get invited to speak at conferences. It’s easy to start believing your own hype. It’s easy to get complacent.
I saw this happen with a portfolio company a few years ago. They were the darlings of Silicon Valley. They had raised a massive round of funding at a crazy valuation. Their growth was off the charts. But they had stopped innovating. They were so focused on optimizing their existing product that they missed a massive platform shift. A smaller, nimbler competitor came out of nowhere and ate their lunch.
The Repentance:
Stay paranoid. Assume that your competitors are smarter than you and are working harder than you. Assume that everything you think you know is wrong. Constantly question your own assumptions.
And most importantly, never fall in love with your own product. Be passionate about the problem you’re solving, not the solution you’ve built. Be willing to tear it all down and start over if that’s what it takes to win.
The Path to Redemption
Avoiding these seven deadly sins is not easy. It requires discipline, humility, and a relentless focus on the user. But it’s the only way to build a product that lasts.
Stop chasing vanity metrics. Stop getting lost in the data. Stop letting A/B tests dictate your strategy. Stop hiding behind the facade of being “data-driven.”
Instead, find your North Star. Talk to your users. Develop a point of view. Be data-informed, not data-driven. Democratize your data. And never, ever believe your own hype.
Do these things, and you won’t just avoid the seven deadly sins of AI product analytics. You’ll be on the path to building a truly great company. '''
Frequently Asked Questions
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.