I once had a dashboard that told me we were killing it. User engagement was through the roof. Daily actives were climbing. I felt like a genius. We even celebrated with the team, popping a bottle of champagne I’d been saving. Then I dug into the raw data, and my heart sank. Turns out, a bug was creating thousands of phantom users, and our “engagement” was just a script looping endlessly. The dashboard wasn’t just wrong; it was lying to my face. That champagne tasted bitter in retrospect.
That was my first lesson in the brutal reality of AI analytics. After founding three startups, two of which were acquired (RemoteTeam by Gusto, MovieLaLa by Gfycat), and angel investing in over 200 companies like Anthropic and Scale AI, I’ve seen this story play out dozens of times. I’ve analyzed over 10 million data points, and I’m here to tell you that most of what you’ve been told about AI data is a fantasy.
Here are the five brutal truths I learned along the way.
1. Your Dashboard is a Vanity Mirror
Most AI dashboards are designed to make you feel good. They’re packed with vanity metrics that look impressive but mean nothing. Daily active users, sign-ups, page views—these numbers are easy to track and easy to manipulate. They don’t tell you if your users are actually getting value from your product.
At MovieLaLa, we had a dashboard that showed a beautiful, upward-sloping curve of user growth. We were high-fiving each other every week. But when we looked closer, we saw that our retention was terrible. Users would sign up, poke around for a few minutes, and then disappear forever. The dashboard was showing us a reflection of our own wishful thinking, not the health of our business.
We had to scrap the whole thing and start over. We built a new dashboard focused on one metric: the number of users who rated at least 10 movies in their first week. That number was much smaller, and it didn’t look nearly as impressive. But it was real. It was a measure of true engagement, and it forced us to confront the fact that our product wasn’t as sticky as we thought. This new metric became our North Star. We rallied the entire team around it. Product discussions started with 'Will this move our 10-movie-rating number?' It changed how we built features and how we measured success. It was painful, but it was necessary.
2. Predictive Analytics is Mostly BS
I’ve seen so many founders get seduced by the promise of predictive analytics. They think they can just feed a bunch of historical data into a machine learning model and it will spit out the future. It’s a tempting idea. It’s also a load of crap.
The problem is that the future doesn’t look like the past. Black swan events happen. Markets shift. Your competitors launch new products. A model trained on last year’s data is going to be useless when the world changes.
I learned this the hard way at RemoteTeam. We built a predictive model to identify customers who were likely to churn. It worked beautifully—for a while. Then the pandemic hit. Suddenly, our model was completely wrong. Customers we thought were safe were churning left and right, and customers we thought were at risk were sticking around. The model was blind to the new reality.
We had to go back to basics. We started talking to our customers. We ran dozens of interviews. We asked them what they were struggling with and how we could help. We learned more from those conversations than we ever did from our predictive model. One customer told us, 'I don't care about your predictions, I care about my team feeling connected.' That single quote was more valuable than a thousand data points. It led us to build a new set of features focused on team morale and connection, which became a huge differentiator for us. The real insights are in the qualitative data, not just the quantitative.
3. "Big Data" is a Trap
Everyone wants to talk about big data. They’re proud of their terabytes and petabytes. But here’s the secret: most of that data is noise. It’s a distraction. The real gold is in the small, weird data points that don’t fit the pattern.
I once found a bug in our payment system by looking at a single customer support ticket. The customer was complaining about a weird error message. It was a one-off thing, and we could have easily ignored it. But I had a hunch. I dug into the logs and found that the error was happening to a small number of users, but it was preventing them from upgrading their accounts. We were leaving money on the table.
Fixing that bug was a huge win for us. It increased our revenue by 5% overnight. And we never would have found it if we were just looking at the big picture. You have to be willing to get your hands dirty and dig into the messy details. The insights are in the outliers, not the averages. I tell my teams to spend 20% of their data analysis time just looking for weird stuff. Things that don't make sense. That's where the gold is.
4. Your Team Hates Your Dashboards
Have you ever seen someone’s eyes glaze over when you show them a dashboard? That’s because most dashboards are terrible. They’re cluttered, confusing, and they don’t help people make decisions. They’re a form of corporate art, not a tool for action.
I used to be guilty of this. I would create these elaborate dashboards with dozens of charts and graphs. I thought I was being helpful. But my team was just overwhelmed. They didn’t know what to focus on, so they ignored the whole thing.
Now, I have a new rule: every chart on a dashboard has to have a clear “so what.” What is the key insight, and what should we do about it? If you can’t answer that question, the chart doesn’t belong on the dashboard.
We also started creating different dashboards for different teams. The marketing team needs to see different data than the product team. The CEO needs to see a high-level overview, not the nitty-gritty details. It’s more work to create custom dashboards, but it’s worth it. A dashboard that nobody uses is just a waste of server space. We even created a 'dashboard of the month' award to encourage teams to build useful and insightful dashboards. It became a point of pride for people to have their dashboard featured.
5. The Real ROI is in Augmenting People, Not Replacing Them
There’s a lot of fear-mongering about AI taking over jobs. But that’s not what I’ve seen. The real power of AI is in making people smarter and more effective. It’s a tool for augmentation, not automation.
At one of my startups, we built an AI tool to help our sales team identify the most promising leads. The tool would analyze a bunch of data points—company size, industry, website traffic—and then assign a score to each lead. It was a huge time-saver for the sales team. They could focus their energy on the leads that were most likely to convert.
But the tool wasn’t perfect. It would sometimes miss a good lead or flag a bad one. That’s where the human element came in. The sales team learned to use the tool as a starting point, not a final answer. They would use their own judgment and experience to decide which leads to pursue. The combination of AI and human intelligence was much more powerful than either one on its own. The sales team's conversion rate went up by 30% after we introduced the tool. But it wasn't because the AI was doing all the work. It was because the AI was freeing up the sales team to do what they do best: build relationships and close deals.
Stop Chasing Shiny Objects
It’s easy to get distracted by the latest AI buzzwords. But the truth is, the fundamentals haven’t changed. You still need to understand your customers, build a great product, and make smart decisions. AI can help with all of those things, but it’s not a magic bullet.
So, stop chasing shiny objects. Stop trying to build the perfect, all-knowing dashboard. Start by asking the right questions. What are the most important metrics for your business? What are the biggest challenges your customers are facing? And how can you use data to help your team make better decisions?
The answers to those questions are worth more than all the petabytes in the world. So, go ahead, build your dashboards. But don't let them become a vanity mirror. Use them to ask hard questions, to challenge your assumptions, and to get closer to the truth. That's the real secret to building a data-driven company. It’s not about having the most data; it’s about having the right conversations, sparked by the right data. Now, go have those conversations.
Frequently Asked Questions
Which item on this list has the highest impact?
It depends on your stage and context, but in my experience, the items near the top of the list tend to have the broadest applicability. That said, sometimes the less obvious items create the biggest breakthroughs for specific situations.
How do I know which items apply to my situation?
Start by honestly assessing where your biggest bottleneck is right now. The items that address that specific constraint will give you the highest return on your time and energy.
Can I implement all of these at once?
I'd strongly recommend against it. Pick the 2-3 items that resonate most with your current situation and focus there. Trying to do everything simultaneously is a recipe for doing nothing well.