I’ve seen it a hundred times. A founder reads a few blog posts on user research, runs a dozen user interviews, and then builds a product that completely misses the mark. Their product is still stuck. Why? Because most of the advice out there is generic garbage, especially when it comes to AI.
This isn’t another one of those guides. This is for founders in the trenches who need to move fast, get real results, and don’t have a massive data science team to back them up. I’m going to tell you what I’ve learned from building two successful companies and investing in over 200 others, including some of the biggest names in AI like Anthropic, OpenAI, and Scale AI.
Just last week, I was talking to a founder who had spent six months and over $500,000 building an AI-powered tool to help marketers write better ad copy. He had done all the “right” things. He had interviewed dozens of marketers, created detailed personas, and even run a survey to validate his idea. The result? A product that nobody wanted. The marketers he was targeting were already using a combination of ChatGPT and their own intuition to write copy. His tool was a solution in search of a problem.
This is the kind of expensive mistake I want to help you avoid.
Your User Research is Probably Useless
Let's be blunt. If you're building an AI product, your traditional user research methods are likely leading you down a dead end. Asking users what they want in an AI-powered feature is like asking someone in the 19th century if they wanted a car. They would have asked for a faster horse.
Traditional user research is great for understanding explicit needs for simple SaaS products. You can ask a user how they manage their invoices, and they can walk you through a clear, linear process. You can then build a tool that automates that process. Simple.
AI is a different beast entirely. You're not just automating a known workflow; you're often creating entirely new capabilities. Your users don't know what's possible, so they can't tell you what they want. You have to show them.
I remember when we were building RemoteTeam, which was later acquired by Gusto. We had an idea for an AI-powered feature that would predict employee churn. We could have run a survey asking managers if they wanted a churn prediction tool. Of course, they would have said yes. But that wouldn't have told us anything useful. It wouldn't have told us how they would use it, what data they would trust, or how they would act on the predictions. The real insights came from observing their existing, messy, and often irrational workflows for trying to keep their best people.
We saw managers relying on gut feelings, hallway conversations, and the occasional one-on-one to gauge employee happiness. We saw them getting blindsided by resignations from their top performers. We saw the pain and the frustration. That’s when we knew we had a real problem to solve. The AI wasn't the solution; it was a tool to help solve the problem we had identified through deep, empathetic observation.
The Concierge MVP: Your Secret Weapon
So if you can't ask users what they want, what do you do? You run a Concierge MVP. This is one of the most powerful and underutilized techniques for AI product development.
The idea is simple: before you write a single line of code, you manually act as the AI. You simulate the output of your magical AI feature to a small group of early users. This does two things:
- It forces you to deeply understand the problem. You can't hide behind a black box algorithm. You have to get your hands dirty and figure out what a useful output actually looks like.
- It gives you incredibly rich feedback. You get to see how users react to the output, what they click on, and where they get confused. It’s a goldmine of qualitative data.
Here’s how you can run your own Concierge MVP:
- Step 1: Identify a small group of users. Find 5-10 people who you think would be a perfect fit for your product. These should be people you have a good relationship with and who are willing to give you honest feedback.
- Step 2: Define the “AI” output. What is the one key thing you want your AI to do? Write it down in a clear, concise sentence. For example, “My AI will generate three personalized email subject lines based on the content of the email.”
- Step 3: Manually generate the output. For each of your users, manually create the output you defined in Step 2. In the email subject line example, you would read the user’s email and then write three subject lines yourself. Yes, it’s manual. That’s the point.
- Step 4: Deliver the output and observe. Send the output to your users and watch how they use it. Do they use one of the subject lines? Do they edit it? Do they ignore it completely? Get on a call with them and ask them to think aloud as they interact with your “AI.”
- Step 5: Iterate. Based on the feedback you receive, refine your output and repeat the process. Keep doing this until you have a high degree of confidence that you’re providing real value.
This process might seem slow and unscalable, but the insights you’ll gain are invaluable. You’ll learn more in a few weeks of running a Concierge MVP than you will in six months of building a product in a vacuum.
The Right Tools for the Job
Once you’ve validated your core assumptions with a Concierge MVP, you can start to think about tools. But don’t just grab the first shiny object you see. You need to be strategic. Here are a few that I’ve seen work well for AI-specific user research, broken down by the job they do:
For Simulating the AI (The Concierge MVP)
- Your Brain: Seriously. The best tool for this is your own ability to think and manually produce the output you expect from your AI. Don't overcomplicate it.
- Spreadsheets (Google Sheets, Excel): Good for organizing data and presenting it to users in a structured way. You can use formulas and scripts to create a surprising amount of automation.
- Typeform/Jotform: Great for creating simple interfaces for your users to input data and for you to deliver the output.
- Figma/Uizard: If your AI has a visual output, you can use these to create mockups and prototypes to show users. You can even use Figma’s interactive components to create a surprisingly realistic simulation of your product.
For Gathering Feedback on Your Concierge MVP
- Zoom/Google Meet: For running remote user sessions. Record everything. And I mean everything. You’ll be surprised at what you pick up on the second or third viewing.
- Loom: Great for users to record their screens and narrate their thoughts as they interact with your mockups. It’s a fantastic way to get asynchronous feedback.
- Dovetail/Notably: For organizing and analyzing your user interview recordings and notes. These tools are great for finding patterns and themes in qualitative data. They use AI to transcribe your interviews and help you tag and organize your findings. This is where AI can be a huge time-saver.
For When You're Ready to Build
Once you have a handle on the problem and you've seen how users react to your simulated AI, then and only then should you start looking at platforms to help you build and test your actual AI features.
- Synthetic Users: This is a fascinating new approach where you can test your ideas on AI-powered synthetic users. It's a great way to get rapid feedback at scale, especially for testing different variations of your UI or copy. It’s not a replacement for talking to real users, but it can be a powerful tool for getting directional feedback quickly.
- UserTesting/Maze: These platforms are good for getting feedback on live prototypes and production-ready features. You can get videos of users interacting with your product and hear their thoughts in real-time. The key is to have a very specific set of tasks you want users to complete. Don’t just ask them to “try out” your product.
- Vercel/Netlify: Not strictly user research tools, but their preview deployment features are invaluable for sharing and testing new AI features with a select group of users before a full-scale launch. You can create a private link to a new feature and share it with your beta testers. This is a great way to get feedback in a real-world environment without exposing your new feature to all of your users.
Don't Boil the Ocean
I’ve seen too many founders get bogged down in analysis paralysis, trying to find the “perfect” user research tool. The truth is, the tool is secondary to the process. Start with the Concierge MVP. Get your hands dirty. Talk to your users. The insights you gain from that process will be far more valuable than any report from a fancy AI-powered analysis tool.
And remember, this is a marathon, not a sprint. You’re not going to get it right on the first try. The key is to build a tight feedback loop where you’re constantly learning from your users and iterating on your product. That’s how you build something people actually want.
So, what are you waiting for? Go find five users and start your Concierge MVP today. I guarantee you’ll learn more in the next week than you have in the last six months. And if you get stuck, feel free to reach out to me on Twitter. I’m always happy to help a fellow founder out.
Frequently Asked Questions
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.