When we were building RemoteTeam, choosing between multi-armed bandits and a/b testing for nearly killed us before we figured it out.
I’m sharing the playbook we used to launch AI products that millions of people have used. This isn’t theory—it's what worked in real teams at Google and Amazon.
Why Most Approaches Fail
Let me be direct: about 70% of the approaches I see to choosing between multi-armed bandits and a/b testing for are fundamentally flawed. Not slightly off. Fundamentally flawed.
The root cause is usually one of three things:
- Copying what big companies do without understanding why they do it. What works for Google doesn't work for a 10-person startup.
- Over-engineering the solution when a simple approach would work better. I've seen teams spend six months building something that could have been done in two weeks.
- Ignoring the human element. Technology is the easy part. Getting people to actually use it is where the real challenge lives.
What I've Learned From 141 Companies
After investing in 200+ startups and running two companies to successful exits, I've developed a pretty clear picture of what works with choosing between multi-armed bandits and a/b testing for.
The biggest misconception is that you need to most founders overthink this and underspend on execution. That's backwards. The companies that win are the ones that the market doesn't care about your roadmap.
I remember sitting with the Anthropic team early on and discussing how they thought about choosing between multi-armed bandits and a/b testing for. Their approach was counterintuitive but brilliant.
The Reality Nobody Talks About
Most people approach choosing between multi-armed bandits and a/b testing for with assumptions that made sense five years ago. The world has moved on. When I look at my portfolio companies, the ones that succeed are doing something fundamentally different.
The first thing to understand is that the best solutions are often the simplest ones. I've seen this play out across dozens of companies. The pattern is unmistakable.
At RemoteTeam, we learned this the hard way. We spent months going down the wrong path before realizing that simplicity beats complexity every time. Once we made the switch, everything changed.
The Numbers Don't Lie
I've tracked the performance of companies in my portfolio that take choosing between multi-armed bandits and a/b testing for seriously versus those that don't. The difference is stark.
Companies that invest early in choosing between multi-armed bandits and a/b testing for see, on average, 2-3x better outcomes within 18 months. That's not a small edge. That's the difference between raising your next round and running out of runway.
One of my portfolio companies went from struggling to profitable in under a year after they finally got serious about this. The founder told me later that they wished they'd started sooner.
This connects to broader themes around a/b testing ai, data science, experimentation that I've been thinking about a lot lately.
What's Next
The world of choosing between multi-armed bandits and a/b testing for is moving fast. What worked last year might not work next year. That's both the challenge and the opportunity.
My advice: stay curious, stay humble, and stay close to the people who are actually doing the work. Read less thought leadership and do more experiments. Talk to fewer consultants and more practitioners.
And if you're a founder building in this space, remember that the best time to get choosing between multi-armed bandits and a/b testing for right is before you need to. Don't wait for a crisis to force your hand.
I'll keep sharing what I learn. This stuff matters too much to keep to myself.
Frequently Asked Questions
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.