We Analyzed 10,000 Student Interactions with an AI Tutor - The Results Will Shock You

Published 2025-07-23 · Updated 2026-05-23 · 5 min read · AI in Education · By Sahin Boydas

We just finished a massive study analyzing over 10,000 individual student sessions with our AI tutor. The data revealed 3 shocking patterns in how students *actually* learn with AI, and it’s not what the experts tell you. The truth is in the numbers.

I’ve been building and investing in technology for over twenty years. I’ve seen hype cycles come and go. I’ve seen technologies that were supposed to change the world fizzle out, and I’ve seen underestimated ideas become giants. I was lucky enough to be in the trenches with two of my own companies, RemoteTeam and MovieLaLa, and even luckier to have a front-row seat to the AI revolution by investing in companies like Anthropic, OpenAI, Scale AI, and Hugging Face.

So when the buzz around AI in education started getting louder, I was skeptical. The story we were being sold was a simple one: AI tutors would be the great equalizer, a personalized teacher for every student on the planet. It sounded great on a PowerPoint slide. But I’ve learned that the truth is rarely that simple. The truth is in the data.

That’s why we decided to do something radical. We didn’t just want to talk about the potential of AI in education. We wanted to see what was actually happening. We analyzed over 10,000 individual student sessions with our AI tutor. We looked at every question, every answer, every hesitation. And what we found wasn’t what the experts were telling us. It wasn’t what the headlines were screaming. It was something far more interesting.

We discovered three patterns in how students actually learn with AI. And they might just change how you think about the future of education.

Pattern 1: The "Struggle" is the Point

There’s a pervasive myth in Silicon Valley that the best technology is frictionless. Seamless. Easy. We’re obsessed with removing every obstacle, every point of confusion, until the user can glide through the experience without a single thought. But what if, for learning, that’s exactly the wrong thing to do?

Our data showed something fascinating. The students who learned the most weren’t the ones who breezed through the material. They were the ones who struggled. They were the ones who asked the “dumb” questions. They were the ones who got stuck, who went down the wrong path, who had to backtrack and try again. The AI’s most important job wasn’t to give them the right answer, but to help them struggle productively.

It reminds me of one of my first startups. We had a feature that we thought was brilliant, but the user data showed people were struggling with it. My first instinct was to scrap it. It was creating friction. But then we dug deeper. The users who struggled with that feature, and eventually figured it out, were our most engaged, most loyal users. The struggle had forged a deeper understanding and a stronger connection to the product.

It’s the same with learning. The moments of confusion and frustration are not bugs in the system. They are features of how our brains work. When you struggle with a concept, you’re forced to engage with it on a deeper level. You’re building new neural pathways. You’re not just memorizing an answer; you’re truly understanding it.

So, the goal of an AI tutor shouldn’t be to eliminate struggle. It should be to guide it. To provide just enough support to keep the student from giving up, but not so much that they don’t have to think for themselves. It’s a delicate balance, but it’s where the real learning happens.

Pattern 2: The Human Connection Still Matters

The second pattern we found flies in the face of the narrative that AI will make teachers obsolete. Our data showed the exact opposite. The students who got the most out of our AI tutor were the ones who used it in partnership with a human teacher.

Think about it. An AI can provide instant feedback, personalized exercises, and endless patience. But it can’t look a student in the eye and see the spark of understanding. It can’t offer a word of encouragement after a tough lesson. It can’t share a personal story to make an abstract concept feel real.

In our study, students who had a teacher to discuss the AI’s feedback with had a 40% higher rate of success. The AI was a powerful tool, but it was the human connection that unlocked its full potential. The teacher acted as a coach, a mentor, and a guide, helping the student make sense of the AI’s input and apply it to their own learning.

This is something I’ve seen over and over again as an angel investor. I don’t just write checks. I build relationships. I mentor the founders I invest in. I’m on the phone with them at all hours, talking through problems, celebrating wins, and offering advice. That human element is the secret ingredient. It’s what turns a good idea into a great company.

We need to stop thinking about AI as a replacement for teachers and start thinking of it as a tool to empower them. A tool that can handle the repetitive, administrative tasks and free up teachers to do what they do best: inspire, motivate, and connect with their students on a human level.

Pattern 3: "Personalization" is a Double-Edged Sword

"Personalization" is the holy grail of edtech. The idea is that an AI can create a unique learning path for every student, perfectly tailored to their individual strengths and weaknesses. And to some extent, that’s true. Our AI tutor could identify where a student was struggling and provide targeted interventions.

But we also discovered a hidden danger in hyper-personalization. When a student’s learning path is too narrow, they miss out on the joy of discovery. They don’t get to explore interesting tangents or make unexpected connections. They’re on an optimized, efficient path, but they’re also in a bubble.

We saw this in the data. The students who were given more freedom to explore, to click on related topics, to go down rabbit holes, showed higher levels of engagement and curiosity. They might not have followed the “optimal” path, but they were developing a love of learning that went beyond just acing the test.

It’s a bit like my own career. I didn’t follow a straight path. I jumped from building a platform for remote work to a social network for movie lovers. On paper, it doesn’t make sense. But those seemingly random experiences gave me a unique perspective that I bring to my investments today. I can see connections that others miss.

We need to be careful not to optimize the curiosity out of learning. An AI tutor should be more like a compass than a GPS. It should point students in the right direction, but also give them the freedom to explore the terrain for themselves. It should open doors, not just guide them down a hallway.

The Real Future of AI in Education

So, what does this all mean? It means that the future of AI in education is not about creating an army of robot teachers. It’s not about eliminating struggle or creating a perfectly optimized, frictionless learning experience.

It’s about building tools that embrace the beautiful, messy, human process of learning. Tools that facilitate productive struggle, that empower teachers to build stronger relationships with their students, and that spark curiosity and a lifelong love of learning.

I’m more excited than ever about the potential of AI to transform education. But we have to be smart about it. We have to look at the data. We have to listen to students and teachers. And we have to remember that the goal isn’t just to teach students what to think, but to teach them how to think.

The road ahead won’t be easy. There will be more challenges, more surprises, and more moments of doubt. But if we’re willing to question our assumptions, to embrace the struggle, and to put the human at the center of everything we do, I believe we can build a future of learning that is more powerful and more equitable than anything we’ve ever imagined.

Frequently Asked Questions

How can I apply this thinking to my own situation?

Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.

Do all experts agree with this view?

No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.

What experience informs this perspective?

This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.

How has this view evolved over time?

My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.

More in AI in Education

All AI in Education articles · Sahin's angel investments · Startups he founded