We threw away six months of code on a Tuesday afternoon. It was the best decision I ever made.
People always ask me how we built our recommendation engine. They want the clean, sanitized version of the story. They want to hear about brilliant whiteboard sessions and perfectly executed sprints. But that is not how real software gets built. Real software, especially AI products, is messy. It is full of dead ends, late-night panic attacks, and moments where you seriously question your own sanity.
This is the unfiltered, behind-the-scenes look at the blood, sweat, and code that went into building our core AI product. No fluff. Just the reality of leading through uncertainty and managing a team when nobody actually knows what the final product will look like.
The MovieLaLa Days: When Recommendations Were Dumb
Let me take you back to my time building MovieLaLa. Before Gfycat acquired us, we were obsessed with movie trailers. We had millions of users watching clips, and we needed a way to show them what to watch next.
Our first recommendation engine was basically a glorified random number generator. I am not kidding. We used basic collaborative filtering, and it was terrible. If you watched a horror movie, it might recommend a romantic comedy just because someone else in a different time zone happened to click both by accident.
I learned a hard lesson back then. You cannot fake good recommendations. Users know instantly when your algorithm is guessing. They lose trust. And once you lose user trust, your product is dead.
When we started building our new AI-powered engine years later, I remembered that failure. I told the team on day one that we were not going to ship a toy. We were going to build something that actually understood intent.
The Data Problem Nobody Talks About
Everyone wants to talk about neural networks and transformer models. Nobody wants to talk about data pipelines.
Before you can have AI, you need data. Clean, structured, reliable data. When we started this project, our data was a disaster. We had user logs in three different formats. We had missing timestamps. We had duplicate entries that were skewing our baseline metrics.
I spent the first three weeks of the project doing nothing but writing SQL queries and Python scripts to clean up the mess. Yes, the CEO was writing data cleaning scripts. Why? Because if the foundation is rotten, the house collapses.
We had to build a real-time ingestion pipeline that could handle 50,000 events per second. We chose Kafka for the event streaming, and it was a nightmare to configure at first. Our lead engineer, a brilliant guy who had previously worked at a major tech giant, spent four days just tuning the partition settings.
This is what AI change management actually looks like. It is not about giving grand speeches. It is about sitting with your team at 2 AM, staring at Grafana dashboards, trying to figure out why your memory usage is spiking.
Enter the AI Era: Lessons from the Giants
Over the years, I have made more than 200 angel investments. I have been lucky enough to back companies like Anthropic, OpenAI, Scale AI, and Hugging Face early on. Watching these companies operate from the inside gave me a massive advantage.
I saw how OpenAI approached model training. I saw how Scale AI handled data labeling at an unprecedented scale.
The biggest takeaway? Compute is cheap, but human intuition is expensive.
When we designed our recommendation engine, we did not try to build a massive foundational model from scratch. That would be stupid. Instead, we focused on fine-tuning existing models on our highly specific, proprietary dataset. We used embeddings to represent user behavior and content features in a high-dimensional space.
We calculated cosine similarity between user vectors and content vectors. It sounds simple in theory. In practice, doing this for millions of users in under 50 milliseconds requires serious engineering.
Building the Core Engine: Blood, Sweat, and Python
Let us talk about the actual build. We started with a monolithic architecture. Big mistake.
Within two months, our training jobs were taking 48 hours to complete. If a developer made a mistake in the code, we would not know until two days later. The feedback loop was too slow. Innovation dies when feedback loops are slow.
I made the call to rip it all apart. That was the Tuesday afternoon I mentioned earlier. We deleted six months of monolithic code and moved to a microservices architecture.
We separated the data ingestion, the model training, and the inference engine. We containerized everything with Docker and orchestrated it with Kubernetes.
The inference engine was written in Go for speed, while the training pipelines stayed in Python. This hybrid approach saved us. Go gave us the sub-millisecond latency we needed for real-time recommendations, and Python gave our data scientists the ecosystem they needed to iterate on the models.
Was it painful? Absolutely. Half the team thought I was crazy for throwing away working code. But as a leader, you have to know when to cut your losses. Sunk cost fallacy kills more startups than bad ideas do.
The RemoteTeam Advantage: Scaling the Humans Behind the AI
You cannot build a world-class AI product with a mediocre team. And you cannot build a world-class team if you only hire within a 30-mile radius of your office.
When I built RemoteTeam, which was later acquired by Gusto, I learned exactly how to manage distributed talent. For this recommendation engine, our team was spread across 12 different time zones.
We had a machine learning expert in Ukraine, a backend architect in Brazil, and data engineers in India.
Managing this kind of team requires a completely different set of AI leadership skills. You cannot rely on shoulder-tapping. You cannot rely on watercooler conversations. Everything must be documented.
We used asynchronous communication for 90% of our work. We recorded Loom videos to explain complex architectural changes. We wrote detailed design docs before writing a single line of code.
This forced clarity. When you have to write down exactly how the embedding layer will interact with the caching layer, you catch design flaws before they become production bugs.
Leadership in the Trenches: Making the Hard Calls
There was a moment, about four months into the rebuild, where everything broke.
We pushed a new version of the model to production, and our click-through rate dropped by 40%. Panic set in. The board was asking questions. The sales team was freaking out because user engagement was tied directly to our revenue.
The natural instinct in that situation is to roll back immediately. But I looked at the data. The new model was actually making better recommendations, but it was surfacing content that users were not used to seeing. It was breaking their filter bubbles.
I made the decision to keep the new model live. I told the team to hold the line.
We spent the next 72 hours tweaking the UI to better explain why certain content was being recommended. We added a simple "Because you watched X" label.
Within a week, the click-through rate not only recovered but surpassed our previous baseline by 25%.
That is AI transformation leadership. It is having the conviction to trust the math, even when the short-term metrics look terrifying. You have to protect your engineering team from the noise of the rest of the company.
The Real Cost of Scaling
Scaling an AI product is not like scaling a traditional web app. When a web app gets more traffic, you just spin up more servers. When an AI engine gets more traffic, the complexity multiplies exponentially.
We hit a wall at 1 million active users. Our vector database started choking. We were using an off-the-shelf solution, and it simply could not handle the read volume.
We had to build a custom caching layer using Redis. We pre-computed recommendations for our most active users during off-peak hours and stored them in memory. For the long-tail users, we did real-time inference.
This hybrid approach reduced our cloud bill by $40,000 a month and dropped our average latency from 120ms to 35ms.
But the real cost of scaling is not just server bills. It is the mental toll on the team. Burnout is a massive risk when you are pushing the boundaries of what is technically possible. As a founder, you have to force your team to take breaks. I literally locked developers out of our GitHub repository on weekends to force them to rest. You cannot write good code when you are running on three hours of sleep and Red Bull.
The Philosophy of Becoming Top 1%
In my book, "Becoming Top 1%", I talk about the concept of relentless iteration. Building this recommendation engine was the ultimate test of that philosophy.
You do not become the top 1% by having a perfect plan. You become the top 1% by iterating faster than anyone else.
Every time our model failed, we learned something. Every time a server crashed, we built a more resilient system. We treated failure as data.
When you are leading a team through uncertainty, your job is not to have all the answers. Your job is to build a system where the team can find the answers quickly and safely.
We set up shadow testing environments where we could run new models against live production traffic without actually showing the results to users. This allowed us to measure the impact of a change with zero risk. It gave our data scientists the confidence to try wild, crazy ideas. Most of them failed. But the ones that succeeded changed the trajectory of the company.
The Final Polish: Why UX Matters in AI
You can have the most sophisticated neural network in the world, but if the user interface is clunky, nobody will care.
We spent the last two months of the project obsessing over the UI. We realized that users do not just want recommendations; they want context. They want to know why the AI is suggesting a specific piece of content.
We implemented a transparency feature. Users could click a button and see exactly which of their past actions influenced a specific recommendation. It was a massive technical challenge to trace the weights back through the model in real-time, but it was worth it.
User trust skyrocketed. People started treating the recommendation engine less like a black box and more like a trusted advisor.
What Actually Matters
Looking back at the entire journey, the code we wrote is already becoming obsolete. The models we trained will be replaced by better, faster models in a year.
What actually matters is the muscle memory we built as a team. We learned how to tackle impossible problems. We learned how to argue productively. We learned how to ship complex AI products without losing our minds.
Building a recommendation engine is hard. Building a team that can build a recommendation engine is much harder.
If you are embarking on a similar journey, my advice is simple. Get your data right first. Hire people who are comfortable with ambiguity. And do not be afraid to delete six months of code on a Tuesday afternoon if it means building something better.
Frequently Asked Questions
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.