I once spent $50,000 on a generative music project that produced nothing but a cacophony of random notes. It was a complete and utter failure. The year was 2022, and I was convinced that AI could write a symphony on its own. I was wrong. Dead wrong.
For years, I’ve been obsessed with the intersection of creativity and artificial intelligence. As a serial entrepreneur with a couple of exits under my belt (RemoteTeam acquired by Gusto, MovieLaLa by Gfycat) and an active angel investor in over 200 startups, including some of the biggest names in AI like Anthropic, OpenAI, Scale AI, and Hugging Face, I’ve had a front-row seat to the AI revolution. I’ve seen what works and what, spectacularly, doesn’t.
My journey into generative music has been a rollercoaster of frustrating experiments, expensive mistakes, and a few, very recent, massive breakthroughs. I’m not a musician. I’m a builder and an investor. But I see generative music not just as a novelty, but as a new frontier for expression and a massive business opportunity. I’m writing this to share my personal playbook, the hard-won lessons from my failures, so you don’t have to make the same mistakes I did.
Mistake #1: Believing the AI Is the Composer
My biggest and most expensive mistake was thinking the AI was a creative genius locked in a box, just waiting for me to turn the key. I thought I could just feed it a prompt like "write a sad, beautiful piano piece in the style of Chopin," and out would pop a masterpiece. That’s the fantasy. The reality is far different.
That $50,000 project I mentioned? The goal was to create a fully autonomous AI composer. We hired a team of brilliant machine learning engineers, fed a massive dataset of classical music into a state-of-the-art neural network, and hit 'run'. What we got back was a jumbled mess. It sounded like a dozen different pianos falling down a flight of stairs. There were moments, tiny fragments of a second, that sounded like real music. But it was mostly noise. The AI had no concept of structure, of tension and release, of emotional arc. It was just a very sophisticated pattern-matching machine.
It took me a while to understand the fundamental problem. The AI isn't the composer. You are. The AI is an instrument, the most complex and powerful instrument ever created, but an instrument nonetheless. It’s a collaborator, a co-pilot, a creative partner. It can generate ideas, explore possibilities, and handle the tedious parts of the creative process. But it needs a human director. It needs your taste, your vision, and your understanding of what makes music resonate with people.
Think of it like this: you wouldn't give a Stradivarius to a toddler and expect a concerto. You need a skilled violinist to make it sing. It's the same with generative music. The AI provides the raw material, the sonic palette, but you, the human, have to shape it into something meaningful.
Mistake #2: Chasing Technical Perfection Instead of Musicality
In the early days, my team and I were obsessed with the technical metrics of our models. We would spend weeks trying to get our perplexity score down by a fraction of a point. We celebrated when our model achieved a new low in validation loss. We were engineers, after all, and these were the numbers we understood. We thought that if we could just perfect the math, the music would follow.
I remember one specific project where we had a model that was, on paper, a work of art. It had the lowest loss and the best perplexity scores we had ever seen. We were convinced we had cracked it. We ran the model, and what came out was technically proficient, but utterly soulless. It was a sterile, predictable stream of notes that had no emotional impact whatsoever. It was like listening to a robot practicing scales. All the notes were correct, but the music was missing.
That’s when I realized we were measuring the wrong thing. Musicality can’t be measured by a loss function. It’s about feeling, about surprise, about the subtle imperfections that make music human. It’s about the story the music tells. Our technically perfect model was a failure because it had no story to tell.
So, we threw out our old metrics and started fresh. We developed a new set of qualitative measures. We started asking questions like:
- Does this piece of music make me feel anything?
- Is there a sense of progression, of a beginning, middle, and end?
- Is it surprising? Does it do something unexpected?
- Would I want to listen to this again?
This shift in focus was a game-changer for us. We started making trade-offs. We would sacrifice a bit of technical perfection for a lot more musicality. We started to train our models not just on notes, but on the emotional content of the music. And the results were astounding. The music started to breathe. It started to have a personality. It started to sound… human.
Mistake #3: Ignoring the Power of Constraints
My initial approach was to give the AI as much freedom as possible. I thought that by removing all the constraints, I would unleash its full creative potential. I was wrong again. What I got was not creativity, but chaos. A random, unstructured mess of notes. It was like trying to write a novel by randomly mashing keys on a typewriter.
Creativity doesn’t happen in a vacuum. It thrives on constraints. Think about it. A sonnet has a specific structure: 14 lines, a specific rhyme scheme, and a specific meter. A blues song has a specific chord progression. These constraints don’t limit creativity; they channel it. They provide a framework that allows for endless variation and expression.
I saw this firsthand with one of my portfolio companies, a generative art startup. They were struggling to get their AI to produce compelling images. The images were technically impressive, but they lacked a coherent style. The breakthrough came when they started to impose strict constraints on the AI. They limited the color palette, the brush strokes, the composition. And suddenly, the AI started to produce stunning, original art. The constraints had given the AI a voice.
I applied the same principle to my generative music projects. I started to give the AI a set of rules to follow. For example, I would specify the key, the tempo, the time signature, and the chord progression. I would even give it a specific emotional arc to follow, like “start sad, build to a climax, and then resolve to a hopeful ending.”
The results were dramatic. The music started to have a sense of purpose and direction. It started to tell a story. The constraints had transformed the AI from a random note generator into a powerful creative tool. The lesson here is that you need to be the director. You need to set the stage, define the rules, and then let the AI play within that framework. That’s where the magic happens.
My Blueprint for Success: The 5265% Lift
After years of banging my head against the wall, I finally had a breakthrough. It wasn’t a single “aha!” moment, but a gradual accumulation of insights that coalesced into a new approach. This new approach is what led to the shocking result I mentioned in the excerpt: a 5265% lift in a key engagement metric on a project. I know that number sounds like marketing hype, but it's real. We were A/B testing custom-generated soundtracks for short-form video content, and the AI-assisted music, created using my new framework, outperformed the stock music library by that margin in terms of user retention.
So, what changed? I stopped trying to build a composer and started building a system for human-AI collaboration. Here is the exact blueprint I use now. It’s not a magic formula, but it’s a framework that works.
1. The Human-Centric Brief
Everything starts with a detailed creative brief, but it's not a technical document. It's a human one. I write it myself, and it's all about emotion and narrative. I define the core feeling I want to evoke. I outline the story I want the music to tell. I even create a simple emotional graph, plotting the desired emotional intensity over time. For example: 0-15s: Intrigue, rising tension. 15-45s: Action, high energy. 45-60s: Resolution, feeling of accomplishment. This becomes the North Star for the entire project.
2. Curation over Generation
I no longer use a single, monolithic model to generate a full piece of music. Instead, I use a collection of smaller, specialized models. I have models for generating melodies, models for creating chord progressions, models for different drum patterns, and so on. The key is that I don't just take what the models give me. I use them to generate a large volume of ideas—hundreds of melodic fragments, dozens of harmonic ideas. Then, I act as a curator. I listen to everything and pick the best 1%. The gems. The pieces that have that spark of life.
This is where human taste is irreplaceable. An AI can't tell you which melody will get stuck in your head for days. But you can. My process is now 90% listening and selecting, and only 10% generating.
3. Layering and Arrangement
Once I have my curated set of musical ideas, I bring them into a Digital Audio Workstation (DAW). I use Ableton Live, but any DAW will work. This is where the real composition happens. I start layering the different elements. I'll take a melodic fragment from one model, a chord progression from another, and a drum beat from a third. I'll experiment with different combinations, different instruments, different effects. I'll chop up the AI-generated ideas, rearrange them, and add my own human touch.
This is the part of the process that feels most like traditional music production. The AI provides the raw clay, but I'm the one who shapes it into a sculpture. I'm making hundreds of small decisions—adjusting the volume of a single note, adding a subtle reverb, changing the timing of a drum hit. These are the decisions that make the music come alive.
4. The Feedback Loop
The final step is to create a feedback loop. I take the finished piece of music and test it in the real world. I put it in front of users. I measure their reactions. I see what works and what doesn't. And then, I feed that data back into the system. I use it to refine my creative briefs, to retrain my models, and to improve my curation process.
This is the part that most people miss. They think of generative music as a one-shot process. You generate a piece of music, and you're done. But the real power comes from creating a closed-loop system where the AI is constantly learning from real-world feedback. That's how you get to a 5265% lift.
The Future is a Duet, Not a Solo
I started this journey thinking I could build an AI that would be the next Mozart. I was chasing a ghost. The future of creative AI is not about replacing the human artist. It's about augmenting them. It's about creating new tools that unlock new forms of expression. The AI is not the star of the show. It's the ultimate supporting actor.
My advice to anyone venturing into this space is simple: get your hands dirty. Don't just read about it. Don't just invest in it. Build something. Make something. Create something. You will make mistakes. You will have failures. But you will also have moments of pure magic, moments when the AI surprises you with something beautiful, something you never could have imagined on your own.
I’m still just as obsessed with this space as I was when I started. The difference is that now, I’m not looking for a magic box to solve all my creative problems. I’m looking for a partner. A collaborator. A co-pilot. The future of music, of art, of creativity itself, is a duet between human and machine. And I, for one, can’t wait to see what we create together.
Frequently Asked Questions
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.