My first images from Stable Diffusion XL were garbage. Absolute trash. I’m talking about seven-fingered hands, distorted faces that looked like they belonged in a Picasso painting gone wrong, and landscapes that seemed to melt into a bizarre digital soup. For a week, I felt like I was going backwards. Here I was, an investor in some of the biggest AI companies on the planet—Anthropic, OpenAI, you name it—and I couldn't generate a decent-looking cat.
It was frustrating. Deeply frustrating. I had seen the incredible, photorealistic images popping up on X (formerly Twitter) and Reddit, all credited to SDXL. People were creating entire worlds, cinematic scenes, and professional-grade product mockups. And me? I was making nightmare fuel. I was on the verge of just giving up and going back to Midjourney, which, let's be honest, is incredibly good at making beautiful images with minimal effort. It’s the Apple of image generation. It just works.
But something bugged me. The open-source nature of Stable Diffusion, the raw power and control it promised, felt like the early days of the internet. It was chaotic, messy, and unapologetically technical. It reminded me of my time building MovieLaLa and RemoteTeam. We had this grand vision, but the first versions of our products were clunky. They barely worked. There were so many moments where quitting seemed like the most logical option. But we didn't, and that persistence is what led to two successful exits. I decided to give SDXL one last shot, treating it not as a simple tool, but as a complex system to be understood.
The Midjourney Mindset is a Trap
My first mistake was treating SDXL like Midjourney. With Midjourney, you can write a short, poetic phrase, add --ar 16:9 --style raw, and get something stunning. It’s designed to be an artistic partner; it fills in the gaps with its own heavily trained aesthetic. I was throwing prompts like “a lone astronaut gazing at a neon-drenched alien city, cinematic, moody” at SDXL and getting… well, mush.
The breakthrough came when I stopped thinking like an art director and started thinking like a programmer. SDXL doesn't want poetry. It wants explicit instructions. It’s not your creative partner; it’s your incredibly powerful, literal-minded, and slightly dumb intern. You have to tell it exactly what to do, and just as importantly, what not to do.
My entire approach to prompting was wrong. I was being too vague, too artistic. I was hoping the model would read my mind. It didn’t. It won’t.
The Breakthrough: From Vague Ideas to Concrete Instructions
The “aha” moment wasn’t a single, magical trick. It was a fundamental shift in my process. It boiled down to a few key changes that, when combined, took my results from amateur hour to professional grade. It was the realization that the prompt is not a suggestion box, it's a specification document.
Here’s the core of it: Structure, Specificity, and Negative Space.
I started breaking my prompts down into a logical structure, almost like a code block:
- Subject & Composition: What is the main focus? What is it doing? Where is it in the frame? Be painfully specific. Not just “a man,” but “a full-body photo of a 40-year-old male entrepreneur with short brown hair and a beard.”
- Style & Medium: Is it a photograph? A digital painting? A watercolor? What kind of camera and lens are you using? “A photograph taken with a Sony a7 IV, 85mm f/1.8 lens” is a thousand times better than “photorealistic.”
- Lighting & Atmosphere: How is the scene lit? Is it “soft morning light,” “dramatic chiaroscuro lighting,” or “harsh neon backlighting”? This has a massive impact on the mood.
- Quality Keywords: This is where you add terms like “ultra-detailed,” “sharp focus,” “8k uhd.” I used to lead with these, but they work best as reinforcers at the end.
But the real secret, the part that truly changed everything, was mastering the negative prompt. I had been ignoring it or just throwing in “ugly, blurry, bad art.” That’s like telling a developer to “avoid bugs.” It’s useless.
My negative prompts became just as detailed as my positive ones. I started explicitly listing out everything I didn’t want to see. The default SDXL model has its quirks, and you need to fight them directly.
My standard negative prompt now looks something like this:
(deformed, distorted, disfigured:1.3)poorly drawn, bad anatomy, wrong anatomyextra limb, missing limb, floating limbsdisconnected limbs, mutation, mutated, ugly, disgustingamputee, blurry, jpeg artifacts, signature, watermark, username
This isn’t just a list of bad things; it’s a targeted attack on the model’s most common failure modes. The (word:1.3) syntax increases the weight on that term, telling the model to really pay attention and avoid it.
My First Success Story
I remember the first image that made me say “whoa.” I wanted to create a portrait of a wise, old investor, someone who has seen it all. My old prompt would have been: “portrait of an old investor, wise, detailed, photorealistic.” The result was a generic, waxy-looking old man.
Here’s the new prompt that changed the game:
Prompt:
close-up photo of a wise 75-year-old angel investor, kind eyes, intricate wrinkles, wearing a tailored tweed jacket. Sharp focus, detailed skin texture. Photographed with a Leica M11, 50mm Noctilux lens, f/1.2. Soft, natural window lighting.
Negative Prompt:
(deformed, plastic, cgi:1.3), cartoon, 3d, (disfigured), (bad art), (b&w), blurry, grainy
And there he was. The image that came back was breathtaking. You could see the texture of the tweed, the subtle reflections in his eyes, the history in the lines on his face. It wasn’t just a picture of an old man; it was a story. It was the result of giving precise, unambiguous instructions. I had finally cracked it.
This experience was a powerful reminder of a lesson I've learned over and over again in my career, from my first startup to my 200+ angel investments: the tools with the steepest learning curves often have the highest ceilings. Anyone can get a decent result from an easy-to-use tool. But the top 1%—the people who truly innovate and create groundbreaking work—are the ones willing to go deep, to struggle through the initial frustration, and to master the complex systems that others abandon.
It’s not about finding a magic wand. It’s about learning how the machine thinks. Whether that machine is a piece of software, a market, or a team of people, the principle is the same. Don't just use the tool. Understand it. Bend it to your will. That’s where the real breakthroughs happen. That’s what separates a fleeting hobby from a lasting advantage. So no, I didn't quit. And I'm so glad I didn't.
Beyond the Prompt: The True Power Lies in the Ecosystem
Mastering the prompt was just the first step. The real, mind-blowing potential of Stable Diffusion XL unlocks when you realize it’s not just a model—it’s a platform. It’s the center of a massive, chaotic, and brilliant ecosystem of open-source tools that bolt onto it. This is the part that Midjourney and DALL-E, in their walled gardens, can never replicate.
Think of the base SDXL model as a powerful engine. Tools like ControlNets and LoRAs are the custom transmissions, steering systems, and turbochargers that let you truly drive the car.
ControlNets: Becoming the Director
ControlNets are a revelation. They let you dictate the composition of an image with incredible precision. Before, I was at the mercy of the model’s interpretation of my prompt. Now, I can provide a blueprint.
For example, I can use an OpenPose ControlNet to define the exact pose of a character. I can literally feed it a stick figure, and the model will generate a person in that exact pose. This is invaluable for creating consistent character art or specific action shots. No more hoping the model understands what “a person jumping” looks like; I can show it.
Or I can use a Canny edge ControlNet, which detects the edges from an input image and uses them as a strict guide. You can sketch a rough outline of a room, a product, or a logo, and SDXL will build a detailed, photorealistic image based on your exact sketch. The level of control is staggering. It’s the difference between being a passenger and being the director of your own film.
LoRAs: Your Own Personal AI Model
LoRAs, or Low-Rank Adaptations, are where it gets personal. These are tiny models (often just 10-200MB) that you can train on your own images to teach SDXL a new concept. This could be a specific person’s face, a particular art style, or a product.
This is not just a gimmick; it’s a commercial powerhouse. I recently advised one of my portfolio companies, a startup that sells custom-designed sneakers, to adopt this workflow. They were spending thousands on photoshoots for every new design. I showed them how to create a LoRA of their flagship sneaker model. It took a few hours and a set of about 50 photos.
Once the LoRA was trained, they could generate infinite marketing images. Their sneaker on a mountaintop at sunset. Their sneaker in a futuristic neon city. Their sneaker worn by a dozen different models in different outfits. They could A/B test different backgrounds, styles, and campaigns without a single photoshoot. The cost savings were massive, but more importantly, their creative velocity went through the roof. That’s the power of this ecosystem. You’re not just using a tool; you’re building your own custom creative factory.
The Real Investment Isn’t Just Money
My journey with Stable Diffusion has been a perfect metaphor for my investment philosophy. As an angel investor, I’ve backed over 200 companies, including many in the AI space like Anthropic, OpenAI, Scale AI, and Hugging Face. I don’t invest in the easy, polished solutions. I invest in the foundational platforms with the messy, chaotic, and world-changing potential.
Easy tools give you predictable results. Complex, open platforms give you leverage. They are force multipliers for creativity and innovation. The initial struggle is the price of admission for an asymmetric upside. The skills you build mastering these systems are a competitive advantage that can’t be easily copied.
So if you’re feeling frustrated with a powerful new tool, whether it’s an AI model, a programming language, or a business strategy, I urge you to lean in. The frustration is a sign that you’re at the edge of your capabilities, and that’s where real growth happens. Don’t just look for the magic prompt. Take the time to understand the system, its quirks, its strengths, and its ecosystem.
Don’t quit. Go deeper. The reward isn’t just a better image of a cat. It’s the mastery of a tool that can reshape your creative and professional world. That’s the breakthrough that changed everything for me.
Frequently Asked Questions
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.