I’ve been lucky enough to be in on the ground floor of the AI revolution. As an angel investor, I’ve written checks to over 180 AI startups, including some of the names you see in the headlines every day like Anthropic, OpenAI, Scale AI, and Hugging Face. I even wrote a book on how to build a career in this space. You’d think I’d be pretty good at using the technology I’m investing in.
But for a long time, my Stable Diffusion results were just… mediocre. I’d see these incredible, photorealistic, mind-bending images on social media, and then I’d go to my own computer and generate a bunch of six-fingered aliens. It was frustrating. It felt like I was just guessing, throwing spaghetti at the wall to see what stuck. As an engineer and an entrepreneur who has built and sold two companies—RemoteTeam to Gusto and MovieLaLa to Gfycat—I can tell you, I hate guessing.
So I decided to do what I always do when I’m faced with a problem I don’t understand: I decided to build a system. I set out to analyze a massive dataset of 50,000 Stable Diffusion XL generations to figure out what actually works. I wanted to replace the cargo-cult advice and the endless lists of "magic words" with cold, hard data. What I found didn't just surprise me; it changed my entire approach to prompting.
The Data Doesn't Lie
Before we get into the weeds, a quick word on how I did this. I pulled together a dataset of 50,000 images and their corresponding prompts. Then, I used a CLIP-based model to score each image for aesthetic quality and how well it matched its prompt. This gave me a ranked list from "absolute garbage" to "Mona Lisa-level masterpiece." With this ranked data, I could finally run some real statistical analysis to see which parts of a prompt had the biggest impact on the final image. It’s the same data-first mindset I used when scaling my companies. You can’t improve what you don’t measure.
What I discovered were three surprising patterns that flew in the face of most of the common wisdom I’d seen online.
Surprising Pattern #1: "Magic Words" Are a Total Myth
You’ve seen the lists. The prompts stuffed with words like "masterpiece," "hyperrealistic," "8k," "award-winning," "trending on ArtStation." The thinking is that by adding these words, you’re telling the model to aim for a higher standard of quality. That makes intuitive sense. And it’s completely wrong.
My analysis showed that these so-called "magic words" have almost zero correlation with high-quality images. In fact, in some cases, they seemed to add noise and make the output worse. Why? My theory is that these models have been trained on such a vast ocean of internet data that these words have become functionally meaningless. They’ve seen "masterpiece" applied to everything from a Rembrandt to a blurry photo of someone’s cat. The signal has been lost in the noise.
Think about it. If every other person is telling the model to create a "masterpiece," the word ceases to have any descriptive power. It’s like walking into a restaurant and telling the waiter you want "delicious food." It’s not helpful. You’re better off describing the flavors, ingredients, and style of cuisine you’re in the mood for.
My advice? Stop wasting your tokens. Be direct. Be descriptive. Tell the model what you want to see, not how you want to feel about it.
Surprising Pattern #2: Negative Prompts Are a Crutch, Not a Solution
This one might be controversial. Negative prompts (the [bad-image-v2] or (deformed, ugly, blurry:1.5) part of the prompt) are often seen as essential for cleaning up AI-generated images. And they can be useful, in moderation. But the data shows that the best images rarely rely on them.
In my dataset of 50,000 images, I found that less than 15% of the top-scoring images used more than a handful of negative keywords. The truly stunning images, the ones that made me stop and say "wow," were almost always the result of a powerful, well-crafted positive prompt, not a long list of things to avoid.
Overusing negative prompts can actively harm your image. It can confuse the model, creating bizarre artifacts or causing it to misinterpret your core subject. It’s like trying to drive a car by only telling it where not to go. You might avoid a few ditches, but you’re not going to get to your destination efficiently.
A great positive prompt is ten times more effective than a great negative prompt. Focus your energy on describing what you want to create. If you find yourself with a negative prompt that’s twenty words long, it’s a sign that your positive prompt isn’t doing its job.
Surprising Pattern #3: Specificity Is Everything (Especially with Artists)
So if magic words are a myth and negative prompts are a crutch, what actually works? The single most impactful factor I found was style. And the most powerful way to define a style is by referencing specific artists, art movements, or even film directors.
This is where you can really start to control the output. The difference between a prompt that says "a fantasy castle" and one that says "a fantasy castle in the style of Hayao Miyazaki" is night and day. The first is a generic lottery. The second is a specific instruction that gives the model a rich visual library to draw from.
Don’t just say "sci-fi." Say "in the style of a 1970s sci-fi book cover by Chris Foss." Don’t just say "a portrait of a woman." Say "a portrait of a woman, oil on canvas, in the style of John Singer Sargent."
This is your secret weapon. To get you started, here are a few less-common artists whose styles produce incredible results in SDXL:
- Zdzisław Beksiński: For surreal, dystopian, and hauntingly beautiful landscapes.
- Eyvind Earle: For magical, stylized forests and landscapes with a unique color palette.
- J.C. Leyendecker: For elegant, painterly figures with a distinct Art Deco feel.
- Frank Frazetta: For dynamic, heroic fantasy art with a raw energy.
Experiment. Find artists you love. The more specific your reference, the more unique and compelling your image will be.
My New Framework: The "Director's Method"
After staring at data for weeks, I realized that the best prompters aren’t just describing things—they’re directing a scene. They’re thinking like a film director. This insight led me to develop a new framework I call the "Director's Method." It’s simple, it’s data-backed, and it works.
It breaks down into four steps:
The Scene: First, describe the core subject and its environment. Be clear and concise. Example: "A lone astronaut standing on a desolate red planet, with two moons in the sky."
The Camera: Next, tell the model how you want to frame the shot. Are you up close or far away? What’s the angle? Example: "Full body shot, from a low angle, looking up at the astronaut."
The Lighting: Now, set the mood. Lighting is one of the most powerful tools for creating atmosphere. Example: "Dramatic cinematic lighting, with the bright light of a nearby nebula casting long, sharp shadows."
The Style: Finally, apply the artistic layer. This is where you bring in your artist references. Example: "In the style of a classic sci-fi painting by John Harris."
Putting it all together: "A lone astronaut standing on a desolate red planet, with two moons in the sky. Full body shot, from a low angle, looking up at the astronaut. Dramatic cinematic lighting, with the bright light of a nearby nebula casting long, sharp shadows. In the style of a classic sci-fi painting by John Harris."
This prompt is specific, it’s descriptive, and it gives the model clear, actionable instructions. There are no magic words, and it doesn’t need a laundry list of negative terms.
Stop Guessing, Start Directing
The world of AI image generation can feel like a black box. But it doesn’t have to be. My analysis of 50,000 images taught me that the principles of good prompting are simpler than we think. Ditch the fluff, focus on what you want, and be incredibly specific about the style.
I challenge you to try the Director’s Method for yourself. See what happens when you stop throwing keywords at the model and start giving it clear, creative direction. I think you’ll be surprised by how much better your images get.
This is a lesson that goes far beyond generating pretty pictures. It’s a lesson I’ve learned over and over again across two successful exits and more than 200 investments: data-driven decisions always win in the long run. Stop following the herd and start looking at the evidence. That’s how you get ahead, whether you’re building a company or creating a work of art.
Frequently Asked Questions
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.