I remember the exact moment the cold dread set in. I was on a call with the CEO of a portfolio company—a brilliant founder with a product I’d backed early on. Their company, which used an AI model to detect manufacturing defects, was suddenly going haywire. Their top-of-the-line system, which boasted 99.8% accuracy in trials, was now missing flaws a human inspector would spot from a mile away. Revenue was tanking, and their biggest client was threatening to pull their contract.
They had spent weeks tearing their hair out, blaming everything from bad data pipelines to a rogue software update. But the real cause was far more sinister. A competitor, we later discovered, had launched a subtle adversarial attack, feeding the model a stream of almost imperceptible "noise" that was invisible to the human eye but catastrophic for the AI. They were intentionally causing the model to fail. It was a targeted, malicious act of industrial sabotage, and it almost killed the company.
This isn’t a one-off story. I’ve seen variations of this nightmare play out over and over again. After analyzing more than 100 security incidents involving AI, a terrifying pattern has emerged: most founders are sleepwalking into a minefield. They’re so focused on building the next groundbreaking model that they completely ignore the new and profound ways these models can be broken.
The New Breed of Threat
Forget what you know about traditional cybersecurity. Hacking an AI is a different beast entirely. It’s less about brute force and more about sophisticated manipulation. It’s about understanding the psychology of the machine and exploiting its blind spots. The attacks I’m seeing are not just clever; they’re devious.
I was an early investor in a company that built a cutting-edge AI for detecting deepfakes. They were the good guys, or so we thought. The problem was, their model was so focused on identifying known deepfake patterns that it was completely blind to a new type of attack. An attacker figured out that by adding a specific, almost invisible layer of digital "static" to a video, they could make a deepfake appear completely authentic to the AI. The very tool designed to protect us was turned into an unwitting accomplice.
These aren’t just theoretical risks. They are active, in-the-wild threats that are costing companies millions. Here are the three main types of attacks I see most often:
Data Poisoning: This is the slow, silent killer. Attackers don’t come through the front door; they poison the well. They find ways to inject malicious data into your training sets. For an e-commerce recommendation engine, this could mean subtly associating a competitor's products with your most popular items. For a medical diagnostic tool, it could mean training the model to ignore the early signs of a disease. The scary part is that you might not even know it’s happening until it’s far too late.
Evasion Attacks: This is the classic cat-and-mouse game. Think of it like a magic trick. The attacker creates an input that looks perfectly normal to a human but is designed to fool the model. We saw this with a security company I advise. Their AI-powered surveillance system, which could identify weapons with incredible accuracy, was completely fooled by a 3D-printed object that looked like a turtle to the human eye but was classified as a rifle by the AI. The attacker had found a "hole" in the model's understanding of the world.
Model Stealing: This is the ultimate heist. Your model is your intellectual property, your competitive advantage. But what if someone could steal it without ever accessing your code? By repeatedly querying your API and analyzing the outputs, an attacker can essentially "clone" your model. They can reconstruct its architecture and weights, effectively stealing years of your work and investment. I saw this happen to a fintech startup that had built a proprietary fraud detection model. A competitor stole it and launched a rival service within months.
The Founder's Delusion
When I talk to founders about these risks, I often hear the same objections. They’re the same objections I heard from the CEO of the manufacturing company before their world came crashing down.
"We're too small to be a target." This is dangerously naive. Attackers are opportunistic. They’re looking for the path of least resistance. A small, fast-moving startup with a valuable AI model and a non-existent security budget is the perfect target. You’re not too small to be a target; you’re an easy target.
"But our model is 99% accurate!" That 1% is where the devil lives. An accuracy score is a vanity metric when it comes to security. It tells you how well your model performs on average, but it tells you nothing about its performance in the face of a determined adversary. An attacker isn’t interested in the 99%; they’re interested in the 1% of cases where your model fails catastrophically.
"My engineers are brilliant." I have no doubt they are. But are they brilliant at adversarial machine learning? Building a secure AI system requires a completely different skill set than building an accurate one. It’s the difference between being a brilliant architect and being a brilliant security consultant. You need both.
A New Playbook for an Age of AI Insecurity
So, what’s the answer? You can’t just unplug your models and go back to the old way of doing things. The genie is out of the bottle. The only way forward is to build a new kind of resilience.
First, you need to adopt a paranoid mindset. Assume your models are already compromised. Assume there are attackers trying to break them right now. This isn’t about creating a culture of fear; it’s about creating a culture of vigilance. From day one, you should be thinking about how your model could be attacked.
Second, invest in specialized expertise. Don’t just task your existing engineering team with "doing AI security." It’s a specialized field. You need to hire people who live and breathe this stuff. They’re the ones who will know how to build the right defenses, how to test your models for vulnerabilities, and how to respond when an attack happens.
Third, make monitoring your obsession. You need to have real-time visibility into how your models are behaving in the wild. This means tracking not just their accuracy but also their predictions, their confidence scores, and their inputs. Look for anomalies. Look for strange patterns. And when you find them, investigate them relentlessly.
Finally, think in layers. There is no single magic bullet that will protect you from adversarial AI. You need a multi-layered defense strategy. This includes everything from robust data validation and sanitization to real-time threat detection and response. It’s about building a system where an attack at one layer is caught by another.
The Wake-Up Call
The story of the manufacturing company had a silver lining. The attack was a brutal wake-up call, but it forced them to take AI security seriously. They brought in experts, they rebuilt their systems from the ground up with security in mind, and they emerged stronger and more resilient than before. They even turned their experience into a new product line, helping other companies secure their AI systems.
But not everyone is so lucky. For every company that survives an attack, there are others that don’t. The threat is real, and it’s growing every day. If you’re a founder in the AI space, you have a choice. You can either bury your head in the sand and hope for the best, or you can face this new reality head-on. Your company’s survival depends on it.
Frequently Asked Questions
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.