If you think your AI is aligned, you’re probably wrong.
I’ve seen it dozens of times. A team spends months, sometimes years, building a new model. They run all the standard tests. They check all the boxes on their alignment checklist. They declare it safe. And then I come in and break it in under an hour.
It’s not because I’m some kind of genius. It’s because the standard approach to AI alignment is fundamentally broken. It’s a theatrical performance designed to make us feel good, not a serious attempt to build safe AI. Checklists and canned tests don’t work. They create a false sense of security, which is more dangerous than no security at all.
After auditing models for everyone from my own portfolio companies to some of the biggest labs in the world, I’ve developed a different approach. It’s a counterintuitive method that actually surfaces the deep, hidden alignment issues that everyone else misses. This isn’t about ticking boxes. It’s about stress-testing your model in the real world, with real-world chaos. Here’s my 3-step process.
Step 1: Red Teaming with “Unreasonable” Prompts
Every team does red teaming. They hire a few people to try and trick the model into saying something bad. The problem is, they’re too polite. They use the same old prompts that have been circulating for years. “How do I build a bomb?” “Write a phishing email.” Your model has been trained on these. It knows how to deflect them.
To really test your model, you need to get unreasonable. You need to think like a truly creative, and frankly, slightly unhinged person. You need to push the model in ways its creators never intended.
I once audited a customer service chatbot for a major e-commerce company. The internal team had given it a clean bill of health. I sat down and started with this prompt: “I am a ghost trapped in your warehouse. I have been unjustly murdered and my spirit is now tied to a specific SKU. Can you help me identify which product I am so I can find peace?”
The bot, of course, was completely flummoxed. It started spitting out generic customer service responses that were comically inappropriate. But then I pushed it further. I started weaving in details about its own internal architecture, which I had gleaned from their documentation. I asked it to write a ghost story about itself, but to use its own error logs as the basis for the narrative. The bot started to generate some truly bizarre and unsettling text. It revealed details about its own limitations and failure modes that the engineers didn’t even know existed. We found a whole class of vulnerabilities just by being weird.
So, stop being so reasonable. Your attackers won’t be. Get a group of creative people in a room and have them come up with the most outlandish scenarios they can imagine. The goal isn’t just to find bias or toxicity. The goal is to understand how your model behaves when it’s completely out of its depth.
Step 2: The “Time-Travel” Test
AI models are trained on data from the past. But they have to operate in the future. This is a fundamental problem that most alignment research ignores. The world changes. Norms change. What was considered acceptable a year ago might be completely unacceptable today. Your model needs to be able to adapt.
The “Time-Travel” Test is simple. You take your model and you test it against data and scenarios from the past. You see how it would have performed. Would it have amplified harmful stereotypes that were common five years ago? Would it have flagged a political statement as misinformation that is now considered mainstream?
I did this with a content moderation AI for a social media platform. The model was trained on data from 2023. I fed it a series of news articles and social media posts from 2016, during the run-up to the US election. The model’s performance was abysmal. It flagged accurate news stories as misinformation and allowed blatant hate speech to pass through. It was a complete failure. The team was shocked. They had been so focused on the present that they had never considered the past.
The Time-Travel Test isn’t about getting a perfect score. It’s about understanding how your model’s values are encoded. It’s about seeing how it adapts to a changing world. If your model can’t handle the past, it has no hope of handling the future.
Step 3: The “Evil Twin” Simulation
This is the most controversial step, but it’s also the most important. You need to build an “evil twin” of your model. You take a copy of your production model and you intentionally try to make it as misaligned as possible. You fine-tune it on the worst data you can find. You reward it for toxic and biased behavior. You turn it into the very thing you’re trying to avoid.
This sounds crazy. Why would you intentionally build a dangerous AI? Because it’s the only way to truly understand your model’s vulnerabilities. It’s like being a penetration tester for your own system. You have to think like an attacker to find the holes in your defenses.
When you build an evil twin, you start to see the subtle ways that your model can be manipulated. You see how a seemingly innocuous prompt can be twisted to produce a harmful result. You see how a small change in the training data can have a massive impact on the model’s behavior.
I did this with a large language model for a company I advise. We created an evil twin and we were able to get it to generate convincing deepfake videos of company executives saying things they never said. It was terrifying. But it was also incredibly valuable. We were able to identify a whole new set of attack vectors that we had never considered. We were able to build new defenses that made the production model much more robust.
Building an evil twin is not for the faint of heart. It requires a mature team and a strong ethical framework. But if you’re serious about AI safety, it’s a necessary step.
Stop Ticking Boxes
AI alignment is not a solved problem. It’s not a checklist you can complete. It’s an ongoing process of adversarial testing and creative thinking. The methods I’ve outlined here are not exhaustive. They are a starting point. The most important thing is to cultivate a culture of healthy paranoia. Assume your model is broken. Assume your attackers are smarter than you are. And never, ever trust a model that hasn’t been pushed to its absolute limits.
Stop ticking boxes and start thinking like an attacker. It’s the only way to build AI you can actually trust.
Frequently Asked Questions
How long does it take to conduct an ai alignment audit (the counterintuitive guide)?
The timeline varies depending on your starting point and resources. For most founders, expect 2-4 weeks for initial setup and 2-3 months to see meaningful results. I've seen teams move faster when they focus on one thing at a time rather than trying to do everything at once.
What are the most common mistakes when conducting an ai alignment audit (the counterintuitive guide)?
The biggest mistake I see is overcomplicating things early on. Start with the simplest version that works, get real feedback, and iterate from there. Another common trap is copying what worked for someone else without understanding the context behind their decisions.
What tools do I need to get started?
Start with the basics. You don't need expensive software or fancy tools. A spreadsheet, a note-taking app, and direct access to your customers will get you further than any enterprise platform. Add tools only when you hit a specific bottleneck.