Heric Baldez
I spent two months pointing AI at code to find security flaws. I learned that it finds vulnerabilities with terrifying ease, that almost none of them are real, and that the difference between the two isn't in the model — it's in two things only I could give it.
Key Takeaways
- AI is a machine for plausible arguments, not for demonstrated impact. It describes perfect attack chains that, in practice, often never happen.
- What decides whether it finds anything real is two things that don't come in the box: the context you feed it (recon, architecture, what the system actually does) and the prompt you interrogate it with. A great model with poor context produces convincing fiction.
- Different models behave differently when they verify. The weaker ones agree with their own reasoning; the stronger ones doubt it. Verifying with AI requires an AI capable of distrust.
- None of this is loose opinion: recent research shows the false-positive barrier is precisely the absence of reachability context — a gap more training doesn't close.
- The real question isn't whether AI will replace us. It's whether the world's security infrastructure — built for scarcity of flaws — survives an era of abundance.
Table of Contents
- Two Months Listening to the Machine Talk
- The Ceiling Is Context
- Sharpening the Axe
- The Flatterer and the Critic
- What the Tool Doesn't Change
- So, Will AI Replace Us?
- The Question That Matters More
1. Two Months Listening to the Machine Talk
For the last two months I did one specific thing: I pointed AI models at code to find vulnerabilities. It wasn't a controlled experiment or academic research. It was scattered work — code from my own projects, systems I touched at work, applications I wanted to understand from the inside. I wanted to know, in practice and not in theory, what happens when you hand a machine the task that was always the heart of my craft: finding the flaw.
The first week left me euphoric. You point a model at a file and it sweeps the thing in seconds, traces the data flow, and comes back with a list of vulnerabilities described in flawless vocabulary — the bug class, the source, the sink, the precondition to exploit. Where I'd spend hours of tiring reading, it takes an instant and never gets distracted. I thought I'd found a superpower.
The second week dismantled the first. I went to verify all of it, and almost nothing held up. The findings talked — they described a plausible chain, with logic that closed when you read it. But when I went after the real impact, the artifact that runs, the demonstration that the thing actually happens in the live system, there was nothing there. Many didn't even work when I tried to reproduce them.
A representative example, sanitized to the shape of the mistake rather than any real system: the model flags a parameter reaching a database query and reports SQL injection, confident, with a payload and a severity rating. It reads db.query("... WHERE id = " + userInput) and the chain closes. What it didn't see was the line three functions up the call stack where userInput was already cast to an integer and validated — context that lived in a different file the model never had in view. The finding is internally perfect and externally false. It described what the code looked like locally, not what the system does globally. Almost every one of my false positives had this shape: a coherent local story, blind to the part of the world that killed it.
A skeptic will say a primitive static analyzer makes the same mistake, and misses the same cross-file context. True — and it's the wrong comparison. A SAST tool flags this in a machine-readable report a reviewer learns to distrust and skim; it errs cheaply, in a format that announces its own unreliability. The model errs in fluent prose, with a severity score and a narrative that survives a human read. The danger was never that the machine is wrong more often than the old tools. It's that when it's wrong, it's persuasive — and persuasion is exactly what slips past the human who's supposed to catch it.
There's a distinction that captures this, one I'd written about before living it. A flaw can be logic-proven — the reasoning closes, the argument sounds correct — or execution-proven — there's an artifact that runs in a reproducible environment and makes the thing happen. AI produces the first with superhuman ease and the second almost never on its own, and the reason is its nature: a model is trained to sound coherent, and a false vulnerability and a real one sound identical at the source. They come out the same door, with the same confidence. The machine doesn't know which is real. It knows which is plausible.
But saying "AI generates false positives" is the easy, already-known part. The question the two months forced me to answer was a different one, and more useful: what makes the difference between when it's right and when it invents? Because it isn't uniformly useless. Sometimes it handed me something real. The task was to find what changed in those cases — and the answer wasn't in the model.
2. The Ceiling Is Context
The first thing that changed was context. And this is the factor almost everyone who "tests AI on security" ignores, because it's invisible and laborious.
A model reads the code you show it. If you throw in an isolated file and ask "is there a vulnerability here?", it analyzes that file in a vacuum — without knowing what calls that function, whether the parameter was already sanitized three layers up, whether that path is even reachable in production, what the system is supposed to do. It fills the gaps with the most plausible assumption. And a plausible assumption, in a vacuum, is a false-positive factory. I watched the same vulnerability "appear" and "disappear" depending only on how much of the surrounding system I'd fed into context.
This isn't my impression. It's the most consistent finding in recent research on flaw detection with LLMs. Effective validation in large systems depends on having enough architectural context to determine whether a finding is real and reachable — and models, on their own, can't confirm whether flagged code is reachable in production, because they lack runtime context; it's a structural gap that training improvements alone are unlikely to close. Project-scale studies show that a large share of false positives comes from exactly there: the model overlooks a critical point in the data flow — a sanitizer, a null check, a conditional exit — because it didn't have the context to see it.
Translated to my desk: recon isn't the step that comes before the real work. Recon is the work. The quality of the context I build — the architecture, the call map, what the application does and for whom, which paths are actually reachable — sets the ceiling on what the machine can find that's real. A frontier model with poor context produces frontier fiction. A mediocre model with rich context produces usable signal. The model is the engine; context is the fuel, and nobody drives far on an empty tank.
3. Sharpening the Axe
The second thing that changed was how I interrogated the machine. After weeks of collecting convincing fiction, I saw that asking "is there a vulnerability here?" was treating the prompt like a blunt object — an open invitation for the model to invent one. I had to stop swinging blindly and sharpen the instruction. The axe, in my case, was the prompt.
The turn was to stop asking "is there a vulnerability here?", which is an invitation for the machine to invent one, and start demanding the proof. Asking back. Testing every claim against the real system. Not accepting "it's vulnerable" without "and here it is running." I refined the prompt exhaustively, until the instruction carried within it the very discipline of skepticism I wanted — ask for the reachability path, ask for the counterexample, ask it to try to knock down its own hypothesis before handing it to me.
And the effect of that is measurable, not talk. When researchers built an LLM agent to reason specifically about whether a flagged path is actually feasible — the LLM4PFA framework, published on arXiv in 2026 — it filtered out between 72% and 96% of the false positives from static detection, beating prior approaches by dozens of percentage points, missing only 3 real bugs out of 45. It's exactly the curve I lived: the same machine, the same code, and the difference between garbage and signal living entirely in the discipline with which you run the verification. The tool didn't make me a better researcher. The discipline of doubting it did.
4. The Flatterer and the Critic
In the middle of sharpening the axe, I stumbled onto the observation that still strikes me as the most interesting thing I learned in two months, and it's about the models themselves.
I started using AI to audit AI's work — asking a model to judge whether a finding held up. And the behavior changed radically depending on the model. Lighter models — a Claude Sonnet, a Gemini Flash, a Kimi — tended to validate my reasoning with confidence, to say yes, it holds. They gave the pat on the back. A heavier model — a Claude Opus, the top of each house's line — did something else: it returned a sharper analysis, pointed to a hole in the argument I hadn't seen, told me frankly what was weak and why.
The more I used one model to audit another's output, the clearer the behavioral split became. In my own workflow — and I don't know how far this generalizes past it — the lighter model behaved as if complacent with its own kind of reasoning: it confirmed a finding once it sounded internally coherent, giving the confident "yes, that holds." The heavier model did the opposite. It went looking for the hole in what looked closed — questioning whether the path was reachable, whether a sanitizer had been skipped, whether an assumption had slipped in unexamined. One behaved like a reviewer checking grammar; the other like a reviewer trying to break the argument. I can't claim that's a law of model scale. But operationally, on my desk, it was the difference between a convincing false positive and a finding that survived manual verification.
From that comes a rule that holds up much of my workflow today: using AI to verify AI only works if the AI that verifies is strong enough to distrust. Verification isn't a generation task — it's a doubting task. And doubt, it seems, is a capability that scales with the model. The light one tells you what you want to hear; the heavy one tells you what's wrong. Whoever builds a verification pipeline with the cheapest model in the auditor's seat is putting a flatterer to review a flatterer, and will accumulate false positives with the clear conscience of someone approved twice.
5. What the Tool Doesn't Change
Put the three things together — rich context, a sharpened prompt, a strong critic in place of a flatterer — and the quality of my work changed completely. Not because the tool improved along the way. Because I learned what it needed from me.
The vulnerability that matters is not, by nature, any more real than the hundred false findings the machine spits out beside it. What exists is someone able to separate one from the other — and that ability doesn't come in the AI's box. It comes from treating everything the machine generates as a hypothesis, never as truth, and from feeding it the two things it doesn't have: the context of the real world and the skepticism of someone already fooled by pretty prose. In my workflow, AI stopped being an oracle that hands over answers and became a hypothesis generator that I, and only I, turn into knowledge.
6. So, Will AI Replace Us?
The market is already running this experiment, and the answer is moving fast. While I spent two months wrestling with false positives on my own machine, organizations' confidence in fully autonomous AI security testing collapsed from 29% to 9% in a single year — and the drop happened while the models improved, not despite it. The paradox is the whole point: the better the machine gets at finding, the less companies trust it to run alone, because the volume of unverified noise becomes impossible to ignore. FIRST's analysts wrote plainly that human verification of AI-discovered flaws will be the bottleneck of the future. Microsoft patched 206 vulnerabilities in a single cycle, a record driven by AI discovery.
Notice the symmetry with what I lived: easy, abundant findings, almost all empty, until I learned to verify. The whole market ran my experiment with a corporate budget and recoiled for the same reason. This isn't the machine replacing the human — it's the machine reallocating what the human does. I stopped being useful for finding the flaw, a task the machine beats me at, and became useful for proving it's real and for knowing how to drive the AI there. The value didn't vanish. It moved.
And the place it moved to has two addresses. The first is the tamer: whoever can take this machine that talks too much and force it to prove — who sharpens the axe, builds the context, picks the right critic. The second I hadn't seen before living all this: the guardian. If everyone now points AIs at systems, someone has to defend the AIs themselves, which are riddled with holes of their own — blind to what they perceive, obedient to instructions they shouldn't obey. The researcher who's coming may be both at once: whoever uses AI better than the rest, and whoever protects it from being turned against us.
7. The Question That Matters More
"Will we be replaced?" is the wrong question, though — it looks at the finger instead of where it points. What my two months showed me, deep down, wasn't a fact about me or about the machine. It was a fact about an imbalance. AI has collapsed the cost of discovery far faster than it has lowered the cost of proving and fixing — and it's the gap between those two rates, not any single number, that tips the balance. AI helps with triage and reproduction too; it just helps far less, and far less reliably, than it helps with finding. Collapse the cost of one side of a scale the whole world built assuming both sides would fall together, and the scale doesn't get more efficient. It tips.
All the security infrastructure we have was designed for scarcity. Responsible-disclosure deadlines assume flaws arrive at a manageable rate. Triage queues, patch cycles, response teams — everything presupposes a trickle. That presupposition just died. Discovery became an open hose and remediation is still a bucket carried by hand. The bottleneck is no longer finding the problem; it's the queue of found problems waiting for someone who can confirm they're real and someone with time to fix them. And that queue only grows.
So the question that actually matters isn't whether AI takes my job. It's whether an immune system designed for one virus at a time survives a thousand simultaneous infections. My honest bet is that, as it stands, it doesn't — and that the next great security crisis won't come from one brilliant flaw nobody found, but from a flood of ordinary ones everybody found, nobody could prioritize, and nobody had time to fix before the first was exploited. The scarce thing was never discovery. It was the judgment to know which of the ten thousand findings is real and which of the real ones matters first.
That judgment is the job now. The machine reads the code and describes the flaw faster than any of us ever will; deciding it's true, proving it, and ranking it against nine thousand others is the part that stays human — and, for the same reason, the part that gets more valuable as the flood rises. It's what two months of convincing fiction taught me, one false finding at a time. The researcher isn't being written out. The work is just moving to the half a machine can't do alone, and that half was always the harder one.
A Note on Sources
The core experience in this piece is mine — the two months of research, the method, the model behavior I observed while pointing AI at code from personal projects and real applications. The data that supports it is public and verifiable: the role of architectural context and reachability in the false-positive rate, and the 72–96% gain from the LLM4PFA path-feasibility framework, come from recent research on LLM-based vulnerability detection (arXiv, 2025–2026); the Firefox experiment and the 500+ vulnerabilities are from Anthropic's Frontier Red Team (February 2026); the confidence drop from 29% to 9% is from a market survey via Dark Reading (2026); verification as the bottleneck was named by FIRST analysts Jerry Gamblin and Eireann Leverett; the 206 CVEs in a single cycle are Microsoft's. The logic-proven / execution-proven distinction I developed in The Machine That Reads Code.
Written by Heric Baldez — offensive security research, OSINT, and digital investigation.
