AI is increasingly being used not only to write code, but also to fix vulnerable code.
That sounds promising. Security teams are overloaded, vulnerability backlogs keep growing, and developers are under pressure to patch faster. If an AI model can generate a security fix in minutes, the appeal is obvious.
But new research from 1Password’s Off-by-1 Labs shows a serious problem: AI-generated patches often look correct while failing to fully fix the vulnerability, changing application behavior, or even introducing new security flaws. Dark Reading reported that the study tested more than 6,000 AI-generated patches and found that only 46% solved the underlying vulnerability.
What the Study Tested
The researchers generated 6,080 patches for six recently disclosed vulnerabilities using two frontier AI models: OpenAI’s ChatGPT-5.5 with Trusted Access for Cyber and Anthropic’s Claude Opus 4.8 with Cyber Verification Program. The selected vulnerabilities were recent enough that they were unlikely to be present in the models’ training data, with disclosure dates ranging from March 26 to May 12, 2026.
The goal was to test whether AI could generate real security patches for complex, modern vulnerabilities, not simple toy examples.
That distinction matters. Fixing a security flaw often requires understanding root cause, data flow, edge cases, exploit paths, and existing application behavior. A patch can compile, pass basic tests, and still leave the vulnerability open.
The Results Were Weak
The study found that only 26% of AI-generated patches fully fixed the vulnerability without materially changing application behavior. Another 20.1% fixed the original issue but changed how the application worked. Meanwhile, 49.3% failed to fix at least one exploit path, 2.3% fixed the issue while introducing a new vulnerability, and 2.2% both failed to fix the issue and introduced a new exploit path.
In simple terms, most outputs were not clean security fixes.
Some were incomplete.
Some were brittle.
Some changed behavior.
Some created new problems.
That is why the researchers described these outputs as FLAWED: “Fix-Like Artifacts With Embedded Defects.”
The “Fake Fix” Problem
One of the biggest risks is that AI patches can create a false sense of security.
A generated patch may look reasonable in a code review. It may even block the exact proof-of-concept exploit shown to the model. But security patches need to fix the underlying weakness, not only one visible symptom.
The research found that more than a third of successful-looking patches were fragile. They guarded against a narrow exploit case instead of solving the real root cause.
This is dangerous because defenders may believe a vulnerability is closed when attackers can still bypass the fix with a slightly different input.
AI Has Tunnel Vision
The 1Password paper highlights a common failure mode: models often patch the specific bug and test what they are shown, rather than reasoning more broadly about the safety of the code they are editing. In one Linux kernel case, the models repeatedly missed a related memory-safety issue in the same function because the task did not explicitly point to it.
That is a serious limitation.
Human security engineers often understand that a proof-of-concept is only one example of a broader bug class. AI models can miss that context and produce a fix that satisfies the prompt while leaving nearby risk untouched.
Guidance Matters, But It Cuts Both Ways
The quality of the prompt had a major impact.
When models received correct, root-cause-oriented guidance, their clean fix-success rate increased to 65%. With no guidance, the rate was 50.4%. But when the guidance was wrong, success dropped to 15.2%.
That shows AI can be useful when guided by skilled engineers.
But it also shows that autonomous patching is risky. If the model receives incomplete or misleading information, it may confidently produce the wrong fix.
Why This Matters for Security Teams
AI-generated patches are attractive because they are fast and cheap. The Register reported that the average successful clean patch in the study cost about $6.74, including failed attempts.
But the real cost is not only token spend.
The real cost is review, testing, validation, regression analysis, and the risk of deploying a broken or incomplete fix into production.
If engineers must review many similar but subtly wrong AI-generated patches, the process may become more expensive than writing a human-guided fix from the start.
What Organizations Should Do
AI should be treated as a patch assistant, not an autonomous security engineer.
Organizations should:
- require human review for every AI-generated security patch
- test against the original exploit and variant exploit paths
- verify root-cause remediation, not only symptom blocking
- run regression tests to detect behavior changes
- use SAST, DAST, fuzzing, and targeted security tests before merging
- avoid accepting AI patches automatically
- benchmark AI patching against known historical bugs in their own codebase
- keep developers in control of the remediation process
1Password’s paper recommends that organizations test model patching accuracy against previously human-fixed vulnerabilities in their own codebase before relying on automated patching at scale.
Final Thoughts
AI-generated patches are not useless. They can help developers understand vulnerable code, propose candidate fixes, explain root causes, and speed up remediation work. But they are not ready to replace expert review!
The research shows that autonomous AI patching can fail silently, produce brittle fixes, change application behavior, or introduce new vulnerabilities. That makes blind trust dangerous.
AI can help write the patch, but humans still need to prove that the patch actually fixes the vulnerability.
