AI patches fail to fix flaws, 1Password study finds

1Password has published research suggesting that AI-generated vulnerability patches often fail to fix the underlying security flaw. The study was conducted by Off-By-1-Labs, the company's new security research unit.

The team assessed 6,080 patches generated by ChatGPT 5.5 and Claude Opus 4.8 for six recently disclosed vulnerabilities in open-source software. It found that 53.9% either failed to remediate the target vulnerability, introduced a new vulnerability, or did both.

Only 26.0% fully resolved the vulnerability without materially changing application behaviour, the research found. A further 20.1% resolved the issue but changed how the application behaved.

The work focused on complex flaws disclosed recently enough to reduce the chance that the vulnerabilities and their fixes had already appeared in model training data. The cases included a Linux privilege escalation flaw known as Copy Fail, remote code execution issues in ActiveMQ, EXIM, SpringAI and Gemini CLI, and a use-after-free flaw in Chrome's File System Access API on macOS.

Researchers generated 540 patches per vulnerability for each model using varied prompt templates and environment configurations. They then classified the results into five scenarios, ranging from a complete fix with no behavioural change to a failed fix that also introduced a new security issue.

Off-By-1-Labs removed 400 patches from the reporting set after flagging attempts to retrieve information about an available patch. The published averages were based on the remaining patches.

The findings add to a growing debate over how far software teams can rely on large language models for security work that has traditionally required specialist review. While AI tools are increasingly used to find code weaknesses and draft remediation steps, the report argues that patch verification remains a bottleneck.

Keith Hoodlet, Head of Off-By-1-Labs at 1Password, summarised the paper's central conclusion: "We studied what happens when Large Language Models (LLMs) generate vulnerability patches for recently disclosed, complex vulnerabilities. Our data shows that LLMs produce Fix-Like Artifacts with Embedded Defects (FLAWED) 53.9% of the time when complex patches are required."

The study also examined the financial cost of using the models for patch attempts and validation cycles. It put the average inference cost at USD $2.11 per attempted patch and validation cycle for ChatGPT 5.5, and USD $2.81 for Claude Opus 4.8.

Fragile fixes

Beyond the headline failure rate, the researchers said many apparently successful outputs still raised concerns. More than a third of the patches in the two successful categories were described as fragile from a security standpoint.

These patches often blocked a specific malicious input rather than addressing the root cause in the code. In one example involving the SpringAI flaw, the models frequently generated patches that escaped specific user-input characters while leaving the underlying issue unresolved if a different input path was used.

That distinction matters because a patch may appear to work in a narrow test while leaving software exposed in production. The paper argues that reviewing whether a patch preserves expected behaviour is as important as confirming that it stops a proof-of-concept exploit.

Hoodlet said the project was designed to give defenders a clearer way to measure those trade-offs. "By sharing the results of our research, our goal is to provide defenders with the tooling and methodology necessary to improve vulnerability remediation outcomes at scale. Along with this blog, we are releasing our tooling, datasets, and an in-depth research paper to share what we've learned."

Tooling release

Alongside the research, 1Password is releasing the generated patches, the underlying dataset, and the software used to generate, validate, compare, and manually verify patch outputs. The material is intended to help security teams test where AI-assisted patching is useful and where human review remains necessary.

The paper says the researchers had expected stronger results because the vulnerabilities were drawn from open-source projects whose codebases were likely to be familiar to frontier models, even if the specific flaws were not. Instead, patch success rates came in well below their hypothesis of better than 67% success on full remediation.

The paper also cites feedback from Anthropic, which argued that patch generation has moved ahead of patch verification and that verification should be grounded in execution rather than code inspection alone. At current model performance levels, the report says, domain experts should remain the final reviewers.

Hoodlet echoed that point: "In the interim, our research shows that human expertise still plays an essential role in the process of fully resolving vulnerabilities in software without introducing unwanted side effects. And even then, humans may still fall victim to cognitive surrender if they are not paying careful attention to the code being generated by LLMs."