Most public vulnerability records do not say which code change fixed the flaw. PatchHolmes, an LLM agent I co-led, finds that commit in the project’s Git history. On 809 CVEs, it ranks the correct fix first for 59.95% of them. Favia, the strongest earlier agent, does so for 34.61%. Each CVE costs about USD 0.007 to process.
PatchHolmes will appear at AACL-IJCNLP 2026, held in Hengqin, China, from 6 to 10 November 2026. The conference is the joint meeting of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing. PatchHolmes builds on SPFinder, an earlier system from our lab led by Jiangrui Zheng, which I co-authored. With SPFinder’s help, the team added fix links for 35 CVEs to the GitHub Advisory Database. PatchHolmes is the third step of my dissertation, where a language model acts with tools inside a code repository.

In this post
Why Most Vulnerability Records Lack a Fix Link
SPFinder Shortlists the Candidates
PatchHolmes Picks the Fix
3.1 A Shortlist, Then an Agent
3.2 Comparing the Whole List Beats Judging One Commit at a Time
What the Results Show
4.1 The Agent Ranks the Fix First 1.7 Times as Often as Favia
4.2 The Gain Comes From the Agent
4.3 A Small Model Is Almost as Good
4.4 The Gain Holds Outside the Main Benchmark
4.5 Each Vulnerability Costs Less Than One Cent
Where the Results Stop
What It Means for Your Team
Questions Readers Ask
Read and Cite the Papers
1. Why Most Vulnerability Records Lack a Fix Link
A CVE record describes a security flaw in text. Advisory databases, such as the GitHub Advisory Database and the US National Vulnerability Database (NVD), add details to it, including a link to the commit that fixed the flaw. According to the SPFinder paper, that link is how a database works out which versions and packages are affected. GitHub’s Dependabot alerts, which warn projects about vulnerable dependencies, also report the fixed version when one is known.
The link is usually missing. VFCFinder, a 2024 study, found that 6,159 of 9,764 GitHub advisories, about 63%, had no link to the fixing commit. The gap can last for years. The NVD published CVE-2013-1814, a flaw in Apache Rave, on 14 March 2013. A maintainer traced its fix by hand on 5 January 2024. The GitHub advisory has listed it since that day. The NVD record still had no fix link on 15 October 2026.
Finding the fix automatically is hard for three reasons:
The search space is large. In GitHubAD, a benchmark of CVEs from the GitHub Advisory Database, 49% of CVEs belong to repositories with more than 5,000 commits.
The changes are long. The SPFinder team measured average commit changes of about 12,000 to 19,000 tokens, depending on the dataset. Older text encoders read only the first 512.
The words differ. The CVE describes the symptom, and the commit describes the code. For CVE-2021-41079, a flaw in Apache Tomcat, the fix commit’s message is “Improving robustness”.
2. SPFinder Shortlists the Candidates
SPFinder, short for Scalable Patch Finder, searches a project’s entire commit history. Earlier evaluations often scored a tool against the correct commit plus 5,000 sampled wrong ones. The SPFinder authors argue that this sampling makes accuracy look higher than in real use. SPFinder ranks commits in three phases:
Rank every commit by keyword matches in its message and code, plus how close its date is to the CVE’s dates. The top 10,000 commits hold the fix for 95% of CVEs.
For those 10,000, compute 10 features. The main one embeds each changed file separately, so a long commit is not cut down to its first 512 tokens.
Order the commits with a learned ranking model, LightGBM, that combines the features.
The measure here is Recall@10, the share of CVEs whose fix appears among the top 10 results, where higher is better. On the GitHubAD test set of 952 CVEs, SPFinder reached 57.7%. VoyageAI’s commercial embedding model, voyage-3, reached 44.6%. The first stage of PatchFinder, an earlier fix-finding tool, reached 5.4%. On a second test set of 629 CVEs, drawn from the PatchFinder authors’ dataset, SPFinder reached 74.4% against 64.9% for voyage-3.

The team then used SPFinder on GitHub advisories that lacked a fix link. They checked the top 30 results by hand and proposed each fix as a pull request to the advisory database. GitHub merged all 35, covering 11 projects, between 8 and 27 May 2025. The list of merged links is public. One of them is CVE-2024-22031, a high-severity flaw in Rancher. Its fix commit, “Add project namespace handling”, changes 19 files and never mentions the flaw. The pull request added it to the advisory on 23 May 2025.
3. PatchHolmes Picks the Fix
3.1 A Shortlist, Then an Agent
PatchHolmes takes a CVE description and a local copy of the project’s repository and returns one commit with a written reason. It works in two phases:
Phase 1 builds a shortlist. One ranking uses keywords and dates, with SPFinder’s weights. A second ranking uses meaning, from the Qwen3-Embedding-8B embedding model. PatchHolmes merges the two and keeps the top 100 commits, which hold the fix for 87.64% of CVEs.
Phase 2 runs an LLM agent for at most 15 steps. The agent can see the whole list, open a commit, open one file’s changes, or submit an answer. A typical run opens 3 to 10 of the 100 commits, 5 on average.
The agent uses Qwen3-235B, an open-weight model used without further training. PatchHolmes needs no paid search service. The authors argue that the cost of training per repository or paying per search would grow with every new CVE.

3.2 Comparing the Whole List Beats Judging One Commit at a Time
Favia, a 2026 agent from another group, judges each candidate commit on its own. It asks “Is this the fix?” once per candidate, ten times per CVE. When several candidates look plausible, those separate answers give no way to compare them. PatchHolmes instead shows the agent all candidates at once and asks which one is the fix.
The jQuery flaw CVE-2015-9251 shows the difference. In jQuery before 3.0.0, a request to another website could run that site’s script. Both systems received the same 10 candidates, with the fix at position 9:
Favia said yes to all 10 with the same top confidence and fell back to list order. It picked commit e7b3bc4, a 2020 change to how jQuery handles JSON data.
PatchHolmes read the list, then opened commits 1, 2, 9, 4, and 3. It picked commit b078a62, “Ajax: Mitigate possible XSS vulnerability”, from 10 September 2015. Only that commit stops scripts from other sites from running.

4. What the Results Show
The main measure is Recall@1, the share of CVEs whose first answer is the correct fix, where higher is better. Unless noted, results come from 809 CVEs sampled from the 8,401 in GitHubAD. Every LLM method used the same model, Qwen3-235B.
4.1 The Agent Ranks the Fix First 1.7 Times as Often as Favia
PatchHolmes reached 59.95% Recall@1, and Favia reached 34.61%. IRCoT, a general method that alternates search and reasoning, reached 28.55%. SPFinder reached 27.32%.
McNemar’s test, which checks whether two methods differ on the same CVEs, found the gap to Favia statistically significant, with p far below 0.001. A gap that large would almost never appear by chance if the two methods were equally good.
Within the top 10, the methods were closer. PatchHolmes found the fix there for 77.26% of CVEs, against 72.31% for Favia. The authors read this as Favia often having the fix in its list but not placing it first.
4.2 The Gain Comes From the Agent
The authors held the 100 candidates fixed and changed only the selection step:
Taking Phase 1’s top-ranked commit gave 32.63% Recall@1. The full agent gave 59.95%.
Given PatchHolmes’s own candidates, Favia reached 39.80% and IRCoT 37.58%.
An agent that could only see the list and submit reached 48.21%, the largest single gain. Opening commit summaries raised it to 57.48%, and reading code changes to 59.95%.
More text did not help. Giving the agent the full NVD report, 1,786 characters on average instead of about 150, left Recall@1 at 59.83%.

4.3 A Small Model Is Almost as Good
Both Qwen3 models activate only part of their weights for each word. That part is called the active parameters. The results with other models inside the agent:
Qwen3-Coder-30B, with 3B active parameters, reached 59.09%, against 59.95% for Qwen3-235B with 22B. It also cut the average time per CVE from 86 to 39 seconds.
gpt-oss-120B reached 56.98% and gpt-oss-20B 49.81%. Both often used up their 15 steps without submitting an answer.
Every model stayed far above the 32.63% without an agent.
The authors read this as evidence that the agent keeps its gain when the model is swapped. They do not claim that model size never matters.
4.4 The Gain Holds Outside the Main Benchmark
PatchFinder’s benchmark has 1,252 CVEs, each with 10 candidates chosen by PatchFinder’s own ranker. PatchHolmes’s agent, unchanged, raised Recall@1 from 24.28%, PatchFinder’s top pick, to 39.86%. Only 577 of the 1,252 CVEs have the fix among their 10 candidates, so no method can exceed 46.09%.
The authors also ran PatchHolmes on 35 CVEs from 11 projects, disclosed between 2013 and 2025. It ranked the fix first for 21 of them, 60.0%, against 8, or 22.9%, for Phase 1 alone.
The 35 CVEs are the same ones whose fix links the SPFinder team merged in May 2025. Their correct answers came from SPFinder’s results plus manual checks, so this set cannot compare the two systems fairly. SPFinder had ranked 26 of them first.
4.5 Each Vulnerability Costs Less Than One Cent
Per CVE, PatchHolmes with Qwen3-235B used on average:
96,118 input tokens and 846 output tokens
8 tool calls, opening 5 commits
86 seconds, with a median of 49 seconds
At OpenRouter’s price on 1 May 2026, that comes to about USD 0.007 per CVE, or about USD 58 for all 8,401 CVEs in GitHubAD. From the token counts in Favia’s paper, the authors estimate that Favia would cost about USD 400 for the same 8,401. Only 0.49% of CVEs exceeded the model’s 32,000-token context window.
5. Where the Results Stop
PatchHolmes has these limits:
Every benchmark holds only CVEs whose fix is already known. The hardest cases, where no one has found the fix, remain untested.
For 12.36% of CVEs, the fix is not among the top 100 candidates, so the agent cannot find it.
For 7.28% of CVEs in GitHubAD, more than one commit fixes the flaw. A valid fix that the records do not list counts as a miss.
The authors recommend using PatchHolmes to suggest candidates for human review. They advise against filling vulnerability databases automatically.
SPFinder has these limits:
SPFinder is an arXiv preprint. As of 16 October 2026, it has no conference or journal venue.
Most of its failures come from CVE text that names nothing found in the code. In 40 failed cases, 37.5% of CVE texts named no identifying file or component, and 42.5% named ones that the fix commit does not contain.
One of its features calls external services for each query. Two of them, GPT-4o and VoyageAI embeddings, are paid, and the third is GitHub code search.
Its code is in an anonymized repository with no license.
6. What It Means for Your Team
Both systems point to practices that apply beyond these papers:
When your team fixes a vulnerability in its own code, link the fix commit from the advisory, and name the CVE ID in the commit message once it exists. Commits that never mention the flaw, such as Rancher’s “Add project namespace handling”, are what makes the search hard.
Split the work into two steps: a cheap ranker builds a shortlist, and an LLM compares the shortlist as one list.
Test a small open model before paying for a large one. Here, 3B active parameters scored 59.09% against 59.95% for 22B.
Measure how often the shortlist contains the answer. PatchHolmes’s agent cannot find the 12.36% of fixes that Phase 1 misses, so a better shortlist raises the best possible score.
Keep a person approving each link before it reaches a database. All 35 merged SPFinder links were checked by hand.
7. Questions Readers Ask
How do I find the commit that fixed a CVE in a GitHub repository?
Start with the GitHub Advisory Database, which lists fix commits for many CVEs. When it has none, a tool like PatchHolmes can rank candidate commits from the repository’s history. On 809 CVEs, PatchHolmes ranked the correct fix first for 59.95% of them, so a person should confirm its answer.
Why do so many CVEs have no fix link?
A person has to trace each fix by hand, and the fix commit often does not mention the flaw. A 2024 study found that about 63% of GitHub advisories had no link to the fixing commit.
How accurate is an AI agent at finding vulnerability fix commits?
PatchHolmes ranks the correct fix first for 59.95% of 809 CVEs. Favia, an earlier agent, does so for 34.61%. Both used the same model, Qwen3-235B.
Do I need the largest LLM for this?
In this study, the size of the model mattered little. Qwen3-Coder-30B, with 3B active parameters, reached 59.09% against 59.95% for Qwen3-235B, and ran in less than half the time.
How much does it cost to run over thousands of CVEs?
About USD 0.007 per CVE, or about USD 58 for 8,401 CVEs, at OpenRouter’s price on 1 May 2026.
Is the code available?
Yes. PatchHolmes is on GitHub under the MIT license. SPFinder’s code is in an anonymized GitHub repository.
8. Read and Cite the Papers
Guanqun Yang, Yingming Zhou, Jiangrui Zheng, Shudong Hao, and Xueqing Liu. “PatchHolmes: Agentic Patch Retrieval via Listwise Selection.” Accepted at AACL-IJCNLP 2026. The first three authors contributed equally. arXiv, code.
Jiangrui Zheng, Xueqing Liu, Guanqun Yang, Siyan Wen, Qiushi Liu, and Xiaoyin Wang. “SPFinder: Improving the Context Length and Scalability for Tracing Known Vulnerability Patches.” arXiv preprint, 2026. arXiv, code.
PatchHolmes is the third of four steps in my dissertation. The overview post explains how the steps fit together. The FewVuln post covers reading vulnerability reports, and the TestAug post covers writing tests.
To cite the papers:
@misc{yang2026patchholmesagenticpatchretrieval,
title = {PatchHolmes: Agentic Patch Retrieval via Listwise Selection},
author = {Guanqun Yang and Yingming Zhou and Jiangrui Zheng and Shudong Hao and Xueqing Liu},
year = {2026},
eprint = {2609.38807},
archivePrefix = {arXiv},
primaryClass = {cs.IR},
note = {Accepted at AACL-IJCNLP 2026},
url = {https://arxiv.org/abs/2609.38807}
}
@misc{zheng2026spfinderimprovingcontextlength,
title = {SPFinder: Improving the Context Length and Scalability for Tracing Known Vulnerability Patches},
author = {Jiangrui Zheng and Xueqing Liu and Guanqun Yang and Siyan Wen and Qiushi Liu and Xiaoyin Wang},
year = {2026},
eprint = {2503.22935},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
url = {https://arxiv.org/abs/2503.22935}
}
Guanqun Yang, LLM Researcher. More work at guanqun-yang.com and on Google Scholar.

