From 2020 to 2026, my research asked how much of the manual work in software testing and security a language model can do. I found that a model can do much of it, if it gets more independence one step at a time. My dissertation, Scalable Software Testing and Analysis with NLP Methods (Stevens Institute of Technology, 2026), tests this in four studies:
FewVuln reads security reports and extracts the affected software and versions.
TestAug writes tests that expose failures in AI text models.
PatchHolmes searches a code repository for the change that fixed a vulnerability.
WatchPoint gives feedback to an AI coding agent whose code failed a test.
Across the four studies, the design around the model mattered more than the model’s size. In PatchHolmes, a model about 7 times smaller lost less than 1 percentage point in its main accuracy score.

In this post
Why Checking Software Still Depends on People
Four Steps of Growing Independence
2.1 Step 1, Reading: FewVuln
2.2 Step 2, Writing Tests: TestAug and HateModerate
2.3 Step 3, Acting With Tools: PatchHolmes
2.4 Step 4, Working With Another Agent: WatchPoint
What It Means for Your Team
3.1 The Design Around a Model Matters More Than Its Size
3.2 A Helper’s Value Depends on the Model That Uses It
Research Beyond the Dissertation
Questions Readers Ask
Read and Cite the Work
1. Why Checking Software Still Depends on People
Poor software quality cost the United States at least USD 2.41 trillion in 2022, according to a report by the Consortium for Information & Software Quality. Much of the work that prevents such losses is still done by hand. Most of that work deals with text:
Security analysts read vulnerability reports, written as paragraphs of English, and search code histories for the change that fixed each flaw.
Test engineers write test cases one at a time, each a sentence with an expected answer.
Developers who use AI coding assistants run and check each change that comes out of a conversation with the assistant.
Language models are built to process text, so they suit this work. The studies below ask how much of the work a model can do without a person checking each step.
2. Four Steps of Growing Independence
My dissertation answers that question with four studies. Each study gives the model more independence than the one before, and each step removes one kind of manual work. The figure below shows where each study works in the life of a software product.

2.1 Step 1, Reading: FewVuln
Each publicly disclosed vulnerability gets a CVE entry that describes the flaw in free text. Before a team can patch, someone must extract which software and which versions are affected. FewVuln fine-tunes a pre-trained language model for this extraction. It was presented at MLHat 2021, a workshop on machine learning for security held with KDD 2021, a major data mining conference.
Trained on 576 labeled sentences, FewVuln reaches an F1 score of 0.97 (on a 0-to-1 scale) for software names. The best earlier system reached 0.98 with 5,758 labeled sentences. In other words, FewVuln needs about 90% less labeling work, at a small cost in accuracy. Read the FewVuln paper. Read the full post on FewVuln.
2.2 Step 2, Writing Tests: TestAug and HateModerate
A model can score well on a standard test set and still fail simple cases. In the TestAug study, text classifiers that were wrong on only 5.7% to 12.6% of a standard test set failed 14.2% to 49.8% of targeted tests, such as sentences with negation. Teams usually write targeted tests by hand, and TestAug lets GPT-3 write them instead. People wrote 29 test templates, and GPT-3 produced the other 1,441, about 98% of the 1,470 in total. TestAug was published at COLING 2022, the International Conference on Computational Linguistics. Read the TestAug paper.
HateModerate applies the same idea to content moderation. It appeared in Findings of NAACL 2024, a peer-reviewed volume of the North American conference of the Association for Computational Linguistics (ACL). The main results:
HateModerate matches 7,704 test cases to the 41 hate speech policies that Facebook had published by November 2022.
Training a hate speech detector on these cases cut its failure rate on policy tests it had not seen from 0.423 to 0.275, where lower is better.
On HateCheck, an independent test set, the same training cut the failure rate from 0.365 to 0.199.
Read the HateModerate paper. Read the full post on TestAug and HateModerate.
2.3 Step 3, Acting With Tools: PatchHolmes
Security databases, and the alert tools built on them, need to know which commit fixed each vulnerability, for example to tell users which version is safe. That link is often missing. A 2024 study, VFCFinder, found no link to the fixing commit for 63% of the advisories in the GitHub Advisory Database.
PatchHolmes gives a language model four tools that can read a code repository but not change it. For each vulnerability, it works in three steps:
A search engine picks 100 candidate commits from the repository.
The agent compares the candidates side by side and opens 3 to 10 of them.
The agent names one commit as the fix.
On a benchmark of 809 vulnerabilities, PatchHolmes ranks the true fix first for 60.0% of them, against 34.6% for Favia, the strongest earlier method. Each search costs about USD 0.007 at the public price of the model it uses. PatchHolmes has been accepted at AACL-IJCNLP 2026, a joint Asia-Pacific conference on natural language processing. Read the PatchHolmes paper. Read the full post on PatchHolmes.
2.4 Step 4, Working With Another Agent: WatchPoint
As of 2026, AI coding agents can build whole web applications. When their code fails a test, they usually see only the error message. A human developer would open the application, click its buttons, and inspect it. WatchPoint simulates that developer. When a task fails:
A second language model writes small diagnostic scripts for the failed task.
It runs the scripts against the live application and reports what it observes.
The coding agent uses the report to repair its work.
WatchPoint is a 2026 arXiv preprint. Its evaluation uses Web-Bench, a benchmark of 50 multi-file web projects:
Of the failed tasks that WatchPoint diagnosed, 57.6% passed on the retry (38 of 66).
When the coding agent saw only the test errors, 45.1% passed on the retry.
In a small study on a different set of tasks, two human testers fixed 12 of 22 tasks (54.5%), so the comparison with people is preliminary.
Read the WatchPoint paper. Read the full post on WatchPoint.
3. What It Means for Your Team
The four studies suggest two lessons for any team that uses AI models. The figure after the lessons shows the numbers behind both.
3.1 The Design Around a Model Matters More Than Its Size
In PatchHolmes, on the benchmark of 809 vulnerabilities:
The search engine alone ranked the true fix first for 32.6% of vulnerabilities.
Adding the agent raised that share to 60.0%.
Switching to a model with about 7 times fewer active parameters lowered the share by only 0.86 percentage points, to 59.1%.
Before paying for a bigger model, check whether a better workflow around the current model gives the same gain.
3.2 A Helper’s Value Depends on the Model That Uses It
In WatchPoint, the measure is the share of coding tasks passed within two attempts:
With WatchPoint’s feedback, a weaker coding model rose from 26.6% to 28.1%.
With the same kind of feedback, a stronger coding model fell from 44.5% to 37.7%.
The drop for the stronger model is within the normal variation between repeated runs, so it needs more tests before anyone draws a firm conclusion.
Measure a helper on your own model before enabling it everywhere.

4. Research Beyond the Dissertation
Later papers follow the same approach. Each one tests a small, cheap change carefully against a costlier one:
Sequential Modality Dropout (SMD), accepted at CIKM 2026, an international conference on information and knowledge management, keeps recommender systems accurate when product photos or descriptions are missing.
DualSpectralCF, also accepted at CIKM 2026, lets a recommender use dislikes without costly training. Read the full post on SMD and DualSpectralCF.
SkillSeek, accepted at AACL-IJCNLP 2026, shows that plain search selects skills for AI agents about as well as a search loop run by a large language model, at a much lower cost.
5. Questions Readers Ask
Can language models replace security analysts and test engineers?
They can do the repetitive parts of the work. In these studies, people still:
define each task,
write a few hundred labels or a few dozen test templates,
check the results.
PatchHolmes, for example, cannot find a fix that its search engine leaves out of the top 100 candidates, which happened for 12.4% of vulnerabilities.
Do these methods need the largest and most expensive models?
The studies suggest they do not. PatchHolmes lost less than 1 percentage point with a model about 7 times smaller, and each search costs about USD 0.007. The gains came mainly from how the model was used.
Where can I read the full dissertation?
The full text is free on guanqun-yang.com. The official record is in ProQuest, the main database of dissertations from US universities.
6. Read and Cite the Work
The dissertation:
Guanqun Yang. Scalable Software Testing and Analysis with NLP Methods. Doctoral dissertation, Stevens Institute of Technology, 2026. Full text (PDF), ProQuest record.
A two-page summary of the research plan, from the doctoral consortium of AAAI-25, the 2025 conference of the Association for the Advancement of Artificial Intelligence:
Guanqun Yang. “Advancing Intelligent Software Development and Trustworthy Models Through the Synergy of Software Engineering and LLMs.” Proceedings of the AAAI Conference on Artificial Intelligence 39(28), 2025. Publisher page.
The papers behind each step:
“Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-Trained Language Models” (FewVuln), MLHat 2021. arXiv, Springer.
“TestAug: A Framework for Augmenting Capability-based NLP Tests”, COLING 2022. ACL Anthology.
“HateModerate: Testing Hate Speech Detectors against Content Moderation Policies”, Findings of NAACL 2024. ACL Anthology.
“PatchHolmes: Agentic Patch Retrieval via Listwise Selection”, AACL-IJCNLP 2026. arXiv.
“WatchPoint: Executable User Feedback for Real-World Agentic Web Development”, arXiv preprint, 2026. arXiv.
To cite the dissertation:
@phdthesis{yang2026scalable,
author = {Guanqun Yang},
title = {Scalable Software Testing and Analysis with NLP Methods},
school = {Stevens Institute of Technology},
year = {2026},
url = {https://guanqun-yang.com/paper/dissertation.pdf}
}
Guanqun Yang, LLM Researcher. More work at guanqun-yang.com and on Google Scholar.

