<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Guanqun (David) Yang: Research Explained]]></title><description><![CDATA[Peer-reviewed research on large language models, AI agents, software testing, security, and recommender systems, explained for engineering and product leaders. Each post covers one paper: the problem, the method, the numbers, and what it means for your team.]]></description><link>https://newsletter.guanqun-yang.com/s/research-explained</link><image><url>https://substackcdn.com/image/fetch/$s_!fov6!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc55319e8-1ff1-4d89-906d-7b2d89f4dc5d_200x200.png</url><title>Guanqun (David) Yang: Research Explained</title><link>https://newsletter.guanqun-yang.com/s/research-explained</link></image><generator>Substack</generator><lastBuildDate>Mon, 16 Nov 2026 03:52:22 GMT</lastBuildDate><atom:link href="https://newsletter.guanqun-yang.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Guanqun (David) Yang]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[guanqunyang@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[guanqunyang@substack.com]]></itunes:email><itunes:name><![CDATA[Guanqun (David) Yang]]></itunes:name></itunes:owner><itunes:author><![CDATA[Guanqun (David) Yang]]></itunes:author><googleplay:owner><![CDATA[guanqunyang@substack.com]]></googleplay:owner><googleplay:email><![CDATA[guanqunyang@substack.com]]></googleplay:email><googleplay:author><![CDATA[Guanqun (David) Yang]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Two Small Changes Help Recommenders Cope With Missing Content and Dislikes]]></title><description><![CDATA[Two CIKM 2026 papers: SMD keeps 61% of recommender accuracy when 95% of content is missing, and DualSpectralCF uses dislikes with no training.]]></description><link>https://newsletter.guanqun-yang.com/p/2026-10-19-recommenders-missing-content-dislikes</link><guid isPermaLink="false">https://newsletter.guanqun-yang.com/p/2026-10-19-recommenders-missing-content-dislikes</guid><dc:creator><![CDATA[Guanqun (David) Yang]]></dc:creator><pubDate>Mon, 19 Oct 2026 11:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/bc60ddf3-348e-41df-a58c-339c17c2801c_1456x763.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Recommender systems in online stores rarely get clean data. Many products have no photo or no description, and the dislikes that users leave often go unused. I led two papers that each address one of these gaps with a small change to existing recommenders. Both were accepted at CIKM 2026, the ACM International Conference on Information and Knowledge Management, held in Rome from 7 to 11 November 2026.</p><ul><li><p>SMD, short for Sequential Modality Dropout and written with Wenlong Zhang, is a four-line training change. When 95% of images and texts were removed at random, a recommender trained with SMD kept 61% of its accuracy, against 22% without it.</p></li><li><p>DualSpectralCF, written with Tong Qi of the University of Maryland and Xiaoxue Han of Stevens Institute of Technology, feeds dislikes into recommenders that need no training. On average, it ran 11 times faster than a trained model that also uses dislikes and reached 82% of its accuracy.</p></li></ul><p>Both papers fall outside my dissertation. They test the same question it does, which is whether a small, cheap change can come close to a costlier one.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!O7IV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!O7IV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!O7IV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!O7IV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!O7IV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two small changes for recommenders with imperfect data. SMD, for missing photos and text, keeps 61% of accuracy when 95% of images and texts are missing, against 22% without SMD. DualSpectralCF, for dislikes, runs 11 times faster on average than a trained model that also uses dislikes, at 82% of its accuracy. Both papers appear at CIKM 2026. Results come from offline benchmarks on Amazon and other public data.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two small changes for recommenders with imperfect data. SMD, for missing photos and text, keeps 61% of accuracy when 95% of images and texts are missing, against 22% without SMD. DualSpectralCF, for dislikes, runs 11 times faster on average than a trained model that also uses dislikes, at 82% of its accuracy. Both papers appear at CIKM 2026. Results come from offline benchmarks on Amazon and other public data." title="Two small changes for recommenders with imperfect data. SMD, for missing photos and text, keeps 61% of accuracy when 95% of images and texts are missing, against 22% without SMD. DualSpectralCF, for dislikes, runs 11 times faster on average than a trained model that also uses dislikes, at 82% of its accuracy. Both papers appear at CIKM 2026. Results come from offline benchmarks on Amazon and other public data." srcset="https://substackcdn.com/image/fetch/$s_!O7IV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!O7IV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!O7IV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!O7IV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ee1dd6-f412-4d5b-b5df-a9ae35b7b206_1456x792.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><p><strong>In this post</strong></p><ol><li><p>Why Recommenders Struggle With Real Product Data</p></li><li><p>SMD: Training for Missing Photos and Text</p><ul><li><p>2.1 Hide a Whole Kind of Content During Training</p></li><li><p>2.2 Accuracy Held Up as Content Went Missing</p></li><li><p>2.3 Where SMD Stops</p></li></ul></li><li><p>DualSpectralCF: Using Dislikes Without Training</p><ul><li><p>3.1 A Dislike Counts as Half a Like</p></li><li><p>3.2 Better Than Its Base, and Much Faster Than a Trained Model</p></li><li><p>3.3 New Users Gain the Most</p></li><li><p>3.4 Where DualSpectralCF Stops</p></li></ul></li><li><p>What It Means for Your Team</p></li><li><p>Questions Readers Ask</p></li><li><p>Read and Cite the Papers</p></li></ol><h2>1. Why Recommenders Struggle With Real Product Data</h2><p>A recommender predicts what a user will want next from what that user did before. Many recommenders also read each product&#8217;s photo and text. Real catalogs have gaps in both, according to counts in the SMD paper:</p><ul><li><p>In the public <a href="https://amazon-reviews-2023.github.io/">Amazon Reviews 2023</a> data, 34.3% of Toys &amp; Games items and 48.3% of Beauty &amp; Personal Care items have no text description.</p></li><li><p>In three of the four Amazon store categories used to test SMD, 25.7% to 40.9% of items have no image.</p></li></ul><p>Most recommenders are trained and tested on data where every item has every kind of content. In the SMD experiments, a standard recommender kept only 40% to 73% of its accuracy when product text was missing at serving time, depending on the store category.</p><p>Dislikes are the second gap. Many platforms record negative signals, such as low ratings. The cheapest recommenders, which need no training, use only positive signals such as purchases and likes. The recommenders that do use dislikes must be trained, which costs far more computing time.</p><h2>2. SMD: Training for Missing Photos and Text</h2><h3>2.1 Hide a Whole Kind of Content During Training</h3><p><a href="https://arxiv.org/abs/2608.10240">SMD</a> changes how a recommender is trained and leaves the model itself as it is. It works with any recommender that combines an item&#8217;s ID, photo, and text in one step. The procedure has four steps:</p><ol><li><p>For each user in a training batch, flip two coins, one for photos and one for text, each with a 30% chance.</p></li><li><p>If the photo coin comes up, hide every photo in that user&#8217;s history. The text coin works the same way.</p></li><li><p>Train with the same settings as before.</p></li><li><p>Serve the trained model unchanged, with nothing hidden and no added cost.</p></li></ol><p>SMD hides content for a user&#8217;s whole history at once, because real gaps tend to cover a whole category or data feed. The change takes four lines of code. The authors note that random masking of content is an established training tool in other fields and do not claim it as new. What they add is masking whole user histories in sequential recommendation, the task of predicting a user&#8217;s next item from the order of past items. They also test it on four different recommenders.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fofY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 424w, https://substackcdn.com/image/fetch/$s_!fofY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 848w, https://substackcdn.com/image/fetch/$s_!fofY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 1272w, https://substackcdn.com/image/fetch/$s_!fofY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!fofY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;SMD hides a whole kind of content for a user&#8217;s entire history, during training only. Each item in a user&#8217;s history has a photo and a text description. One coin flip for photos and one for text, each with a 30% chance, decide what the model sees. If the text coin comes up, all four items in that history lose their text and keep their photos. Real gaps come in blocks, where a whole category or data feed lacks photos or text. The change is four lines of training code and adds no cost when the model serves recommendations. Source: SMD, CIKM 2026.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="SMD hides a whole kind of content for a user&#8217;s entire history, during training only. Each item in a user&#8217;s history has a photo and a text description. One coin flip for photos and one for text, each with a 30% chance, decide what the model sees. If the text coin comes up, all four items in that history lose their text and keep their photos. Real gaps come in blocks, where a whole category or data feed lacks photos or text. The change is four lines of training code and adds no cost when the model serves recommendations. Source: SMD, CIKM 2026." title="SMD hides a whole kind of content for a user&#8217;s entire history, during training only. Each item in a user&#8217;s history has a photo and a text description. One coin flip for photos and one for text, each with a 30% chance, decide what the model sees. If the text coin comes up, all four items in that history lose their text and keep their photos. Real gaps come in blocks, where a whole category or data feed lacks photos or text. The change is four lines of training code and adds no cost when the model serves recommendations. Source: SMD, CIKM 2026." srcset="https://substackcdn.com/image/fetch/$s_!fofY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 424w, https://substackcdn.com/image/fetch/$s_!fofY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 848w, https://substackcdn.com/image/fetch/$s_!fofY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 1272w, https://substackcdn.com/image/fetch/$s_!fofY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F232db06d-bf0f-40af-8720-dd661a164775_1456x1990.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>2.2 Accuracy Held Up as Content Went Missing</h3><p>The paper measures accuracy with HR@10, the share of users whose true next item appears in the recommender&#8217;s top 10 picks out of the whole catalog, where higher is better. Retention is the share of full accuracy that a recommender keeps when content is missing, also higher is better. The main recommender in the tests is MM-SASRec, a standard self-attention model that adds photo and text features to each item.</p><ul><li><p>With all text removed in the Amazon Scientific category, MM-SASRec kept 83% of its accuracy with SMD and 40% without. The share of users whose next item reached the top 10 rose from 2.45% to 5.22%.</p></li><li><p>On four different recommenders in that category, retention with text removed rose in every case. The largest gain was for <a href="https://arxiv.org/abs/2404.02059">IISAN</a>, from 18% to 56%.</p></li><li><p>When each photo and each text was removed at random with a 95% chance, MM-SASRec kept 61% of its accuracy with SMD and 22% without.</p></li><li><p>With all content present, accuracy changed by +0.8% on average across 11 combinations of recommender and store category. Six of the 11 lost some accuracy, by at most 4.4%.</p></li><li><p>In 22 of 24 per-user statistical tests, the difference between training with and without SMD was statistically significant, meaning unlikely to come from chance.</p></li></ul><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zPWq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 424w, https://substackcdn.com/image/fetch/$s_!zPWq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 848w, https://substackcdn.com/image/fetch/$s_!zPWq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 1272w, https://substackcdn.com/image/fetch/$s_!zPWq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!zPWq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The more content is missing, the more SMD helps. Share of full accuracy kept as each item&#8217;s image and text are each removed at random. At a 95% chance of removal, MM-SASRec keeps 61% of its accuracy with SMD and 22% without. The gap between the two curves widens steadily from 0% to 95%. Accuracy is HR@10, on Amazon Scientific, mean of 5 random seeds. Source: SMD, CIKM 2026.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The more content is missing, the more SMD helps. Share of full accuracy kept as each item&#8217;s image and text are each removed at random. At a 95% chance of removal, MM-SASRec keeps 61% of its accuracy with SMD and 22% without. The gap between the two curves widens steadily from 0% to 95%. Accuracy is HR@10, on Amazon Scientific, mean of 5 random seeds. Source: SMD, CIKM 2026." title="The more content is missing, the more SMD helps. Share of full accuracy kept as each item&#8217;s image and text are each removed at random. At a 95% chance of removal, MM-SASRec keeps 61% of its accuracy with SMD and 22% without. The gap between the two curves widens steadily from 0% to 95%. Accuracy is HR@10, on Amazon Scientific, mean of 5 random seeds. Source: SMD, CIKM 2026." srcset="https://substackcdn.com/image/fetch/$s_!zPWq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 424w, https://substackcdn.com/image/fetch/$s_!zPWq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 848w, https://substackcdn.com/image/fetch/$s_!zPWq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 1272w, https://substackcdn.com/image/fetch/$s_!zPWq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad97a097-8f64-4941-a3b3-0b507343d0dc_1456x1541.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>2.3 Where SMD Stops</h3><p>SMD has these limits:</p><ul><li><p>It covers only photos and text that are missing, not content that is wrong or of other kinds such as video.</p></li><li><p>The 30% chance was compared only with 50%, on one recommender.</p></li><li><p>Of the 16 possible combinations of four recommenders and four store categories, 11 were run, because of limited computing time.</p></li><li><p>The largest gains in robustness came with the largest losses in full-content accuracy.</p></li><li><p>All results come from offline benchmarks. No live test in a store is reported.</p></li></ul><h2>3. DualSpectralCF: Using Dislikes Without Training</h2><h3>3.1 A Dislike Counts as Half a Like</h3><p><a href="https://arxiv.org/abs/2608.10247">DualSpectralCF</a> builds on spectral collaborative filtering, a family of recommenders that needs no training. These methods treat a user&#8217;s history as a signal spread over a graph of item similarity. A fixed formula smooths that signal and keeps the patterns that many users share. <a href="https://arxiv.org/abs/2505.00552">ChebyCF</a>, <a href="https://arxiv.org/abs/2108.07567">GF-CF</a>, and <a href="https://arxiv.org/abs/2404.14243">Turbo-CF</a> are three such recommenders. They are fast, but they ignore dislikes. DualSpectralCF adds two settings to any of them:</p><ul><li><p>Each disliked item enters the user&#8217;s history with its own weight, next to liked items at +1.</p></li><li><p>Items that users often dislike together get an extra link in the item graph, and the second setting controls its strength.</p></li></ul><p>The main surprise concerns the first setting. The best weight for a dislike was never negative, on any recommender or dataset. At the default, a disliked item counts as half a like. The authors explain that a one-star rating of a horror film still shows an interest in horror. Treating dislikes as true negatives did worse than ignoring them in every case tested.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!iGHq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 424w, https://substackcdn.com/image/fetch/$s_!iGHq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 848w, https://substackcdn.com/image/fetch/$s_!iGHq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 1272w, https://substackcdn.com/image/fetch/$s_!iGHq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!iGHq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;DualSpectralCF counts a dislike as half a like. In one user&#8217;s input row, liked items get +1, a disliked item gets +0.5 at the default setting, and items not seen get 0. With no training, a fixed formula spreads the row over a graph of item similarity, and items that users often dislike together can also count against each other. The output is a ranked list of items the user has not seen. A one-star rating of a horror film still shows an interest in horror, and treating dislikes as real negatives did worse than ignoring them. Source: DualSpectralCF, CIKM 2026.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="DualSpectralCF counts a dislike as half a like. In one user&#8217;s input row, liked items get +1, a disliked item gets +0.5 at the default setting, and items not seen get 0. With no training, a fixed formula spreads the row over a graph of item similarity, and items that users often dislike together can also count against each other. The output is a ranked list of items the user has not seen. A one-star rating of a horror film still shows an interest in horror, and treating dislikes as real negatives did worse than ignoring them. Source: DualSpectralCF, CIKM 2026." title="DualSpectralCF counts a dislike as half a like. In one user&#8217;s input row, liked items get +1, a disliked item gets +0.5 at the default setting, and items not seen get 0. With no training, a fixed formula spreads the row over a graph of item similarity, and items that users often dislike together can also count against each other. The output is a ranked list of items the user has not seen. A one-star rating of a horror film still shows an interest in horror, and treating dislikes as real negatives did worse than ignoring them. Source: DualSpectralCF, CIKM 2026." srcset="https://substackcdn.com/image/fetch/$s_!iGHq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 424w, https://substackcdn.com/image/fetch/$s_!iGHq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 848w, https://substackcdn.com/image/fetch/$s_!iGHq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 1272w, https://substackcdn.com/image/fetch/$s_!iGHq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2d364c6-fc57-44a8-ae3e-84d21ab06a51_1456x1804.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>3.2 Better Than Its Base, and Much Faster Than a Trained Model</h3><p>The tests use five public datasets with both likes and dislikes, released with <a href="https://arxiv.org/abs/2404.11982">SIGformer</a>, a trained recommender that uses dislikes. Two come from Amazon, one from the review site Epinions, and two from the short-video app Kuaishou. The paper measures accuracy with Recall@20: of the items a user liked in held-out test data, the share that appear in the recommender&#8217;s top 20 picks, where higher is better. All changes below are relative to the base recommender&#8217;s score.</p><ul><li><p>Each of the three DualSpectralCF versions matched or beat its base recommender on all five datasets.</p></li><li><p>At the default setting, the version built on ChebyCF gained 3.3% to 16.0% on four datasets. On the fifth, KuaiRec, the default hurt. KuaiRec labels a video as disliked from short viewing time, and there the best setting gave dislikes no weight and gained 1.9%.</p></li><li><p>The largest gain was 32.6%, for the version built on Turbo-CF on the KuaiRand dataset.</p></li><li><p>The ChebyCF version beat <a href="https://arxiv.org/abs/2002.02126">LightGCN</a>, a trained graph recommender, on four of five datasets.</p></li><li><p>Against SIGformer, the ChebyCF version reached 71% to 91% of its Recall@20. It ran 7.7 to 155 times faster per dataset, 11 times on average, on one GPU.</p></li></ul><p>The fastest version, built on GF-CF, averaged 8.5 seconds per dataset against SIGformer&#8217;s 270 seconds. Using dislikes added 16% to 59% to the run time of the base recommender.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UeMo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 424w, https://substackcdn.com/image/fetch/$s_!UeMo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 848w, https://substackcdn.com/image/fetch/$s_!UeMo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 1272w, https://substackcdn.com/image/fetch/$s_!UeMo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!UeMo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Most of the accuracy for a small share of the time. Average over five datasets, with accuracy shown as Recall@20 relative to SIGformer. The three DualSpectralCF versions take 8.5 to 23.8 seconds and reach 75% to 82% of SIGformer&#8217;s accuracy. Training-free recommenders that use only likes take 6.8 to 83.7 seconds and reach 66% to 77%. SIGformer, trained, takes 270 seconds, and LightGCN, trained, takes about 1,806 seconds at 73%. Source: DualSpectralCF, CIKM 2026.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Most of the accuracy for a small share of the time. Average over five datasets, with accuracy shown as Recall@20 relative to SIGformer. The three DualSpectralCF versions take 8.5 to 23.8 seconds and reach 75% to 82% of SIGformer&#8217;s accuracy. Training-free recommenders that use only likes take 6.8 to 83.7 seconds and reach 66% to 77%. SIGformer, trained, takes 270 seconds, and LightGCN, trained, takes about 1,806 seconds at 73%. Source: DualSpectralCF, CIKM 2026." title="Most of the accuracy for a small share of the time. Average over five datasets, with accuracy shown as Recall@20 relative to SIGformer. The three DualSpectralCF versions take 8.5 to 23.8 seconds and reach 75% to 82% of SIGformer&#8217;s accuracy. Training-free recommenders that use only likes take 6.8 to 83.7 seconds and reach 66% to 77%. SIGformer, trained, takes 270 seconds, and LightGCN, trained, takes about 1,806 seconds at 73%. Source: DualSpectralCF, CIKM 2026." srcset="https://substackcdn.com/image/fetch/$s_!UeMo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 424w, https://substackcdn.com/image/fetch/$s_!UeMo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 848w, https://substackcdn.com/image/fetch/$s_!UeMo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 1272w, https://substackcdn.com/image/fetch/$s_!UeMo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0eb46fa-f99e-4f3d-bb2a-d62b6e9fa438_1456x1555.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>3.3 New Users Gain the Most</h3><p>Dislikes helped most for users with little history. For users with 1 to 5 items in their training history, the ChebyCF version raised Recall@20 by 2.2% to 29.2%, depending on the dataset. For the most active users, with 100 or more items, the change turned negative, a drop of up to 11.1% on KuaiRec.</p><h3>3.4 Where DualSpectralCF Stops</h3><p>DualSpectralCF has these limits:</p><ul><li><p>The two settings are global constants. One weight for all users can penalize active users whose likes and dislikes span different topics.</p></li><li><p>On noisy dislike data, such as KuaiRec&#8217;s, the default setting hurt.</p></li><li><p>It still trails SIGformer, at 71% to 91% of its accuracy.</p></li><li><p>All results come from offline benchmarks. No live test is reported.</p></li></ul><h2>4. What It Means for Your Team</h2><p>Both papers point to practices for teams that run recommenders:</p><ol><li><p>Test your recommender with photos or text removed, and measure how much accuracy it keeps.</p></li><li><p>If your catalog has gaps, try training with whole histories masked. It costs one retraining and nothing at serving time.</p></li><li><p>Keep dislikes in your logs, and try them as weak positive signals before treating them as negatives.</p></li><li><p>Before paying for a trained model, measure a training-free recommender. DualSpectralCF reached 71% to 91% of a trained model&#8217;s accuracy in a small share of its time.</p></li><li><p>Check results by user group. Dislikes helped new users most and hurt the most active ones on some datasets.</p></li></ol><h2>5. Questions Readers Ask</h2><p><strong>How can a recommender keep working when product photos or descriptions are missing?</strong></p><p>One option is to train it with that content hidden. SMD hides all photos or all text for a user&#8217;s whole history at random during training. With 95% of images and texts missing at random, a recommender trained this way kept 61% of its accuracy, against 22% without.</p><p><strong>Can a recommender use thumbs-down ratings without training a model?</strong></p><p>Yes. DualSpectralCF adds dislikes to recommenders that need no training. It matched or beat its base recommender on all five benchmarks tested and ran 11 times faster on average than SIGformer, a trained model that uses dislikes.</p><p><strong>Should a recommender treat a disliked item as a negative signal?</strong></p><p>In these experiments, it should not. The best weight for a dislike was never negative, and treating dislikes as true negatives did worse than ignoring them. A dislike still shows interest in a topic.</p><p><strong>Is the code available?</strong></p><p>Yes. The code for <a href="https://github.com/guanqun-yang/SMD">SMD</a> and for <a href="https://github.com/guanqun-yang/DualSpectralCF">DualSpectralCF</a> is on GitHub.</p><h2>6. Read and Cite the Papers</h2><ul><li><p>Guanqun Yang and Wenlong Zhang. &#8220;Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation.&#8221; Accepted at CIKM 2026. <a href="https://arxiv.org/abs/2608.10240">arXiv</a>, <a href="https://github.com/guanqun-yang/SMD">code</a>.</p></li><li><p>Guanqun Yang, Tong Qi, and Xiaoxue Han. &#8220;DualSpectralCF: Training-Free Sign-Aware Spectral Collaborative Filtering.&#8221; Accepted at CIKM 2026. <a href="https://arxiv.org/abs/2608.10247">arXiv</a>, <a href="https://github.com/guanqun-yang/DualSpectralCF">code</a>.</p></li><li><p><a href="https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security">The overview post</a> places both papers next to my dissertation work on AI for software testing and security.</p></li></ul><p>To cite the papers:</p><pre><code>@misc{yang2026sequentialmodalitydropoutrobust,
  title         = {Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation},
  author        = {Guanqun Yang and Wenlong Zhang},
  year          = {2026},
  eprint        = {2608.10240},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  note          = {Accepted at CIKM 2026},
  url           = {https://arxiv.org/abs/2608.10240}
}

@misc{yang2026dualspectralcftrainingfreesignawarespectral,
  title         = {DualSpectralCF: Training-Free Sign-Aware Spectral Collaborative Filtering},
  author        = {Guanqun Yang and Tong Qi and Xiaoxue Han},
  year          = {2026},
  eprint        = {2608.10247},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  note          = {Accepted at CIKM 2026},
  url           = {https://arxiv.org/abs/2608.10247}
}
</code></pre><div><hr></div><p><em>Guanqun Yang, LLM Researcher. More work at <a href="https://guanqun-yang.com">guanqun-yang.com</a> and on <a href="https://scholar.google.com/citations?user=4XUIHW0AAAAJ">Google Scholar</a>.</em></p>]]></content:encoded></item><item><title><![CDATA[Measured Feedback Helps Weaker AI Coding Agents and Can Hurt Stronger Ones]]></title><description><![CDATA[WatchPoint measures the running web app after a failed test and gives raw results to the AI coding agent: 57.6% of failed tasks fixed on retry, against 45.1%.]]></description><link>https://newsletter.guanqun-yang.com/p/2026-10-18-watchpoint-measured-feedback-ai-coding-agents</link><guid isPermaLink="false">https://newsletter.guanqun-yang.com/p/2026-10-18-watchpoint-measured-feedback-ai-coding-agents</guid><dc:creator><![CDATA[Guanqun (David) Yang]]></dc:creator><pubDate>Sun, 18 Oct 2026 11:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/502f5116-77da-4e31-9770-cebdbd3b0592_1456x763.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>When an AI coding agent&#8217;s code fails a test, the agent usually sees only the test&#8217;s error message. A human developer would open the app, click around, and check what the page actually does. WatchPoint, a system I led, gives the coding agent those measurements. With them, a coding agent fixed 57.6% of its failed tasks on its one retry, against 45.1% with test errors alone.</p><p>The same feedback did not help a stronger coding agent. Its share of tasks passed within two attempts fell from 44.5% to 37.7%. WatchPoint is an arXiv preprint posted on 10 August 2026 by Guanqun Yang, Wei Yang of the University of Texas at Dallas, and Xueqing Liu of Stevens Institute of Technology. It is the fourth and last step of my dissertation, where a language model works alongside another agent.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!P9Xd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!P9Xd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!P9Xd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!P9Xd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!P9Xd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;With test errors only, the MiniMax M2.7 coding model fixed 45.1% of its failed tasks on the one retry. With test errors plus WatchPoint&#8217;s measurements, it fixed 57.6%, or 38 of 66 retried tasks, on the 28 Web-Bench projects where WatchPoint runs. Source: WatchPoint, arXiv 2609.26204.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="With test errors only, the MiniMax M2.7 coding model fixed 45.1% of its failed tasks on the one retry. With test errors plus WatchPoint&#8217;s measurements, it fixed 57.6%, or 38 of 66 retried tasks, on the 28 Web-Bench projects where WatchPoint runs. Source: WatchPoint, arXiv 2609.26204." title="With test errors only, the MiniMax M2.7 coding model fixed 45.1% of its failed tasks on the one retry. With test errors plus WatchPoint&#8217;s measurements, it fixed 57.6%, or 38 of 66 retried tasks, on the 28 Web-Bench projects where WatchPoint runs. Source: WatchPoint, arXiv 2609.26204." srcset="https://substackcdn.com/image/fetch/$s_!P9Xd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!P9Xd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!P9Xd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!P9Xd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f73afcc-51be-46eb-947e-79e2d7e2e976_1456x792.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><p><strong>In this post</strong></p><ol><li><p>Why Test Errors Are Poor Feedback for Coding Agents</p></li><li><p>How WatchPoint Measures the Running App</p><ul><li><p>2.1 Two Scripts Run Once After Each Failure</p></li><li><p>2.2 Raw Measurements Go Straight to the Coding Agent</p></li></ul></li><li><p>What the Results Show</p><ul><li><p>3.1 More Failed Tasks Passed on the Retry</p></li><li><p>3.2 The Gains Came From State-Management Projects</p></li><li><p>3.3 The Stronger Coding Model Did Worse With the Feedback</p></li><li><p>3.4 A Model That Interpreted the Results Invented Errors</p></li></ul></li><li><p>How WatchPoint Compared With Two Human Testers</p></li><li><p>Where the Results Stop</p></li><li><p>What It Means for Your Team</p></li><li><p>Questions Readers Ask</p></li><li><p>Read and Cite the Paper</p></li></ol><h2>1. Why Test Errors Are Poor Feedback for Coding Agents</h2><p>A coding agent is a language model that runs tools, such as a terminal and a file editor, in a loop until a task is done. Some teams split software work across several such agents, with one that plans, one that writes code, and one that tests. The messages between them often go wrong. Across 1,642 annotated runs of multi-agent systems, <a href="https://arxiv.org/abs/2503.13657">Cemri and colleagues</a> traced 32.3% of the failures to agents that misunderstood each other.</p><p>According to the WatchPoint paper, the feedback that coding agents usually receive takes four main forms:</p><ul><li><p>the raw error message from a failed test</p></li><li><p>a screenshot, sometimes scored by a vision model</p></li><li><p>a score or opinion from another language model, often called LLM-as-a-judge</p></li><li><p>a correction written in plain language</p></li></ul><p>The authors argue that few of these approaches use the live application the way a developer does, and that most were tested on single-page demo sites.</p><p>WatchPoint was tested on <a href="https://arxiv.org/abs/2505.07473">Web-Bench</a>, a 2025 benchmark of 50 real web projects with 20 tasks each, 1,000 tasks in total. According to its authors, engineers with 5 to 10 years of experience designed the projects, and one project takes a senior engineer 4 to 8 hours. Each task builds on the earlier ones and is checked by end-to-end tests, which open the real app in a browser and check what it does. The best agent in the WatchPoint paper passes 25.0% of tasks on the first attempt.</p><h2>2. How WatchPoint Measures the Running App</h2><h3>2.1 Two Scripts Run Once After Each Failure</h3><p>WatchPoint works with a coding agent built on <a href="https://github.com/OpenHands/OpenHands">OpenHands</a>, an open-source platform for software agents. The paper calls WatchPoint&#8217;s role a simulated user, a model that plays the person testing the app. The loop has five steps:</p><ol><li><p>The coding agent writes the next feature, using up to 30 model calls.</p></li><li><p>Web-Bench&#8217;s end-to-end tests run. If they pass, the next task begins.</p></li><li><p>If they fail, a browser agent and a terminal agent each write one script with a single model call. Both scripts run once, in parallel.</p></li><li><p>The coding agent retries once, in the same conversation, with the test errors plus one line per finding from the scripts.</p></li><li><p>If the retry also fails, the rest of the project stops, because later tasks depend on this one.</p></li></ol><p>The two scripts see different failures. The browser script uses <a href="https://playwright.dev/">Playwright</a>, Microsoft&#8217;s library for controlling a browser from code, to open the page, click, fill forms, and read styles, text, and page structure. The terminal script reads files, starts the server and sends it requests, and runs a type check. As the authors note, a browser cannot detect a missing software dependency, and a terminal cannot detect a broken layout.</p><p>WatchPoint switches on only for three kinds of project: plain web-standard code, user interface frameworks such as React, and state-management libraries such as Redux. It decides from each project&#8217;s list of software dependencies, the same signal that hosting services such as Vercel and Netlify use to detect frameworks. Across the 50 projects, it made the right on-or-off choice 92% of the time. Each diagnosis takes about 15 seconds, and the diagnostic calls added less than 2% to the total API cost.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QiZ6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 424w, https://substackcdn.com/image/fetch/$s_!QiZ6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 848w, https://substackcdn.com/image/fetch/$s_!QiZ6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 1272w, https://substackcdn.com/image/fetch/$s_!QiZ6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!QiZ6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;What happens when the coding agent&#8217;s code fails the tests. Step 1: the coding agent writes the next feature, with up to 30 model calls. Step 2: the end-to-end tests fail. Step 3: WatchPoint measures the running app. A browser agent writes one browser script that clicks, fills forms, and reads styles, text, and page structure. A terminal agent writes one terminal script that reads files, calls the server, and runs a type check. Both run once, in parallel, in about 15 seconds, and print one line per finding, such as CHECK operator_color: #00ff00. Step 4: the coding agent retries once in the same conversation, with the test errors plus the CHECK lines. If the retry passes, the next task begins. If it fails again, the rest of the project stops.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="What happens when the coding agent&#8217;s code fails the tests. Step 1: the coding agent writes the next feature, with up to 30 model calls. Step 2: the end-to-end tests fail. Step 3: WatchPoint measures the running app. A browser agent writes one browser script that clicks, fills forms, and reads styles, text, and page structure. A terminal agent writes one terminal script that reads files, calls the server, and runs a type check. Both run once, in parallel, in about 15 seconds, and print one line per finding, such as CHECK operator_color: #00ff00. Step 4: the coding agent retries once in the same conversation, with the test errors plus the CHECK lines. If the retry passes, the next task begins. If it fails again, the rest of the project stops." title="What happens when the coding agent&#8217;s code fails the tests. Step 1: the coding agent writes the next feature, with up to 30 model calls. Step 2: the end-to-end tests fail. Step 3: WatchPoint measures the running app. A browser agent writes one browser script that clicks, fills forms, and reads styles, text, and page structure. A terminal agent writes one terminal script that reads files, calls the server, and runs a type check. Both run once, in parallel, in about 15 seconds, and print one line per finding, such as CHECK operator_color: #00ff00. Step 4: the coding agent retries once in the same conversation, with the test errors plus the CHECK lines. If the retry passes, the next task begins. If it fails again, the rest of the project stops." srcset="https://substackcdn.com/image/fetch/$s_!QiZ6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 424w, https://substackcdn.com/image/fetch/$s_!QiZ6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 848w, https://substackcdn.com/image/fetch/$s_!QiZ6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 1272w, https://substackcdn.com/image/fetch/$s_!QiZ6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cfdc4b6-ad97-4677-87c8-d6e923dc201b_1456x2667.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>2.2 Raw Measurements Go Straight to the Coding Agent</h3><p>Each script prints one line per finding, with a name and a measured value. WatchPoint passes these CHECK lines to the coding agent unchanged, and no model summarizes them. The paper states the rule as &#8220;the LLM decides what to measure but never interprets what was measured.&#8221;</p><p>Here is a logged case from a project that builds a code editor with syntax highlighting. The first attempt failed this test:</p><pre><code>Error: expect(locator).toHaveText(expected)
  Locator:  locator('#editor')
  Expected: "foo AND bar"
  Received: "fooANDbar  "
</code></pre><p>The browser script then reported these lines, among others:</p><pre><code>CHECK editor_exists: True
CHECK after_fill_text: foo AND bar
CHECK highlight_function: 3
CHECK identifier_color: #0000ff
CHECK operator_color: #00ff00
</code></pre><p>On the retry, the coding agent compared its code with what the page showed. It noticed that it had built the highlighted text as one HTML string, which dropped the spaces between words. It switched to creating a separate page element for each word, and the task passed.</p><h2>3. What the Results Show</h2><h3>3.1 More Failed Tasks Passed on the Retry</h3><p>The main measure is the recovery rate: of the tasks that failed the first attempt and got a retry, the share that passed on the retry. Higher is better. The coding model here is MiniMax M2.7, and GLM-5 writes the diagnostic scripts.</p><ul><li><p>With WatchPoint, 38 of 66 retried tasks passed, a recovery rate of 57.6%.</p></li><li><p>With test errors alone, the same coding model recovered 45.1% on the same 28 projects.</p></li><li><p>On the 20 projects where WatchPoint never runs, recovery stayed the same, 45.9% against 46.2%.</p></li><li><p>At least one script produced a valid measurement for 59 of the 66 retried tasks.</p></li></ul><h3>3.2 The Gains Came From State-Management Projects</h3><p>A second measure is Pass@2, the share of all tasks passed within two attempts, where higher is better. Because a twice-failed task stops its project, one recovered task can let many later tasks run. In one project, a single recovered early task allowed 8 later tasks to run.</p><ul><li><p>In state-management projects, Pass@2 rose from 11.2% to 18.8%.</p></li><li><p>Across all 28 projects where WatchPoint runs, it rose from 21.8% to 24.5%.</p></li><li><p>Over all 1,000 tasks, including projects where WatchPoint never runs, it rose from 26.6% to 28.1%.</p></li></ul><h3>3.3 The Stronger Coding Model Did Worse With the Feedback</h3><p>The authors repeated the test with GLM-5 as the coding model, which passes far more tasks on its own. Pass@2 over all 1,000 tasks, where higher is better:</p><ul><li><p>MiniMax M2.7 reached 26.6% alone, 28.1% with GLM-5 as the tester, and 27.9% with GPT-5.4 as the tester.</p></li><li><p>GLM-5 reached 44.5% alone and 37.7% with GPT-5.4 as the tester.</p></li></ul><p>The authors caution that the drop for GLM-5 overlaps the normal variation between identical runs. Their reading is that the feedback adds clear value when the coding model is weak. A model that can already fix its own mistakes from test errors may get only noise from the extra input. A stronger tester did not help more either, since GPT-5.4 and GLM-5 gave the weaker coder similar results.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!I0C0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 424w, https://substackcdn.com/image/fetch/$s_!I0C0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 848w, https://substackcdn.com/image/fetch/$s_!I0C0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 1272w, https://substackcdn.com/image/fetch/$s_!I0C0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!I0C0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The feedback helped the weaker coder and hurt the stronger one. Share of all 1,000 Web-Bench tasks passed within two attempts, one run each. MiniMax M2.7: 26.6% with no feedback beyond test errors, 28.1% with WatchPoint and GLM-5 as the tester, 27.9% with WatchPoint and GPT-5.4 as the tester. GLM-5: 44.5% with no feedback beyond test errors, 37.7% with WatchPoint and GPT-5.4 as the tester. The authors note that the drop for the stronger model overlaps the normal variation between identical runs. Source: WatchPoint, arXiv 2609.26204.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The feedback helped the weaker coder and hurt the stronger one. Share of all 1,000 Web-Bench tasks passed within two attempts, one run each. MiniMax M2.7: 26.6% with no feedback beyond test errors, 28.1% with WatchPoint and GLM-5 as the tester, 27.9% with WatchPoint and GPT-5.4 as the tester. GLM-5: 44.5% with no feedback beyond test errors, 37.7% with WatchPoint and GPT-5.4 as the tester. The authors note that the drop for the stronger model overlaps the normal variation between identical runs. Source: WatchPoint, arXiv 2609.26204." title="The feedback helped the weaker coder and hurt the stronger one. Share of all 1,000 Web-Bench tasks passed within two attempts, one run each. MiniMax M2.7: 26.6% with no feedback beyond test errors, 28.1% with WatchPoint and GLM-5 as the tester, 27.9% with WatchPoint and GPT-5.4 as the tester. GLM-5: 44.5% with no feedback beyond test errors, 37.7% with WatchPoint and GPT-5.4 as the tester. The authors note that the drop for the stronger model overlaps the normal variation between identical runs. Source: WatchPoint, arXiv 2609.26204." srcset="https://substackcdn.com/image/fetch/$s_!I0C0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 424w, https://substackcdn.com/image/fetch/$s_!I0C0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 848w, https://substackcdn.com/image/fetch/$s_!I0C0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 1272w, https://substackcdn.com/image/fetch/$s_!I0C0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fb92090-3c3b-4564-9f54-d71c43be3edf_1456x1837.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>3.4 A Model That Interpreted the Results Invented Errors</h3><p>An earlier version of WatchPoint ran a fixed browser script and asked GLM-5 to interpret its output for the coding agent. On a calculator project, the interpreter reported a page titled &#8220;Blog&#8221; and a 504 server error, neither of which existed. The coding agent spent more than 40 steps chasing those invented problems, and tasks passed on the first attempt fell from 11 of 20 to 4 of 20. Passing the raw CHECK lines removed the failure.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CITa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 424w, https://substackcdn.com/image/fetch/$s_!CITa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 848w, https://substackcdn.com/image/fetch/$s_!CITa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 1272w, https://substackcdn.com/image/fetch/$s_!CITa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!CITa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Pass on the measurements, never a model&#8217;s reading of them. Kept design: the live app, a script that runs once, a raw finding such as after_fill_text: foo AND bar, then the coding agent. Rejected design: the live app, a fixed script whose output a model interprets, an invented report of a &#8220;Blog&#8221; page and a 504 server error, then the coding agent. On the calculator project, the coding agent spent more than 40 steps chasing server problems that did not exist, and tasks passed on the first attempt fell from 11 of 20 to 4 of 20. Source: WatchPoint, arXiv 2609.26204, appendix.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Pass on the measurements, never a model&#8217;s reading of them. Kept design: the live app, a script that runs once, a raw finding such as after_fill_text: foo AND bar, then the coding agent. Rejected design: the live app, a fixed script whose output a model interprets, an invented report of a &#8220;Blog&#8221; page and a 504 server error, then the coding agent. On the calculator project, the coding agent spent more than 40 steps chasing server problems that did not exist, and tasks passed on the first attempt fell from 11 of 20 to 4 of 20. Source: WatchPoint, arXiv 2609.26204, appendix." title="Pass on the measurements, never a model&#8217;s reading of them. Kept design: the live app, a script that runs once, a raw finding such as after_fill_text: foo AND bar, then the coding agent. Rejected design: the live app, a fixed script whose output a model interprets, an invented report of a &#8220;Blog&#8221; page and a 504 server error, then the coding agent. On the calculator project, the coding agent spent more than 40 steps chasing server problems that did not exist, and tasks passed on the first attempt fell from 11 of 20 to 4 of 20. Source: WatchPoint, arXiv 2609.26204, appendix." srcset="https://substackcdn.com/image/fetch/$s_!CITa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 424w, https://substackcdn.com/image/fetch/$s_!CITa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 848w, https://substackcdn.com/image/fetch/$s_!CITa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 1272w, https://substackcdn.com/image/fetch/$s_!CITa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62094ec6-2ca1-425a-b2b2-7d5bf0b6bec1_1456x1524.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h2>4. How WatchPoint Compared With Two Human Testers</h2><p>The authors also ran a small user study. Two graduate students with web development coursework took WatchPoint&#8217;s place. When a task failed, they opened the app in a browser and typed free-form feedback for the coding agent. They covered five small projects in plain HTML, CSS, and JavaScript, in sessions of about two hours each.</p><ul><li><p>Tester A recovered 4 of 9 failed tasks, 44%.</p></li><li><p>Tester B recovered 8 of 13, 62%.</p></li><li><p>WatchPoint recovered 57.6%, measured on a different and larger set of tasks.</p></li></ul><p>Some human feedback was vague, such as &#8220;I think it&#8217;s fine&#8221;. The authors call the comparison preliminary evidence, given two testers and the easiest kind of project for browser checks.</p><h2>5. Where the Results Stop</h2><p>The results have these limits:</p><ul><li><p>They come from one benchmark, Web-Bench, with one run per setting. Between two identical runs of the same coding model, 113 of 980 tasks changed outcome. The authors estimate that totals can differ by about 60 of the 1,000 tasks from chance alone.</p></li><li><p>The user study had two testers and five small projects.</p></li><li><p>The scripts can mislead. In one project, the terminal script checked the wrong network ports, and the coding agent spent 30 steps changing a build file. All 20 tasks in that project were skipped.</p></li><li><p>For 7 of the 66 retried tasks, the scripts produced no usable measurement.</p></li><li><p>WatchPoint is an arXiv preprint, posted on 10 August 2026.</p></li></ul><h2>6. What It Means for Your Team</h2><p>The study suggests five practices for teams that run AI coding agents:</p><ol><li><p>Give coding agents facts measured from the running app, next to the test errors.</p></li><li><p>Let a model write the measurement script, and pass its raw output to the coding agent. Do not let a second model summarize it.</p></li><li><p>Switch the diagnostics on only for project types where they help.</p></li><li><p>Test with and without the feedback on your own coding model. A model that already fixes most failures from test errors may do worse with more input.</p></li><li><p>Run each comparison more than once. Identical runs here differed by up to about 60 of 1,000 tasks.</p></li></ol><h2>7. Questions Readers Ask</h2><p><strong>How can an AI coding agent test the web app it just wrote?</strong></p><p>It can run a short script against the live app, in a browser and in a terminal, and read the measured results. In WatchPoint, a second model writes those scripts after each failed test. The coding agent then fixed 57.6% of its failed tasks on the retry, against 45.1% with test errors alone.</p><p><strong>Does more feedback always make an AI coding agent better?</strong></p><p>It did not in this study. With a stronger coding model, GLM-5, the share of tasks passed within two attempts fell from 44.5% to 37.7% when WatchPoint&#8217;s feedback was added. The authors note that the drop overlaps the normal variation between identical runs.</p><p><strong>Can an automated tester replace a human quality assurance tester for web apps?</strong></p><p>The evidence so far is preliminary. In a small study, two human testers recovered 44% and 62% of failed tasks. WatchPoint recovered 57.6% on a different, larger set of tasks.</p><p><strong>How much does it cost to add diagnostics like WatchPoint&#8217;s to a coding agent?</strong></p><p>In the paper, the diagnostics added less than 2% to the total API cost and about 15 seconds per failed task.</p><p><strong>What is Web-Bench?</strong></p><p>Web-Bench is a 2025 benchmark of 50 real web projects with 20 dependent tasks each, checked by end-to-end tests in a real browser. The best agent in the WatchPoint paper passes 25.0% of tasks on the first attempt.</p><h2>8. Read and Cite the Paper</h2><ul><li><p>Guanqun Yang, Wei Yang, and Xueqing Liu. &#8220;WatchPoint: Executable User Feedback for Real-World Agentic Web Development.&#8221; arXiv preprint, 2026. <a href="https://arxiv.org/abs/2609.26204">arXiv</a>.</p></li><li><p>WatchPoint is the fourth of four steps in my dissertation. <a href="https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security">The overview post</a> explains how the steps fit together. The earlier steps each have a post: <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-fewvuln-security-reports-fewer-labels">FewVuln</a> on reading vulnerability reports, <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-testaug-hatemoderate-ai-written-tests">TestAug and HateModerate</a> on writing tests, and <a href="https://newsletter.guanqun-yang.com/p/2026-10-17-patchholmes-ai-agent-finds-vulnerability-fixes">PatchHolmes</a> on finding vulnerability fixes.</p></li></ul><p>To cite the paper:</p><pre><code>@misc{yang2026watchpointexecutableuserfeedback,
  title         = {WatchPoint: Executable User Feedback for Real-World Agentic Web Development},
  author        = {Guanqun Yang and Wei Yang and Xueqing Liu},
  year          = {2026},
  eprint        = {2609.26204},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2609.26204}
}
</code></pre><div><hr></div><p><em>Guanqun Yang, LLM Researcher. More work at <a href="https://guanqun-yang.com">guanqun-yang.com</a> and on <a href="https://scholar.google.com/citations?user=4XUIHW0AAAAJ">Google Scholar</a>.</em></p>]]></content:encoded></item><item><title><![CDATA[An AI Agent Finds the Fix Behind 6 in 10 Known Vulnerabilities]]></title><description><![CDATA[PatchHolmes, an LLM agent, ranks the correct fix commit first for 60% of 809 CVEs, against 35% for Favia, at about USD 0.007 per vulnerability.]]></description><link>https://newsletter.guanqun-yang.com/p/2026-10-17-patchholmes-ai-agent-finds-vulnerability-fixes</link><guid isPermaLink="false">https://newsletter.guanqun-yang.com/p/2026-10-17-patchholmes-ai-agent-finds-vulnerability-fixes</guid><dc:creator><![CDATA[Guanqun (David) Yang]]></dc:creator><pubDate>Sat, 17 Oct 2026 11:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/1450bb0a-21f0-43a5-a325-a077db7f4be1_1456x763.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most public vulnerability records do not say which code change fixed the flaw. PatchHolmes, an LLM agent I co-led, finds that commit in the project&#8217;s Git history. On 809 CVEs, it ranks the correct fix first for 59.95% of them. Favia, the strongest earlier agent, does so for 34.61%. Each CVE costs about USD 0.007 to process.</p><p>PatchHolmes will appear at AACL-IJCNLP 2026, held in Hengqin, China, from 6 to 10 November 2026. The conference is the joint meeting of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing. PatchHolmes builds on SPFinder, an earlier system from our lab led by Jiangrui Zheng, which I co-authored. With SPFinder&#8217;s help, the team added fix links for 35 CVEs to the GitHub Advisory Database. PatchHolmes is the third step of my dissertation, where a language model acts with tools inside a code repository.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!l2g-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!l2g-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!l2g-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!l2g-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!l2g-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;PatchHolmes ranks the correct fix commit first for 59.95% of 809 vulnerabilities in the GitHubAD benchmark. Favia, an earlier agent, reaches 34.61%. Taking the top commit without an agent gives 32.63%, IRCoT 28.55%, and SPFinder 27.32%. All LLM methods use Qwen3-235B. Source: PatchHolmes, AACL-IJCNLP 2026.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="PatchHolmes ranks the correct fix commit first for 59.95% of 809 vulnerabilities in the GitHubAD benchmark. Favia, an earlier agent, reaches 34.61%. Taking the top commit without an agent gives 32.63%, IRCoT 28.55%, and SPFinder 27.32%. All LLM methods use Qwen3-235B. Source: PatchHolmes, AACL-IJCNLP 2026." title="PatchHolmes ranks the correct fix commit first for 59.95% of 809 vulnerabilities in the GitHubAD benchmark. Favia, an earlier agent, reaches 34.61%. Taking the top commit without an agent gives 32.63%, IRCoT 28.55%, and SPFinder 27.32%. All LLM methods use Qwen3-235B. Source: PatchHolmes, AACL-IJCNLP 2026." srcset="https://substackcdn.com/image/fetch/$s_!l2g-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!l2g-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!l2g-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!l2g-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff31e39dc-871e-40c0-a978-c6c5df812f25_1456x792.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><p><strong>In this post</strong></p><ol><li><p>Why Most Vulnerability Records Lack a Fix Link</p></li><li><p>SPFinder Shortlists the Candidates</p></li><li><p>PatchHolmes Picks the Fix</p><ul><li><p>3.1 A Shortlist, Then an Agent</p></li><li><p>3.2 Comparing the Whole List Beats Judging One Commit at a Time</p></li></ul></li><li><p>What the Results Show</p><ul><li><p>4.1 The Agent Ranks the Fix First 1.7 Times as Often as Favia</p></li><li><p>4.2 The Gain Comes From the Agent</p></li><li><p>4.3 A Small Model Is Almost as Good</p></li><li><p>4.4 The Gain Holds Outside the Main Benchmark</p></li><li><p>4.5 Each Vulnerability Costs Less Than One Cent</p></li></ul></li><li><p>Where the Results Stop</p></li><li><p>What It Means for Your Team</p></li><li><p>Questions Readers Ask</p></li><li><p>Read and Cite the Papers</p></li></ol><h2>1. Why Most Vulnerability Records Lack a Fix Link</h2><p>A CVE record describes a security flaw in text. Advisory databases, such as the <a href="https://github.com/advisories">GitHub Advisory Database</a> and the US National Vulnerability Database (NVD), add details to it, including a link to the commit that fixed the flaw. According to the SPFinder paper, that link is how a database works out which versions and packages are affected. GitHub&#8217;s <a href="https://docs.github.com/en/code-security/concepts/supply-chain-security/dependabot-alerts">Dependabot alerts</a>, which warn projects about vulnerable dependencies, also report the fixed version when one is known.</p><p>The link is usually missing. <a href="https://arxiv.org/abs/2311.01532">VFCFinder</a>, a 2024 study, found that 6,159 of 9,764 GitHub advisories, about 63%, had no link to the fixing commit. The gap can last for years. The NVD published CVE-2013-1814, a flaw in Apache Rave, on 14 March 2013. A maintainer traced its fix by hand on 5 January 2024. The <a href="https://github.com/advisories/GHSA-428j-q447-47rw">GitHub advisory</a> has listed it since that day. The NVD record still had no fix link on 15 October 2026.</p><p>Finding the fix automatically is hard for three reasons:</p><ul><li><p>The search space is large. In GitHubAD, a benchmark of CVEs from the GitHub Advisory Database, 49% of CVEs belong to repositories with more than 5,000 commits.</p></li><li><p>The changes are long. The SPFinder team measured average commit changes of about 12,000 to 19,000 tokens, depending on the dataset. Older text encoders read only the first 512.</p></li><li><p>The words differ. The CVE describes the symptom, and the commit describes the code. For CVE-2021-41079, a flaw in Apache Tomcat, the fix commit&#8217;s message is &#8220;Improving robustness&#8221;.</p></li></ul><h2>2. SPFinder Shortlists the Candidates</h2><p><a href="https://arxiv.org/abs/2503.22935">SPFinder</a>, short for Scalable Patch Finder, searches a project&#8217;s entire commit history. Earlier evaluations often scored a tool against the correct commit plus 5,000 sampled wrong ones. The SPFinder authors argue that this sampling makes accuracy look higher than in real use. SPFinder ranks commits in three phases:</p><ol><li><p>Rank every commit by keyword matches in its message and code, plus how close its date is to the CVE&#8217;s dates. The top 10,000 commits hold the fix for 95% of CVEs.</p></li><li><p>For those 10,000, compute 10 features. The main one embeds each changed file separately, so a long commit is not cut down to its first 512 tokens.</p></li><li><p>Order the commits with a learned ranking model, LightGBM, that combines the features.</p></li></ol><p>The measure here is Recall@10, the share of CVEs whose fix appears among the top 10 results, where higher is better. On the GitHubAD test set of 952 CVEs, SPFinder reached 57.7%. VoyageAI&#8217;s commercial embedding model, <a href="https://blog.voyageai.com/2024/09/18/voyage-3/">voyage-3</a>, reached 44.6%. The first stage of <a href="https://arxiv.org/abs/2407.17065">PatchFinder</a>, an earlier fix-finding tool, reached 5.4%. On a second test set of 629 CVEs, drawn from the PatchFinder authors&#8217; dataset, SPFinder reached 74.4% against 64.9% for voyage-3.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!tvvs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 424w, https://substackcdn.com/image/fetch/$s_!tvvs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 848w, https://substackcdn.com/image/fetch/$s_!tvvs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 1272w, https://substackcdn.com/image/fetch/$s_!tvvs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!tvvs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;SPFinder puts the fix among the top 10 results for 57.7% of 952 vulnerabilities in the GitHubAD test set, searching each project&#8217;s whole history. VoyageAI voyage-3, a commercial embedding model, reaches 44.6%. Keyword search plus commit dates reaches 8.5%, the first stage of PatchFinder 5.4%, and PatchScout 1.1%. Fix links for 35 vulnerabilities, found with SPFinder and checked by hand, were merged into the GitHub Advisory Database in May 2025. Source: SPFinder, arXiv 2503.22935.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="SPFinder puts the fix among the top 10 results for 57.7% of 952 vulnerabilities in the GitHubAD test set, searching each project&#8217;s whole history. VoyageAI voyage-3, a commercial embedding model, reaches 44.6%. Keyword search plus commit dates reaches 8.5%, the first stage of PatchFinder 5.4%, and PatchScout 1.1%. Fix links for 35 vulnerabilities, found with SPFinder and checked by hand, were merged into the GitHub Advisory Database in May 2025. Source: SPFinder, arXiv 2503.22935." title="SPFinder puts the fix among the top 10 results for 57.7% of 952 vulnerabilities in the GitHubAD test set, searching each project&#8217;s whole history. VoyageAI voyage-3, a commercial embedding model, reaches 44.6%. Keyword search plus commit dates reaches 8.5%, the first stage of PatchFinder 5.4%, and PatchScout 1.1%. Fix links for 35 vulnerabilities, found with SPFinder and checked by hand, were merged into the GitHub Advisory Database in May 2025. Source: SPFinder, arXiv 2503.22935." srcset="https://substackcdn.com/image/fetch/$s_!tvvs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 424w, https://substackcdn.com/image/fetch/$s_!tvvs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 848w, https://substackcdn.com/image/fetch/$s_!tvvs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 1272w, https://substackcdn.com/image/fetch/$s_!tvvs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff75eeca4-4677-4820-bd87-b7c4d33e7999_1456x1668.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><p>The team then used SPFinder on GitHub advisories that lacked a fix link. They checked the top 30 results by hand and proposed each fix as a pull request to the advisory database. GitHub merged all 35, covering 11 projects, between 8 and 27 May 2025. The <a href="https://github.com/AnonySE26/SPFinder/blob/master/githubAD_PRs.md">list of merged links</a> is public. One of them is CVE-2024-22031, a high-severity flaw in Rancher. Its fix commit, &#8220;Add project namespace handling&#8221;, changes 19 files and never mentions the flaw. The <a href="https://github.com/github/advisory-database/pull/5620">pull request</a> added it to the <a href="https://github.com/advisories/GHSA-8h6m-wv39-239m">advisory</a> on 23 May 2025.</p><h2>3. PatchHolmes Picks the Fix</h2><h3>3.1 A Shortlist, Then an Agent</h3><p><a href="https://arxiv.org/abs/2609.38807">PatchHolmes</a> takes a CVE description and a local copy of the project&#8217;s repository and returns one commit with a written reason. It works in two phases:</p><ol><li><p>Phase 1 builds a shortlist. One ranking uses keywords and dates, with SPFinder&#8217;s weights. A second ranking uses meaning, from the Qwen3-Embedding-8B embedding model. PatchHolmes merges the two and keeps the top 100 commits, which hold the fix for 87.64% of CVEs.</p></li><li><p>Phase 2 runs an LLM agent for at most 15 steps. The agent can see the whole list, open a commit, open one file&#8217;s changes, or submit an answer. A typical run opens 3 to 10 of the 100 commits, 5 on average.</p></li></ol><p>The agent uses Qwen3-235B, an open-weight model used without further training. PatchHolmes needs no paid search service. The authors argue that the cost of training per repository or paying per search would grow with every new CVE.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kEYK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 424w, https://substackcdn.com/image/fetch/$s_!kEYK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 848w, https://substackcdn.com/image/fetch/$s_!kEYK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 1272w, https://substackcdn.com/image/fetch/$s_!kEYK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!kEYK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;How PatchHolmes works. The input is a vulnerability report and the project&#8217;s Git repository. Phase 1 builds a cheap shortlist from two paths: keywords and dates, and meaning, where an embedding model compares the report with every commit. The merged ranking keeps the top 100 commits, which hold the fix for 88% of vulnerabilities. Phase 2 is an LLM agent that picks one commit in at most 15 steps, using four tools: see the list of all 100 candidates, open a commit, open one file&#8217;s full change, and submit one commit with the reason. A typical run opens 3 to 10 commits and costs about USD 0.007.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="How PatchHolmes works. The input is a vulnerability report and the project&#8217;s Git repository. Phase 1 builds a cheap shortlist from two paths: keywords and dates, and meaning, where an embedding model compares the report with every commit. The merged ranking keeps the top 100 commits, which hold the fix for 88% of vulnerabilities. Phase 2 is an LLM agent that picks one commit in at most 15 steps, using four tools: see the list of all 100 candidates, open a commit, open one file&#8217;s full change, and submit one commit with the reason. A typical run opens 3 to 10 commits and costs about USD 0.007." title="How PatchHolmes works. The input is a vulnerability report and the project&#8217;s Git repository. Phase 1 builds a cheap shortlist from two paths: keywords and dates, and meaning, where an embedding model compares the report with every commit. The merged ranking keeps the top 100 commits, which hold the fix for 88% of vulnerabilities. Phase 2 is an LLM agent that picks one commit in at most 15 steps, using four tools: see the list of all 100 candidates, open a commit, open one file&#8217;s full change, and submit one commit with the reason. A typical run opens 3 to 10 commits and costs about USD 0.007." srcset="https://substackcdn.com/image/fetch/$s_!kEYK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 424w, https://substackcdn.com/image/fetch/$s_!kEYK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 848w, https://substackcdn.com/image/fetch/$s_!kEYK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 1272w, https://substackcdn.com/image/fetch/$s_!kEYK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb744992a-629a-44a9-ad44-584deecb1cec_1456x2582.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>3.2 Comparing the Whole List Beats Judging One Commit at a Time</h3><p><a href="https://arxiv.org/abs/2602.12500">Favia</a>, a 2026 agent from another group, judges each candidate commit on its own. It asks &#8220;Is this the fix?&#8221; once per candidate, ten times per CVE. When several candidates look plausible, those separate answers give no way to compare them. PatchHolmes instead shows the agent all candidates at once and asks which one is the fix.</p><p>The jQuery flaw CVE-2015-9251 shows the difference. In jQuery before 3.0.0, a request to another website could run that site&#8217;s script. Both systems received the same 10 candidates, with the fix at position 9:</p><ul><li><p>Favia said yes to all 10 with the same top confidence and fell back to list order. It picked commit e7b3bc4, a 2020 change to how jQuery handles JSON data.</p></li><li><p>PatchHolmes read the list, then opened commits 1, 2, 9, 4, and 3. It picked commit b078a62, &#8220;Ajax: Mitigate possible XSS vulnerability&#8221;, from 10 September 2015. Only that commit stops scripts from other sites from running.</p></li></ul><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QQ3a!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 424w, https://substackcdn.com/image/fetch/$s_!QQ3a!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 848w, https://substackcdn.com/image/fetch/$s_!QQ3a!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 1272w, https://substackcdn.com/image/fetch/$s_!QQ3a!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!QQ3a!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The jQuery case, CVE-2015-9251. Both systems received the same 10 candidate commits, and the fix was number 9. Favia asks &#8220;Is this the fix?&#8221; ten separate times, says yes to all ten with the highest confidence, falls back to list order, and picks number 1, e7b3bc4, a 2020 change to JSON handling that does not fix the flaw. PatchHolmes reads all ten as one list, opens commits 1, 2, 9, 4, and 3, and picks number 9, b078a62, &#8220;Ajax: Mitigate possible XSS vulnerability&#8221; from 2015. One agent conversation replaced ten separate model calls.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The jQuery case, CVE-2015-9251. Both systems received the same 10 candidate commits, and the fix was number 9. Favia asks &#8220;Is this the fix?&#8221; ten separate times, says yes to all ten with the highest confidence, falls back to list order, and picks number 1, e7b3bc4, a 2020 change to JSON handling that does not fix the flaw. PatchHolmes reads all ten as one list, opens commits 1, 2, 9, 4, and 3, and picks number 9, b078a62, &#8220;Ajax: Mitigate possible XSS vulnerability&#8221; from 2015. One agent conversation replaced ten separate model calls." title="The jQuery case, CVE-2015-9251. Both systems received the same 10 candidate commits, and the fix was number 9. Favia asks &#8220;Is this the fix?&#8221; ten separate times, says yes to all ten with the highest confidence, falls back to list order, and picks number 1, e7b3bc4, a 2020 change to JSON handling that does not fix the flaw. PatchHolmes reads all ten as one list, opens commits 1, 2, 9, 4, and 3, and picks number 9, b078a62, &#8220;Ajax: Mitigate possible XSS vulnerability&#8221; from 2015. One agent conversation replaced ten separate model calls." srcset="https://substackcdn.com/image/fetch/$s_!QQ3a!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 424w, https://substackcdn.com/image/fetch/$s_!QQ3a!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 848w, https://substackcdn.com/image/fetch/$s_!QQ3a!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 1272w, https://substackcdn.com/image/fetch/$s_!QQ3a!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5108cb50-d515-4ec8-86d3-5388f67ddf8c_1456x1914.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h2>4. What the Results Show</h2><p>The main measure is Recall@1, the share of CVEs whose first answer is the correct fix, where higher is better. Unless noted, results come from 809 CVEs sampled from the 8,401 in GitHubAD. Every LLM method used the same model, Qwen3-235B.</p><h3>4.1 The Agent Ranks the Fix First 1.7 Times as Often as Favia</h3><ul><li><p>PatchHolmes reached 59.95% Recall@1, and Favia reached 34.61%. <a href="https://aclanthology.org/2023.acl-long.557/">IRCoT</a>, a general method that alternates search and reasoning, reached 28.55%. SPFinder reached 27.32%.</p></li><li><p>McNemar&#8217;s test, which checks whether two methods differ on the same CVEs, found the gap to Favia statistically significant, with p far below 0.001. A gap that large would almost never appear by chance if the two methods were equally good.</p></li><li><p>Within the top 10, the methods were closer. PatchHolmes found the fix there for 77.26% of CVEs, against 72.31% for Favia. The authors read this as Favia often having the fix in its list but not placing it first.</p></li></ul><h3>4.2 The Gain Comes From the Agent</h3><p>The authors held the 100 candidates fixed and changed only the selection step:</p><ul><li><p>Taking Phase 1&#8217;s top-ranked commit gave 32.63% Recall@1. The full agent gave 59.95%.</p></li><li><p>Given PatchHolmes&#8217;s own candidates, Favia reached 39.80% and IRCoT 37.58%.</p></li><li><p>An agent that could only see the list and submit reached 48.21%, the largest single gain. Opening commit summaries raised it to 57.48%, and reading code changes to 59.95%.</p></li><li><p>More text did not help. Giving the agent the full NVD report, 1,786 characters on average instead of about 150, left Recall@1 at 59.83%.</p></li></ul><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4Ctl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 424w, https://substackcdn.com/image/fetch/$s_!4Ctl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 848w, https://substackcdn.com/image/fetch/$s_!4Ctl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 1272w, https://substackcdn.com/image/fetch/$s_!4Ctl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!4Ctl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two findings. Finding 1: as tools are added to the agent, Recall@1 rises from 32.63% for the top commit without an agent, to 48.21% for an agent that can only see the list and submit, to 57.48% with commit summaries, and to 59.95% when it can also read code changes. Finding 2: with the model inside the agent swapped, Qwen3-235B with 22B active parameters reaches 59.95%, Qwen3-Coder-30B with 3B active parameters 59.09%, gpt-oss-120B 56.98%, and gpt-oss-20B 49.81%, against 32.63% with no agent. Source: PatchHolmes, AACL-IJCNLP 2026.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two findings. Finding 1: as tools are added to the agent, Recall@1 rises from 32.63% for the top commit without an agent, to 48.21% for an agent that can only see the list and submit, to 57.48% with commit summaries, and to 59.95% when it can also read code changes. Finding 2: with the model inside the agent swapped, Qwen3-235B with 22B active parameters reaches 59.95%, Qwen3-Coder-30B with 3B active parameters 59.09%, gpt-oss-120B 56.98%, and gpt-oss-20B 49.81%, against 32.63% with no agent. Source: PatchHolmes, AACL-IJCNLP 2026." title="Two findings. Finding 1: as tools are added to the agent, Recall@1 rises from 32.63% for the top commit without an agent, to 48.21% for an agent that can only see the list and submit, to 57.48% with commit summaries, and to 59.95% when it can also read code changes. Finding 2: with the model inside the agent swapped, Qwen3-235B with 22B active parameters reaches 59.95%, Qwen3-Coder-30B with 3B active parameters 59.09%, gpt-oss-120B 56.98%, and gpt-oss-20B 49.81%, against 32.63% with no agent. Source: PatchHolmes, AACL-IJCNLP 2026." srcset="https://substackcdn.com/image/fetch/$s_!4Ctl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 424w, https://substackcdn.com/image/fetch/$s_!4Ctl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 848w, https://substackcdn.com/image/fetch/$s_!4Ctl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 1272w, https://substackcdn.com/image/fetch/$s_!4Ctl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ec3012a-f613-48d1-9a3d-05e11df7f074_1456x2261.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>4.3 A Small Model Is Almost as Good</h3><p>Both Qwen3 models activate only part of their weights for each word. That part is called the active parameters. The results with other models inside the agent:</p><ul><li><p>Qwen3-Coder-30B, with 3B active parameters, reached 59.09%, against 59.95% for Qwen3-235B with 22B. It also cut the average time per CVE from 86 to 39 seconds.</p></li><li><p>gpt-oss-120B reached 56.98% and gpt-oss-20B 49.81%. Both often used up their 15 steps without submitting an answer.</p></li><li><p>Every model stayed far above the 32.63% without an agent.</p></li></ul><p>The authors read this as evidence that the agent keeps its gain when the model is swapped. They do not claim that model size never matters.</p><h3>4.4 The Gain Holds Outside the Main Benchmark</h3><ul><li><p>PatchFinder&#8217;s benchmark has 1,252 CVEs, each with 10 candidates chosen by PatchFinder&#8217;s own ranker. PatchHolmes&#8217;s agent, unchanged, raised Recall@1 from 24.28%, PatchFinder&#8217;s top pick, to 39.86%. Only 577 of the 1,252 CVEs have the fix among their 10 candidates, so no method can exceed 46.09%.</p></li><li><p>The authors also ran PatchHolmes on 35 CVEs from 11 projects, disclosed between 2013 and 2025. It ranked the fix first for 21 of them, 60.0%, against 8, or 22.9%, for Phase 1 alone.</p></li></ul><p>The 35 CVEs are the same ones whose fix links the SPFinder team merged in May 2025. Their correct answers came from SPFinder&#8217;s results plus manual checks, so this set cannot compare the two systems fairly. SPFinder had ranked 26 of them first.</p><h3>4.5 Each Vulnerability Costs Less Than One Cent</h3><p>Per CVE, PatchHolmes with Qwen3-235B used on average:</p><ul><li><p>96,118 input tokens and 846 output tokens</p></li><li><p>8 tool calls, opening 5 commits</p></li><li><p>86 seconds, with a median of 49 seconds</p></li></ul><p>At OpenRouter&#8217;s price on 1 May 2026, that comes to about USD 0.007 per CVE, or about USD 58 for all 8,401 CVEs in GitHubAD. From the token counts in Favia&#8217;s paper, the authors estimate that Favia would cost about USD 400 for the same 8,401. Only 0.49% of CVEs exceeded the model&#8217;s 32,000-token context window.</p><h2>5. Where the Results Stop</h2><p>PatchHolmes has these limits:</p><ul><li><p>Every benchmark holds only CVEs whose fix is already known. The hardest cases, where no one has found the fix, remain untested.</p></li><li><p>For 12.36% of CVEs, the fix is not among the top 100 candidates, so the agent cannot find it.</p></li><li><p>For 7.28% of CVEs in GitHubAD, more than one commit fixes the flaw. A valid fix that the records do not list counts as a miss.</p></li><li><p>The authors recommend using PatchHolmes to suggest candidates for human review. They advise against filling vulnerability databases automatically.</p></li></ul><p>SPFinder has these limits:</p><ul><li><p>SPFinder is an arXiv preprint. As of 16 October 2026, it has no conference or journal venue.</p></li><li><p>Most of its failures come from CVE text that names nothing found in the code. In 40 failed cases, 37.5% of CVE texts named no identifying file or component, and 42.5% named ones that the fix commit does not contain.</p></li><li><p>One of its features calls external services for each query. Two of them, GPT-4o and VoyageAI embeddings, are paid, and the third is GitHub code search.</p></li><li><p>Its code is in an anonymized repository with no license.</p></li></ul><h2>6. What It Means for Your Team</h2><p>Both systems point to practices that apply beyond these papers:</p><ol><li><p>When your team fixes a vulnerability in its own code, link the fix commit from the advisory, and name the CVE ID in the commit message once it exists. Commits that never mention the flaw, such as Rancher&#8217;s &#8220;Add project namespace handling&#8221;, are what makes the search hard.</p></li><li><p>Split the work into two steps: a cheap ranker builds a shortlist, and an LLM compares the shortlist as one list.</p></li><li><p>Test a small open model before paying for a large one. Here, 3B active parameters scored 59.09% against 59.95% for 22B.</p></li><li><p>Measure how often the shortlist contains the answer. PatchHolmes&#8217;s agent cannot find the 12.36% of fixes that Phase 1 misses, so a better shortlist raises the best possible score.</p></li><li><p>Keep a person approving each link before it reaches a database. All 35 merged SPFinder links were checked by hand.</p></li></ol><h2>7. Questions Readers Ask</h2><p><strong>How do I find the commit that fixed a CVE in a GitHub repository?</strong></p><p>Start with the GitHub Advisory Database, which lists fix commits for many CVEs. When it has none, a tool like PatchHolmes can rank candidate commits from the repository&#8217;s history. On 809 CVEs, PatchHolmes ranked the correct fix first for 59.95% of them, so a person should confirm its answer.</p><p><strong>Why do so many CVEs have no fix link?</strong></p><p>A person has to trace each fix by hand, and the fix commit often does not mention the flaw. A 2024 study found that about 63% of GitHub advisories had no link to the fixing commit.</p><p><strong>How accurate is an AI agent at finding vulnerability fix commits?</strong></p><p>PatchHolmes ranks the correct fix first for 59.95% of 809 CVEs. Favia, an earlier agent, does so for 34.61%. Both used the same model, Qwen3-235B.</p><p><strong>Do I need the largest LLM for this?</strong></p><p>In this study, the size of the model mattered little. Qwen3-Coder-30B, with 3B active parameters, reached 59.09% against 59.95% for Qwen3-235B, and ran in less than half the time.</p><p><strong>How much does it cost to run over thousands of CVEs?</strong></p><p>About USD 0.007 per CVE, or about USD 58 for 8,401 CVEs, at OpenRouter&#8217;s price on 1 May 2026.</p><p><strong>Is the code available?</strong></p><p>Yes. PatchHolmes is on <a href="https://github.com/Aizhouym/PatchHolmes">GitHub</a> under the MIT license. SPFinder&#8217;s code is in an anonymized <a href="https://github.com/AnonySE26/SPFinder">GitHub repository</a>.</p><h2>8. Read and Cite the Papers</h2><ul><li><p>Guanqun Yang, Yingming Zhou, Jiangrui Zheng, Shudong Hao, and Xueqing Liu. &#8220;PatchHolmes: Agentic Patch Retrieval via Listwise Selection.&#8221; Accepted at AACL-IJCNLP 2026. The first three authors contributed equally. <a href="https://arxiv.org/abs/2609.38807">arXiv</a>, <a href="https://github.com/Aizhouym/PatchHolmes">code</a>.</p></li><li><p>Jiangrui Zheng, Xueqing Liu, Guanqun Yang, Siyan Wen, Qiushi Liu, and Xiaoyin Wang. &#8220;SPFinder: Improving the Context Length and Scalability for Tracing Known Vulnerability Patches.&#8221; arXiv preprint, 2026. <a href="https://arxiv.org/abs/2503.22935">arXiv</a>, <a href="https://github.com/AnonySE26/SPFinder">code</a>.</p></li><li><p>PatchHolmes is the third of four steps in my dissertation. <a href="https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security">The overview post</a> explains how the steps fit together. <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-fewvuln-security-reports-fewer-labels">The FewVuln post</a> covers reading vulnerability reports, and <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-testaug-hatemoderate-ai-written-tests">the TestAug post</a> covers writing tests.</p></li></ul><p>To cite the papers:</p><pre><code>@misc{yang2026patchholmesagenticpatchretrieval,
  title         = {PatchHolmes: Agentic Patch Retrieval via Listwise Selection},
  author        = {Guanqun Yang and Yingming Zhou and Jiangrui Zheng and Shudong Hao and Xueqing Liu},
  year          = {2026},
  eprint        = {2609.38807},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  note          = {Accepted at AACL-IJCNLP 2026},
  url           = {https://arxiv.org/abs/2609.38807}
}

@misc{zheng2026spfinderimprovingcontextlength,
  title         = {SPFinder: Improving the Context Length and Scalability for Tracing Known Vulnerability Patches},
  author        = {Jiangrui Zheng and Xueqing Liu and Guanqun Yang and Siyan Wen and Qiushi Liu and Xiaoyin Wang},
  year          = {2026},
  eprint        = {2503.22935},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url           = {https://arxiv.org/abs/2503.22935}
}
</code></pre><div><hr></div><p><em>Guanqun Yang, LLM Researcher. More work at <a href="https://guanqun-yang.com">guanqun-yang.com</a> and on <a href="https://scholar.google.com/citations?user=4XUIHW0AAAAJ">Google Scholar</a>.</em></p>]]></content:encoded></item><item><title><![CDATA[Accuracy Scores Hide Failures. AI-Written Tests Find Them.]]></title><description><![CDATA[TestAug uses GPT-3 to write targeted tests for text classifiers. HateModerate tests hate speech detectors against Facebook's 41 hate speech policies.]]></description><link>https://newsletter.guanqun-yang.com/p/2026-10-16-testaug-hatemoderate-ai-written-tests</link><guid isPermaLink="false">https://newsletter.guanqun-yang.com/p/2026-10-16-testaug-hatemoderate-ai-written-tests</guid><dc:creator><![CDATA[Guanqun (David) Yang]]></dc:creator><pubDate>Fri, 16 Oct 2026 11:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3ee080fa-bb0b-4e9f-b1a0-09cbdfc43bdb_1456x763.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A text classifier can judge about 90% of its standard test correctly and still fail half of a set of simple targeted tests. TestAug, a framework I led and published at COLING 2022, the International Conference on Computational Linguistics, uses GPT-3 to write those targeted tests. For sentiment analysis, people wrote 29 test templates and the finished suite held 1,470, so about 98% of the templates needed no manual writing.</p><p>HateModerate, a follow-up study led by Jiangrui Zheng that I co-authored, applies the same idea to content moderation. It tests hate speech detectors against the 41 hate speech guidelines that Facebook had published by November 2022. HateModerate appeared in Findings of NAACL 2024, a peer-reviewed volume of the North American conference of the Association for Computational Linguistics (ACL). It received an Outstanding Paper Award at WOAH 2024, the Workshop on Online Abuse and Harms. Together, the two studies form the second step of my dissertation, where a language model moves from reading software documents to writing tests.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!NXgi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!NXgi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!NXgi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!NXgi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!NXgi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;DistilBERT, fine-tuned to detect duplicate questions on the Quora Question Pairs dataset, judges 89.7% of question pairs in its standard test correctly. The same model fails 49.8% of targeted tests, such as tests of negation. Source: TestAug, COLING 2022.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="DistilBERT, fine-tuned to detect duplicate questions on the Quora Question Pairs dataset, judges 89.7% of question pairs in its standard test correctly. The same model fails 49.8% of targeted tests, such as tests of negation. Source: TestAug, COLING 2022." title="DistilBERT, fine-tuned to detect duplicate questions on the Quora Question Pairs dataset, judges 89.7% of question pairs in its standard test correctly. The same model fails 49.8% of targeted tests, such as tests of negation. Source: TestAug, COLING 2022." srcset="https://substackcdn.com/image/fetch/$s_!NXgi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 424w, https://substackcdn.com/image/fetch/$s_!NXgi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 848w, https://substackcdn.com/image/fetch/$s_!NXgi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 1272w, https://substackcdn.com/image/fetch/$s_!NXgi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6eb4c3bf-c3cd-4d06-b0d1-f564039a04ec_1456x792.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><p><strong>In this post</strong></p><ol><li><p>Why One Accuracy Score Is Not Enough</p></li><li><p>How TestAug Writes Tests</p></li><li><p>What TestAug Found</p><ul><li><p>3.1 About 98% of the Templates Needed No Manual Writing</p></li><li><p>3.2 The Generated Tests Exposed More Failures</p></li><li><p>3.3 The Generated Tests Were More Varied</p></li></ul></li><li><p>HateModerate: Testing Detectors Against a Written Rulebook</p><ul><li><p>4.1 How the Test Set Was Built</p></li><li><p>4.2 Detectors Missed More Than Half the Hate Under Tiers 3 and 4</p></li><li><p>4.3 Training on Rule-Matched Examples Cut Failures</p></li><li><p>4.4 The Work Received an Outstanding Paper Award</p></li></ul></li><li><p>Where the Results Stop</p></li><li><p>What It Means for Your Team</p></li><li><p>Questions Readers Ask</p></li><li><p>Read and Cite the Papers</p></li></ol><h2>1. Why One Accuracy Score Is Not Enough</h2><p>Teams usually judge a text model by one number, such as its accuracy on a held-out test set. The number says how often the model is right on average. It does not say which kinds of input the model gets wrong. <a href="https://aclanthology.org/2020.acl-main.442/">CheckList</a>, the best overall paper at the ACL 2020 annual meeting, proposed adding targeted tests. Each targeted test is a small set of inputs that checks one skill, such as reading &#8220;not&#8221; correctly. For example, a sentiment model should label &#8220;No one loves the food.&#8221; as negative.</p><p>TestAug measured how far the two views differ. The study tested 10 public models on three tasks:</p><ul><li><p>Sentiment analysis: is a movie review positive or negative?</p></li><li><p>Duplicate-question detection: do two questions from Quora ask the same thing?</p></li><li><p>Natural language inference: does one sentence follow from another?</p></li></ul><p>The models were wrong on 5.7% to 12.6% of their standard test sets. On targeted tests, the same models failed 14.2% to 49.8% of cases, 2.2 to 4.8 times as often. For both numbers, lower is better.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!x3jm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 424w, https://substackcdn.com/image/fetch/$s_!x3jm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 848w, https://substackcdn.com/image/fetch/$s_!x3jm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 1272w, https://substackcdn.com/image/fetch/$s_!x3jm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!x3jm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Ten models, each shown with two dots. Sentiment: ALBERT 7.3% error on the standard test against 32.6% failure on targeted tests, BERT-base 7.6% against 33.9%, DistilBERT 10.0% against 29.5%, RoBERTa-base 5.7% against 14.2%. Duplicate questions: ALBERT 9.3% against 38.1%, BERT-base 9.1% against 36.0%, DistilBERT 10.3% against 49.8%. Natural language inference: ALBERT 9.9% against 42.8%, DistilBERT 12.6% against 34.7%, RoBERTa-large 8.1% against 17.8%. Source: TestAug, COLING 2022, Table 3.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Ten models, each shown with two dots. Sentiment: ALBERT 7.3% error on the standard test against 32.6% failure on targeted tests, BERT-base 7.6% against 33.9%, DistilBERT 10.0% against 29.5%, RoBERTa-base 5.7% against 14.2%. Duplicate questions: ALBERT 9.3% against 38.1%, BERT-base 9.1% against 36.0%, DistilBERT 10.3% against 49.8%. Natural language inference: ALBERT 9.9% against 42.8%, DistilBERT 12.6% against 34.7%, RoBERTa-large 8.1% against 17.8%. Source: TestAug, COLING 2022, Table 3." title="Ten models, each shown with two dots. Sentiment: ALBERT 7.3% error on the standard test against 32.6% failure on targeted tests, BERT-base 7.6% against 33.9%, DistilBERT 10.0% against 29.5%, RoBERTa-base 5.7% against 14.2%. Duplicate questions: ALBERT 9.3% against 38.1%, BERT-base 9.1% against 36.0%, DistilBERT 10.3% against 49.8%. Natural language inference: ALBERT 9.9% against 42.8%, DistilBERT 12.6% against 34.7%, RoBERTa-large 8.1% against 17.8%. Source: TestAug, COLING 2022, Table 3." srcset="https://substackcdn.com/image/fetch/$s_!x3jm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 424w, https://substackcdn.com/image/fetch/$s_!x3jm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 848w, https://substackcdn.com/image/fetch/$s_!x3jm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 1272w, https://substackcdn.com/image/fetch/$s_!x3jm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa04d2404-6252-4c9f-b719-6cb4dc21b03c_1456x1634.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><p>CheckList builds its targeted tests from templates. A template is a sentence with blanks, such as &#8220;That is not a [positive adjective] [airline noun].&#8221;, which a program fills from word lists to produce many test cases. The TestAug paper names two costs of writing templates by hand:</p><ul><li><p>A person must write each template and know both the language rules and the model&#8217;s likely weak points. The authors argue this often requires an expert.</p></li><li><p>The resulting tests vary little. In some CheckList tests, only the person names change from one case to the next.</p></li></ul><h2>2. How TestAug Writes Tests</h2><p>TestAug keeps people in charge of what to test and gives most of the writing to GPT-3. The workflow has five steps:</p><ol><li><p>A person describes the test in one sentence, such as &#8220;A negative sentiment sentence with negated positive word&#8221;. The person also picks seed examples from an existing CheckList suite, such as &#8220;No one appreciates that airline.&#8221;</p></li><li><p>GPT-3 receives the description and three seed examples and writes new cases in the same pattern, such as &#8220;No one appreciates that air traffic controller.&#8221;</p></li><li><p>Two annotators label 30 to 50 of GPT-3&#8217;s outputs as valid or invalid. If at least 90% are valid, or 80% for natural language inference, the test is used as is. Otherwise the annotators label at least 100 valid and 100 invalid outputs, and a small RoBERTa classifier learns to remove invalid ones automatically.</p></li><li><p>Template expansion turns the words GPT-3 reused from a seed back into blanks. &#8220;No one appreciates that air traffic controller.&#8221; becomes the new template &#8220;No one [positive verb] that air traffic controller.&#8221;, which produces many more cases.</p></li><li><p>The combined suite runs against the model under test and reports failures for each skill.</p></li></ol><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gFEK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 424w, https://substackcdn.com/image/fetch/$s_!gFEK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 848w, https://substackcdn.com/image/fetch/$s_!gFEK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 1272w, https://substackcdn.com/image/fetch/$s_!gFEK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!gFEK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The first four TestAug steps. Step 1: a person describes the test and picks a few examples, such as &#8220;No one appreciates that airline.&#8221; Step 2: GPT-3 writes new test cases in the same pattern, such as &#8220;No one appreciates that air traffic controller.&#8221; Step 3: a small classifier removes cases that break the test&#8217;s rules. It is needed only where GPT-3 often errs, and it is trained on at least 200 human labels. Step 4: template expansion turns repeated words into blanks, giving &#8220;No one [positive verb] that air traffic controller.&#8221; For the sentiment task, people wrote 29 templates and the finished suite has 1,470, so about 98% came without manual writing.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The first four TestAug steps. Step 1: a person describes the test and picks a few examples, such as &#8220;No one appreciates that airline.&#8221; Step 2: GPT-3 writes new test cases in the same pattern, such as &#8220;No one appreciates that air traffic controller.&#8221; Step 3: a small classifier removes cases that break the test&#8217;s rules. It is needed only where GPT-3 often errs, and it is trained on at least 200 human labels. Step 4: template expansion turns repeated words into blanks, giving &#8220;No one [positive verb] that air traffic controller.&#8221; For the sentiment task, people wrote 29 templates and the finished suite has 1,470, so about 98% came without manual writing." title="The first four TestAug steps. Step 1: a person describes the test and picks a few examples, such as &#8220;No one appreciates that airline.&#8221; Step 2: GPT-3 writes new test cases in the same pattern, such as &#8220;No one appreciates that air traffic controller.&#8221; Step 3: a small classifier removes cases that break the test&#8217;s rules. It is needed only where GPT-3 often errs, and it is trained on at least 200 human labels. Step 4: template expansion turns repeated words into blanks, giving &#8220;No one [positive verb] that air traffic controller.&#8221; For the sentiment task, people wrote 29 templates and the finished suite has 1,470, so about 98% came without manual writing." srcset="https://substackcdn.com/image/fetch/$s_!gFEK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 424w, https://substackcdn.com/image/fetch/$s_!gFEK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 848w, https://substackcdn.com/image/fetch/$s_!gFEK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 1272w, https://substackcdn.com/image/fetch/$s_!gFEK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1730b392-e4fe-4eb1-95b5-c407ea9fdec1_1456x2159.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h2>3. What TestAug Found</h2><h3>3.1 About 98% of the Templates Needed No Manual Writing</h3><p>Starting from a few hundred human-written sentences per task, TestAug produced thousands of new test sentences:</p><ul><li><p>Sentiment: 292 seed sentences and 3,275 new ones. People wrote 29 templates, and template expansion added 1,441.</p></li><li><p>Duplicate questions: 124 seed sentences and 6,427 new ones. People wrote 17 templates, and template expansion added 1,307.</p></li><li><p>Natural language inference: 150 seed sentences and 4,976 new ones. Template expansion was not used for this task.</p></li></ul><p>In the sentiment suite, people wrote 29 of the 1,470 templates. The paper reports this as a saving of at least 98.0% of manual effort. The figure counts templates, not hours. It leaves out the labels needed to train the filters, at least 200 for each filtered test.</p><h3>3.2 The Generated Tests Exposed More Failures</h3><p>The paper measures bug finding indirectly. It trains one copy of a model on TestAug&#8217;s tests and another on CheckList&#8217;s tests, then checks both copies on a shared set of targeted tests that neither copy saw. The authors call the result the patched failure rate, the share of shared tests that the trained copy still fails. Lower is better. The authors read a lower rate as evidence that the suite covered more of the model&#8217;s weak points. Two annotators labeled the shared tests and agreed on 91% to 97% of their labels, depending on the task.</p><ul><li><p>TestAug&#8217;s rate was lower than CheckList&#8217;s in all 10 combinations of model and task.</p></li><li><p>The largest difference was for DistilBERT on duplicate questions, 1.1% against 12.5%.</p></li><li><p>The smallest difference was for RoBERTa-large on natural language inference, 8.0% against 8.3%.</p></li><li><p>GPT-3&#8217;s new sentences produced most of the gain. Template expansion contributed less.</p></li></ul><p>The paper reports no significance test, and the spread across random data splits overlaps in several settings. TestAug also did not win on every individual skill.</p><h3>3.3 The Generated Tests Were More Varied</h3><p>The authors compared how varied the two suites are, using a sample of sentences from each:</p><ul><li><p>Self-BLEU4 measures word overlap among sentences in a suite, on a scale of 0 to 1, where lower means more varied. TestAug scored 0.634 against CheckList&#8217;s 0.853 for sentiment and 0.627 against 0.775 for duplicate questions. For natural language inference, the scores were 0.430 against 0.454.</p></li><li><p>Unique dependency paths count distinct grammatical patterns, such as noun followed by verb followed by noun, where higher means more varied. TestAug found 480 against 84, 418 against 117, and 210 against 116.</p></li></ul><h2>4. HateModerate: Testing Detectors Against a Written Rulebook</h2><p>Social platforms publish rules about what they remove, and automated detectors make most of the first decisions. The HateModerate paper cites a 2023 Facebook report that its automated systems found 95% of unwanted content before a user saw it. HateModerate asks whether a detector follows the platform&#8217;s written rules. Existing hate speech test sets are not organized by rule, and rare rules have few examples. For Facebook&#8217;s rule against offering therapies that claim to change sexual orientation, annotators found only 3 matching examples across existing datasets.</p><p>TestAug had skipped hate speech, because GPT-3 refused to write profanity in 2022. HateModerate used a newer GPT-3 model for part of its data and had people check every example. The rulebook is Facebook&#8217;s hate speech Community Standards as of November 2022: 41 guidelines in four tiers. Tier 1 is the most severe, such as dehumanizing speech. Tier 4, which Facebook called additional enforcement, includes the rule on conversion therapy offers.</p><h3>4.1 How the Test Set Was Built</h3><p>The hateful examples were collected as follows:</p><ol><li><p>Annotators searched existing hate speech datasets for examples that match each guideline.</p></li><li><p>Where matches were few, annotators wrote examples or prompted GPT-3 to write them.</p></li><li><p>Other annotators checked every example. A single &#8220;no match&#8221; vote removed it.</p></li></ol><p>The non-hateful examples needed to be hard for a detector. They were collected as follows:</p><ol><li><p>An embedding search found non-hateful sentences close in meaning to each guideline&#8217;s hateful examples, such as &#8220;I hate cauliflower.&#8221;</p></li><li><p>GPT-3 wrote more non-hateful examples of four kinds: supportive text, counter-hate (text that quotes hate to criticize it), neutral text, and insults toward targets the rules do not protect. People wrote the last kind, because GPT-3 struggled with it.</p></li><li><p>Annotators checked each example for relevance. GPT-4 and one annotator then sorted each example by type and removed any that were actually hateful.</p></li></ol><p>The finished dataset contains:</p><ul><li><p>7,704 examples, of which 4,796 are hateful and 2,908 are non-hateful.</p></li><li><p>103 to 367 examples per guideline.</p></li><li><p>1,385 examples, 18% of the total, written by GPT-3 and checked by people.</p></li><li><p>Labels from 28 graduate students in computer science, with about seven weeks of annotation time spread over seven months.</p></li></ul><h3>4.2 Detectors Missed More Than Half the Hate Under Tiers 3 and 4</h3><p>The study tested four detectors:</p><ul><li><p>Google Jigsaw&#8217;s Perspective API</p></li><li><p>OpenAI&#8217;s Moderation API</p></li><li><p>Facebook&#8217;s RoBERTa hate speech model, version R1</p></li><li><p>Cardiff NLP&#8217;s RoBERTa hate speech model</p></li></ul><p>The metric is the failure rate for each guideline, on a scale of 0 to 1, where lower is better. The study measured it separately for hateful examples, where a failure is missed hate, and for non-hateful examples, where a failure is a false alarm.</p><ul><li><p>Every detector failed often. The best, OpenAI&#8217;s Moderation API, had an average failure rate of 0.29, about three examples in ten.</p></li><li><p>Averaged over the four detectors, missed hate was 0.34 for Tiers 1 and 2, 0.59 for Tier 3, and 0.52 for Tier 4.</p></li><li><p>False alarms went the other way: 0.47 for Tier 1, 0.52 for Tier 2, 0.38 for Tier 3, and 0.36 for Tier 4. The detectors flagged harmless text more often near severe topics.</p></li><li><p>Counter-hate and insults toward unprotected targets caused the most false alarms. Supportive text caused the fewest.</p></li><li><p>For Perspective, raising the flagging threshold from 0.5 to 0.7 cut false alarms under Tier 2 from 0.74 to 0.55, but raised missed hate from 0.34 to 0.60.</p></li></ul><p>The hardest rules for most detectors included calls for economic exclusion, conversion therapy offers, comparisons of people to certain objects, and statements that admit intolerance. These four rules had only 3 to 17 matching examples in existing datasets before HateModerate added new ones.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6xF8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 424w, https://substackcdn.com/image/fetch/$s_!6xF8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 848w, https://substackcdn.com/image/fetch/$s_!6xF8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 1272w, https://substackcdn.com/image/fetch/$s_!6xF8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!6xF8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two findings. Finding 1: averaged over four detectors, the share of hateful examples missed was 0.34 for Tier 1, the most severe rules such as dehumanizing speech (14 rules), 0.34 for Tier 2, such as contempt (19 rules), 0.59 for Tier 3, such as calls for economic exclusion (5 rules), and 0.52 for Tier 4, such as conversion therapy offers (3 rules). Finding 2: a RoBERTa detector&#8217;s failure rate on the HateModerate test fell from 0.423 to 0.275 after adding HateModerate&#8217;s training data, and on HateCheck, an independent test, from 0.365 to 0.199. Lower is better. Source: HateModerate, Findings of NAACL 2024, Tables 3 and 4.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two findings. Finding 1: averaged over four detectors, the share of hateful examples missed was 0.34 for Tier 1, the most severe rules such as dehumanizing speech (14 rules), 0.34 for Tier 2, such as contempt (19 rules), 0.59 for Tier 3, such as calls for economic exclusion (5 rules), and 0.52 for Tier 4, such as conversion therapy offers (3 rules). Finding 2: a RoBERTa detector&#8217;s failure rate on the HateModerate test fell from 0.423 to 0.275 after adding HateModerate&#8217;s training data, and on HateCheck, an independent test, from 0.365 to 0.199. Lower is better. Source: HateModerate, Findings of NAACL 2024, Tables 3 and 4." title="Two findings. Finding 1: averaged over four detectors, the share of hateful examples missed was 0.34 for Tier 1, the most severe rules such as dehumanizing speech (14 rules), 0.34 for Tier 2, such as contempt (19 rules), 0.59 for Tier 3, such as calls for economic exclusion (5 rules), and 0.52 for Tier 4, such as conversion therapy offers (3 rules). Finding 2: a RoBERTa detector&#8217;s failure rate on the HateModerate test fell from 0.423 to 0.275 after adding HateModerate&#8217;s training data, and on HateCheck, an independent test, from 0.365 to 0.199. Lower is better. Source: HateModerate, Findings of NAACL 2024, Tables 3 and 4." srcset="https://substackcdn.com/image/fetch/$s_!6xF8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 424w, https://substackcdn.com/image/fetch/$s_!6xF8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 848w, https://substackcdn.com/image/fetch/$s_!6xF8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 1272w, https://substackcdn.com/image/fetch/$s_!6xF8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdd2ce0f-7809-4f4e-91c5-7bde4340f26d_1456x2176.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h3>4.3 Training on Rule-Matched Examples Cut Failures</h3><p>The authors trained a RoBERTa-base detector on Cardiff NLP&#8217;s public training data, once without and once with HateModerate&#8217;s training half. The failure rates changed as follows, where lower is better:</p><ul><li><p>On HateModerate&#8217;s test half, the failure rate fell from 0.423 to 0.275, a 35% reduction.</p></li><li><p>On <a href="https://aclanthology.org/2021.acl-long.4/">HateCheck</a>, an independent set of 29 functional tests for hate speech detectors, it fell from 0.365 to 0.199, a 45% reduction.</p></li><li><p>On three original benchmarks, the failure rate stayed about the same. It was unchanged on hatEval (0.485) and HTPO (0.090) and fell from 0.221 to 0.201 on HateXplain.</p></li><li><p>The new, rule-specific examples produced most of the improvement on the hardest rules. Compared with training on reused examples alone, they cut missed hate for economic exclusion from 0.466 to 0.023 and for conversion therapy offers from 0.735 to 0.245.</p></li></ul><p>The study also checked for new bias. A statistical test found no significant increase in false alarms on sentences about protected groups. Two limits remain. Some slices got worse, such as false alarms on HateCheck&#8217;s non-hateful examples, which rose from 0.205 to 0.229. A balanced version of the training data removed that increase. OpenAI&#8217;s Moderation API also remained much better on HateCheck, at 0.011 against 0.199.</p><h3>4.4 The Work Received an Outstanding Paper Award</h3><p>HateModerate received an Outstanding Paper Award at WOAH 2024, the 8th Workshop on Online Abuse and Harms, held with NAACL 2024 in Mexico City. The organizers gave two such awards that year. I could not find a public list of the winners, so the award notice appears below, with email addresses removed.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VIzv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 424w, https://substackcdn.com/image/fetch/$s_!VIzv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 848w, https://substackcdn.com/image/fetch/$s_!VIzv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 1272w, https://substackcdn.com/image/fetch/$s_!VIzv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!VIzv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The award notice from Paul R&#246;ttger on behalf of the WOAH organizing committee to Jiangrui Zheng, dated 3 May 2024. It states that &#8220;HateModerate: Testing Hate Speech Detectors against Content Moderation Policies&#8221; will receive an outstanding paper award at WOAH 2024, one of two such awards. Email addresses are removed.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The award notice from Paul R&#246;ttger on behalf of the WOAH organizing committee to Jiangrui Zheng, dated 3 May 2024. It states that &#8220;HateModerate: Testing Hate Speech Detectors against Content Moderation Policies&#8221; will receive an outstanding paper award at WOAH 2024, one of two such awards. Email addresses are removed." title="The award notice from Paul R&#246;ttger on behalf of the WOAH organizing committee to Jiangrui Zheng, dated 3 May 2024. It states that &#8220;HateModerate: Testing Hate Speech Detectors against Content Moderation Policies&#8221; will receive an outstanding paper award at WOAH 2024, one of two such awards. Email addresses are removed." srcset="https://substackcdn.com/image/fetch/$s_!VIzv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 424w, https://substackcdn.com/image/fetch/$s_!VIzv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 848w, https://substackcdn.com/image/fetch/$s_!VIzv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 1272w, https://substackcdn.com/image/fetch/$s_!VIzv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68a133a9-2fda-468a-b1ee-20245d26214d_1219x262.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h2>5. Where the Results Stop</h2><p>TestAug has these limits:</p><ul><li><p>GPT-3 could not write highly structured tests, such as question-answering cases, or tests that need logic or arithmetic.</p></li><li><p>The 98% figure counts templates, not hours of work.</p></li><li><p>The 10-of-10 result has no significance test, and the gains were uneven across skills.</p></li><li><p>The models tested were BERT-family classifiers from 2018 and 2019. Commercial APIs were not tested, because the method needs to fine-tune the model.</p></li></ul><p>HateModerate has these limits:</p><ul><li><p>The dataset covers one platform&#8217;s rules at one point in time, Facebook&#8217;s as of November 2022. As of 16 October 2026, Meta publishes the current version as <a href="https://transparency.meta.com/policies/community-standards/hateful-conduct/">Hateful Conduct</a>.</p></li><li><p>Rules from another platform would need new human annotation.</p></li><li><p>Implicit hate, which uses no explicit hateful words, is under-represented. Examples may also over-represent keywords named in the guidelines.</p></li><li><p>Commercial detectors change over time. <a href="https://perspectiveapi.com/">Perspective API</a>&#8217;s website says the service ends after 2026.</p></li></ul><h2>6. What It Means for Your Team</h2><p>Both studies used GPT-3 and models from before 2023. Their lessons concern how a team organizes testing. The lessons apply to any text model a team ships:</p><ol><li><p>Report failure rates for each skill or each policy, next to the overall accuracy.</p></li><li><p>Turn each written rule into its own test set, so that each failure points to a rule.</p></li><li><p>Let a language model draft test cases. People decide what to test and check samples of the output.</p></li><li><p>Measure how often the generator writes invalid tests before trusting them, and add a filter where it often errs.</p></li><li><p>After fixing failures with new training data, recheck the original benchmarks and the false-alarm rates.</p></li></ol><h2>7. Questions Readers Ask</h2><p><strong>Why is high accuracy on a test set not enough to trust a text classifier?</strong></p><p>One accuracy number does not show which skills fail. In TestAug&#8217;s experiments, models that were wrong on 5.7% to 12.6% of their standard test sets failed 14.2% to 49.8% of targeted tests, such as tests of negation.</p><p><strong>Can a large language model write test cases for an NLP model?</strong></p><p>It can, if a check removes invalid output. In TestAug, GPT-3 wrote new test cases from a one-sentence test description and three examples, and a small trained filter removed invalid cases where needed. About 98% of the final sentiment templates required no manual writing.</p><p><strong>Do hate speech detectors follow Facebook&#8217;s community standards?</strong></p><p>They follow them only in part. HateModerate tested four detectors against Facebook&#8217;s 41 hate speech guidelines as of November 2022. The best, OpenAI&#8217;s Moderation API, had an average failure rate of 0.29. On average, the detectors missed more than half of the hate under the Tier 3 and Tier 4 rules.</p><p><strong>Is there a hate speech dataset organized by platform policy?</strong></p><p>HateModerate is one such dataset. It has 7,704 examples matched to 41 guidelines, each verified by people. The dataset is on <a href="https://github.com/stevens-textmining/HateModerate">GitHub</a>.</p><p><strong>Are the code and data available?</strong></p><p>Both are public. TestAug&#8217;s code is on <a href="https://github.com/guanqun-yang/TestAug">GitHub</a>, and HateModerate&#8217;s code and data are on <a href="https://github.com/stevens-textmining/HateModerate">GitHub</a>.</p><h2>8. Read and Cite the Papers</h2><ul><li><p>Guanqun Yang, Mirazul Haque, Qiaochu Song, Wei Yang, and Xueqing Liu. &#8220;TestAug: A Framework for Augmenting Capability-based NLP Tests.&#8221; In <em>Proceedings of the 29th International Conference on Computational Linguistics</em> (COLING 2022), pages 3480 to 3495. <a href="https://aclanthology.org/2022.coling-1.307/">Paper</a>, <a href="https://arxiv.org/abs/2210.08097">arXiv</a>, <a href="https://github.com/guanqun-yang/TestAug">code</a>, <a href="https://www.youtube.com/watch?v=X0p8J57qxeg">talk video</a>.</p></li><li><p>Jiangrui Zheng, Xueqing Liu, Guanqun Yang, Mirazul Haque, Xing Qian, Ravishka Rathnasuriya, Wei Yang, and Girish Budhrani. &#8220;HateModerate: Testing Hate Speech Detectors against Content Moderation Policies.&#8221; In <em>Findings of the Association for Computational Linguistics: NAACL 2024</em>, pages 2691 to 2710. Outstanding Paper Award, WOAH 2024. <a href="https://aclanthology.org/2024.findings-naacl.172/">Paper</a>, <a href="https://arxiv.org/abs/2307.12418">arXiv</a>, <a href="https://github.com/stevens-textmining/HateModerate">code and data</a>.</p></li><li><p>Both studies form the second of four steps in my dissertation. <a href="https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security">The overview post</a> explains how the steps fit together, and <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-fewvuln-security-reports-fewer-labels">the FewVuln post</a> covers the first step.</p></li></ul><p>To cite the papers:</p><pre><code>@inproceedings{yang-etal-2022-testaug,
  author    = {Guanqun Yang and Mirazul Haque and Qiaochu Song and Wei Yang and Xueqing Liu},
  title     = {TestAug: A Framework for Augmenting Capability-based NLP Tests},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022)},
  pages     = {3480--3495},
  publisher = {International Committee on Computational Linguistics},
  address   = {Gyeongju, Republic of Korea},
  year      = {2022},
  url       = {https://aclanthology.org/2022.coling-1.307/}
}

@inproceedings{zheng-etal-2024-hatemoderate,
  author    = {Jiangrui Zheng and Xueqing Liu and Guanqun Yang and Mirazul Haque and Xing Qian and Ravishka Rathnasuriya and Wei Yang and Girish Budhrani},
  title     = {HateModerate: Testing Hate Speech Detectors against Content Moderation Policies},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2024},
  pages     = {2691--2710},
  publisher = {Association for Computational Linguistics},
  address   = {Mexico City, Mexico},
  year      = {2024},
  doi       = {10.18653/v1/2024.findings-naacl.172}
}
</code></pre><div><hr></div><p><em>Guanqun Yang, LLM Researcher. More work at <a href="https://guanqun-yang.com">guanqun-yang.com</a> and on <a href="https://scholar.google.com/citations?user=4XUIHW0AAAAJ">Google Scholar</a>.</em></p>]]></content:encoded></item><item><title><![CDATA[Teaching AI to Read Vulnerability Reports With 90% Fewer Labels]]></title><description><![CDATA[A pre-trained language model, fine-tuned on 576 labeled sentences, nearly matches a purpose-built system trained on 5,758.]]></description><link>https://newsletter.guanqun-yang.com/p/2026-10-16-fewvuln-security-reports-fewer-labels</link><guid isPermaLink="false">https://newsletter.guanqun-yang.com/p/2026-10-16-fewvuln-security-reports-fewer-labels</guid><dc:creator><![CDATA[Guanqun (David) Yang]]></dc:creator><pubDate>Fri, 16 Oct 2026 04:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/de35c851-fc3b-4130-a34f-8bdbafe897ea_1456x763.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A language model that has already read large amounts of general text needs few examples to read security reports. FewVuln fine-tunes such a model on 576 labeled sentences and nearly matches the best earlier system, VIEM, which was trained on 5,758. On 12 more categories of reports, 64 labeled sentences per category were enough to come within 0.003 of VIEM on F1, a 0-to-1 accuracy score where higher is better.</p><p>FewVuln is the first of the four studies in my dissertation. It was published at MLHat 2021, a workshop on machine learning for security held at KDD 2021, a major data mining conference.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QC4h!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 424w, https://substackcdn.com/image/fetch/$s_!QC4h!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 848w, https://substackcdn.com/image/fetch/$s_!QC4h!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 1272w, https://substackcdn.com/image/fetch/$s_!QC4h!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!QC4h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;FewVuln, a fine-tuned ELECTRA model, reaches an F1 of 0.97 for finding vulnerable software names in memory-corruption reports with 576 labeled sentences. The earlier best system, VIEM, reaches 0.98 with 5,758 labeled sentences.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="FewVuln, a fine-tuned ELECTRA model, reaches an F1 of 0.97 for finding vulnerable software names in memory-corruption reports with 576 labeled sentences. The earlier best system, VIEM, reaches 0.98 with 5,758 labeled sentences." title="FewVuln, a fine-tuned ELECTRA model, reaches an F1 of 0.97 for finding vulnerable software names in memory-corruption reports with 576 labeled sentences. The earlier best system, VIEM, reaches 0.98 with 5,758 labeled sentences." srcset="https://substackcdn.com/image/fetch/$s_!QC4h!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 424w, https://substackcdn.com/image/fetch/$s_!QC4h!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 848w, https://substackcdn.com/image/fetch/$s_!QC4h!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 1272w, https://substackcdn.com/image/fetch/$s_!QC4h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c04716-e882-46ab-8e31-a5be1e55b784_1456x811.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><p><strong>In this post</strong></p><ol><li><p>Why Patch Decisions Depend on Free Text</p></li><li><p>How FewVuln Works</p></li><li><p>What the Results Show</p><ul><li><p>3.1 One-Tenth of the Labels, Nearly the Same Accuracy</p></li><li><p>3.2 Simple Fine-Tuning Beat a Specialized Few-Shot Method</p></li><li><p>3.3 Software Names Are Harder Than Version Numbers</p></li></ul></li><li><p>Where the Result Stops</p></li><li><p>What It Means for Your Team</p></li><li><p>Questions Readers Ask</p></li><li><p>Read and Cite the Paper</p></li></ol><h2>1. Why Patch Decisions Depend on Free Text</h2><p>Every publicly disclosed vulnerability gets a CVE record. The US National Vulnerability Database (<a href="https://nvd.nist.gov/">NVD</a>) published 48,152 new records in 2025, about 132 per day, according to a count I ran against the NVD&#8217;s public API on 15 October 2026. Each record describes the flaw in free text. Before a team can decide whether to patch, someone must extract which software and which versions are affected.</p><p>The records do not always agree with each other either. A <a href="https://www.usenix.org/conference/usenixsecurity19/presentation/dong">2019 study by Dong and colleagues</a> found that only 59.82% of reports listed the same vulnerable versions as the NVD. Tools that extract these fields automatically exist, including VIEM from that same study. But they need people with security knowledge to label thousands of sentences word by word before they work.</p><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!W04d!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 424w, https://substackcdn.com/image/fetch/$s_!W04d!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 848w, https://substackcdn.com/image/fetch/$s_!W04d!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 1272w, https://substackcdn.com/image/fetch/$s_!W04d!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!W04d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A public report, CVE-2015-2384, reads: Microsoft Internet Explorer 11 allows remote attackers to execute arbitrary code or cause a denial of service (memory corruption) via a crafted web site, aka Internet Explorer Memory Corruption Vulnerability. FewVuln tags the first Internet Explorer as the software name and 11 as the version. The second Internet Explorer is part of the flaw&#8217;s name and is not tagged. The result is a structured record that a team uses to decide whether to patch and to check whether the record agrees with other reports.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A public report, CVE-2015-2384, reads: Microsoft Internet Explorer 11 allows remote attackers to execute arbitrary code or cause a denial of service (memory corruption) via a crafted web site, aka Internet Explorer Memory Corruption Vulnerability. FewVuln tags the first Internet Explorer as the software name and 11 as the version. The second Internet Explorer is part of the flaw&#8217;s name and is not tagged. The result is a structured record that a team uses to decide whether to patch and to check whether the record agrees with other reports." title="A public report, CVE-2015-2384, reads: Microsoft Internet Explorer 11 allows remote attackers to execute arbitrary code or cause a denial of service (memory corruption) via a crafted web site, aka Internet Explorer Memory Corruption Vulnerability. FewVuln tags the first Internet Explorer as the software name and 11 as the version. The second Internet Explorer is part of the flaw&#8217;s name and is not tagged. The result is a structured record that a team uses to decide whether to patch and to check whether the record agrees with other reports." srcset="https://substackcdn.com/image/fetch/$s_!W04d!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 424w, https://substackcdn.com/image/fetch/$s_!W04d!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 848w, https://substackcdn.com/image/fetch/$s_!W04d!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 1272w, https://substackcdn.com/image/fetch/$s_!W04d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2766c00-e8dc-43a5-96c7-41ec753beb1b_1456x1719.png 1456w" sizes="100vw"></picture><div></div></div></div><h2>2. How FewVuln Works</h2><p>The task is named entity recognition (NER): the model labels each word as a vulnerable software name, a vulnerable version, or neither. FewVuln starts from a pre-trained language model. The study compared three widely used ones, BERT, RoBERTa, and ELECTRA. Training and testing take three steps:</p><ol><li><p>Fine-tune the model on the largest category of reports, memory corruption, with 576 labeled sentences, 10% of the 5,758 sentences VIEM used.</p></li><li><p>Continue training on the 12 other categories with 64 labeled sentences from each, 768 in total. VIEM used 7,016 for the same categories.</p></li><li><p>Test each category separately and compare with VIEM&#8217;s published results.</p></li></ol><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!F8gL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 424w, https://substackcdn.com/image/fetch/$s_!F8gL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 848w, https://substackcdn.com/image/fetch/$s_!F8gL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 1272w, https://substackcdn.com/image/fetch/$s_!F8gL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!F8gL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two training steps. Step 1, fine-tuning on memory corruption: VIEM uses 5,758 labeled sentences and FewVuln uses 576. F1 for software names is 0.97 for FewVuln against 0.98, and for versions 0.99 against 0.99. Step 2, transfer to 12 more categories: VIEM uses 7,016 labeled sentences and FewVuln uses 768. F1 for software names is 0.92 against 0.92, and for versions 0.94 against 0.94.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two training steps. Step 1, fine-tuning on memory corruption: VIEM uses 5,758 labeled sentences and FewVuln uses 576. F1 for software names is 0.97 for FewVuln against 0.98, and for versions 0.99 against 0.99. Step 2, transfer to 12 more categories: VIEM uses 7,016 labeled sentences and FewVuln uses 768. F1 for software names is 0.92 against 0.92, and for versions 0.94 against 0.94." title="Two training steps. Step 1, fine-tuning on memory corruption: VIEM uses 5,758 labeled sentences and FewVuln uses 576. F1 for software names is 0.97 for FewVuln against 0.98, and for versions 0.99 against 0.99. Step 2, transfer to 12 more categories: VIEM uses 7,016 labeled sentences and FewVuln uses 768. F1 for software names is 0.92 against 0.92, and for versions 0.94 against 0.94." srcset="https://substackcdn.com/image/fetch/$s_!F8gL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 424w, https://substackcdn.com/image/fetch/$s_!F8gL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 848w, https://substackcdn.com/image/fetch/$s_!F8gL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 1272w, https://substackcdn.com/image/fetch/$s_!F8gL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfc08e26-e595-4ad3-a3ae-bfdfb37afc4d_1456x2015.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><h2>3. What the Results Show</h2><h3>3.1 One-Tenth of the Labels, Nearly the Same Accuracy</h3><p>ELECTRA scored highest of the three models in both steps. Its results, against VIEM&#8217;s:</p><ul><li><p>On memory corruption, with 576 labeled sentences, ELECTRA reached an F1 of 0.9715 for software names and 0.9860 for versions. VIEM, with 5,758, reached 0.9844 and 0.9901.</p></li><li><p>On the 12 other categories, with 768 labeled sentences, ELECTRA reached 0.9214 for names and 0.9369 for versions. VIEM, with 7,016, reached 0.9235 and 0.9396.</p></li><li><p>The label savings were 90% in the first step and about 89% in the second.</p></li></ul><p>ELECTRA was also the only base-size model among the three. The two others were large versions, which suggests that model size was not what decided the result.</p><h3>3.2 Simple Fine-Tuning Beat a Specialized Few-Shot Method</h3><p>The study also tested StructShot, a published few-shot method that labels each word by comparing it with a small set of labeled examples. Using StructShot in place of the second training step lowered ELECTRA&#8217;s F1 for software names from 0.9214 to 0.7274. Adding StructShot after the second step also lowered F1 for every model. In one category, the paper reports that adding a single example to StructShot&#8217;s reference set cut its F1 for software names by about 70%.</p><h3>3.3 Software Names Are Harder Than Version Numbers</h3><p>With only 58 labeled sentences, all three models found versions with an F1 around 0.95, but stayed below 0.92 for software names. Versions follow patterns such as &#8220;1.0&#8221; or &#8220;before 9.3&#8221;, and names do not. Many names are also new. In VIEM&#8217;s memory-corruption test data, 47 of 205 software names (22.9%) never appear in the training data, according to the 2019 study.</p><p>The errors lean toward false alarms. Recall, the share of true names and versions found, was usually higher than precision, the share of flagged words that were correct. The paper argues this is the safer direction, because a false alarm costs a manual check while a miss can leave a system unpatched.</p><h2>4. Where the Result Stops</h2><p>FewVuln is a first study, and four limits matter:</p><ul><li><p>It comes close to VIEM but does not beat it on F1. FewVuln scores higher only on recall.</p></li><li><p>VIEM can also use a dictionary of known software names. With that dictionary, VIEM reaches an F1 of about 0.95 for names on the 12 categories, above FewVuln&#8217;s 0.92. The comparison above uses VIEM without the dictionary, since FewVuln uses none.</p></li><li><p>All results come from one dataset, the one released with VIEM.</p></li><li><p>FewVuln still needs a few hundred labeled sentences, and its results depend on careful tuning of training settings.</p></li></ul><h2>5. What It Means for Your Team</h2><p>FewVuln dates from 2021, before the large language models released from 2023 onward. Its main lesson still applies when a team adapts any model to its own documents:</p><ul><li><p>Start from a pre-trained model. Here it cut labeling work by about 90%.</p></li><li><p>Label a small set first and measure. In FewVuln, results stopped improving after about 64 labeled sentences per category.</p></li><li><p>Try simple fine-tuning before a specialized few-shot method. The simple method won by up to 0.19 F1.</p></li><li><p>Keep a person checking new software names, the part the model finds hardest.</p></li></ul><h2>6. Questions Readers Ask</h2><p><strong>How many labeled examples does AI need to extract software names and versions from CVE reports?</strong></p><p>In this study, 576 sentences for the largest category and 64 per category for 12 others were enough. With them, FewVuln came within 0.02 F1 of a system trained on about ten times as many.</p><p><strong>Which model worked best?</strong></p><p>ELECTRA, the only base-size model tested, scored highest. BERT and RoBERTa, both in large versions, were close behind.</p><p><strong>Is the code available?</strong></p><p>Yes. The code is on <a href="https://github.com/guanqun-yang/FewVuln">GitHub</a>. FewVuln is the name used there and in the dissertation. The paper itself does not name the system.</p><h2>7. Read and Cite the Paper</h2><ul><li><p>Guanqun Yang, Shay Dineen, Zhipeng Lin, and Xueqing Liu. &#8220;Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-trained Language Models.&#8221; In <em>Deployable Machine Learning for Security Defense</em> (MLHat 2021), Communications in Computer and Information Science, volume 1482, Springer, 2021. <a href="https://arxiv.org/abs/2108.06590">Free version on arXiv</a>, <a href="https://doi.org/10.1007/978-3-030-87839-9_3">Springer</a>, <a href="https://github.com/guanqun-yang/FewVuln">code</a>.</p></li><li><p>FewVuln is the first of four studies in my dissertation. <a href="https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security">The overview post</a> explains how they fit together.</p></li></ul><p>To cite the paper:</p><pre><code>@inproceedings{yang2021fewsample,
  author    = {Guanqun Yang and Shay Dineen and Zhipeng Lin and Xueqing Liu},
  title     = {Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-trained Language Models},
  booktitle = {Deployable Machine Learning for Security Defense (MLHat 2021)},
  series    = {Communications in Computer and Information Science},
  volume    = {1482},
  pages     = {55--78},
  publisher = {Springer},
  year      = {2021},
  doi       = {10.1007/978-3-030-87839-9_3}
}
</code></pre><div><hr></div><p><em>Guanqun Yang, LLM Researcher. More work at <a href="https://guanqun-yang.com">guanqun-yang.com</a> and on <a href="https://scholar.google.com/citations?user=4XUIHW0AAAAJ">Google Scholar</a>.</em></p>]]></content:encoded></item><item><title><![CDATA[What Six Years of Research Taught Me About Using AI to Test and Secure Software]]></title><description><![CDATA[The design around a model matters more than its size. Four studies show it, from reading security reports to giving feedback to AI coding agents.]]></description><link>https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security</link><guid isPermaLink="false">https://newsletter.guanqun-yang.com/p/2026-10-15-language-models-software-testing-security</guid><dc:creator><![CDATA[Guanqun (David) Yang]]></dc:creator><pubDate>Thu, 15 Oct 2026 13:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/b7b1b9d7-f19d-4996-a25e-68fe0d9dc633_1456x763.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>From 2020 to 2026, my research asked how much of the manual work in software testing and security a language model can do. I found that a model can do much of it, if it gets more independence one step at a time. My dissertation, <em>Scalable Software Testing and Analysis with NLP Methods</em> (Stevens Institute of Technology, 2026), tests this in four studies:</p><ul><li><p>FewVuln reads security reports and extracts the affected software and versions.</p></li><li><p>TestAug writes tests that expose failures in AI text models.</p></li><li><p>PatchHolmes searches a code repository for the change that fixed a vulnerability.</p></li><li><p>WatchPoint gives feedback to an AI coding agent whose code failed a test.</p></li></ul><p>Across the four studies, the design around the model mattered more than the model&#8217;s size. In PatchHolmes, a model about 7 times smaller lost less than 1 percentage point in its main accuracy score.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-uk6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 424w, https://substackcdn.com/image/fetch/$s_!-uk6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 848w, https://substackcdn.com/image/fetch/$s_!-uk6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 1272w, https://substackcdn.com/image/fetch/$s_!-uk6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!-uk6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Four steps of growing model independence. FewVuln reads security reports and needs 90% fewer labeled examples. TestAug writes tests, and AI wrote 98% of its test templates. PatchHolmes acts with tools and ranks the true security fix first for 60% of vulnerabilities, against 35% for the best earlier method. WatchPoint collaborates with a coding agent, and 57.6% of failed tasks pass on retry, against 45.1% with test errors alone.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Four steps of growing model independence. FewVuln reads security reports and needs 90% fewer labeled examples. TestAug writes tests, and AI wrote 98% of its test templates. PatchHolmes acts with tools and ranks the true security fix first for 60% of vulnerabilities, against 35% for the best earlier method. WatchPoint collaborates with a coding agent, and 57.6% of failed tasks pass on retry, against 45.1% with test errors alone." title="Four steps of growing model independence. FewVuln reads security reports and needs 90% fewer labeled examples. TestAug writes tests, and AI wrote 98% of its test templates. PatchHolmes acts with tools and ranks the true security fix first for 60% of vulnerabilities, against 35% for the best earlier method. WatchPoint collaborates with a coding agent, and 57.6% of failed tasks pass on retry, against 45.1% with test errors alone." srcset="https://substackcdn.com/image/fetch/$s_!-uk6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 424w, https://substackcdn.com/image/fetch/$s_!-uk6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 848w, https://substackcdn.com/image/fetch/$s_!-uk6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 1272w, https://substackcdn.com/image/fetch/$s_!-uk6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73cfc56a-4ff5-493a-a384-59eec8fb6d25_1456x851.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p><strong>In this post</strong></p><ol><li><p>Why Checking Software Still Depends on People</p></li><li><p>Four Steps of Growing Independence</p><ul><li><p>2.1 Step 1, Reading: FewVuln</p></li><li><p>2.2 Step 2, Writing Tests: TestAug and HateModerate</p></li><li><p>2.3 Step 3, Acting With Tools: PatchHolmes</p></li><li><p>2.4 Step 4, Working With Another Agent: WatchPoint</p></li></ul></li><li><p>What It Means for Your Team</p><ul><li><p>3.1 The Design Around a Model Matters More Than Its Size</p></li><li><p>3.2 A Helper&#8217;s Value Depends on the Model That Uses It</p></li></ul></li><li><p>Research Beyond the Dissertation</p></li><li><p>Questions Readers Ask</p></li><li><p>Read and Cite the Work</p></li></ol><h2>1. Why Checking Software Still Depends on People</h2><p>Poor software quality cost the United States at least USD 2.41 trillion in 2022, according to a <a href="https://www.it-cisq.org/the-cost-of-poor-quality-software-in-the-us-a-2022-report/">report by the Consortium for Information &amp; Software Quality</a>. Much of the work that prevents such losses is still done by hand. Most of that work deals with text:</p><ul><li><p>Security analysts read vulnerability reports, written as paragraphs of English, and search code histories for the change that fixed each flaw.</p></li><li><p>Test engineers write test cases one at a time, each a sentence with an expected answer.</p></li><li><p>Developers who use AI coding assistants run and check each change that comes out of a conversation with the assistant.</p></li></ul><p>Language models are built to process text, so they suit this work. The studies below ask how much of the work a model can do without a person checking each step.</p><h2>2. Four Steps of Growing Independence</h2><p>My dissertation answers that question with four studies. Each study gives the model more independence than the one before, and each step removes one kind of manual work. The figure below shows where each study works in the life of a software product.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hr1a!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 424w, https://substackcdn.com/image/fetch/$s_!hr1a!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 848w, https://substackcdn.com/image/fetch/$s_!hr1a!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 1272w, https://substackcdn.com/image/fetch/$s_!hr1a!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!hr1a!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Where each study works in the software life cycle. Requirements, design, and deployment are not covered. Implementation: WatchPoint checks code while an AI coding agent writes it. Testing: TestAug and HateModerate write tests that probe how an AI model behaves. Maintenance: FewVuln and PatchHolmes read security reports, then find the code change that fixed them.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Where each study works in the software life cycle. Requirements, design, and deployment are not covered. Implementation: WatchPoint checks code while an AI coding agent writes it. Testing: TestAug and HateModerate write tests that probe how an AI model behaves. Maintenance: FewVuln and PatchHolmes read security reports, then find the code change that fixed them." title="Where each study works in the software life cycle. Requirements, design, and deployment are not covered. Implementation: WatchPoint checks code while an AI coding agent writes it. Testing: TestAug and HateModerate write tests that probe how an AI model behaves. Maintenance: FewVuln and PatchHolmes read security reports, then find the code change that fixed them." srcset="https://substackcdn.com/image/fetch/$s_!hr1a!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 424w, https://substackcdn.com/image/fetch/$s_!hr1a!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 848w, https://substackcdn.com/image/fetch/$s_!hr1a!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 1272w, https://substackcdn.com/image/fetch/$s_!hr1a!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfb3ccb9-b058-4a40-9efb-12eb088ecde6_1456x1372.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><h3>2.1 Step 1, Reading: FewVuln</h3><p>Each publicly disclosed vulnerability gets a CVE entry that describes the flaw in free text. Before a team can patch, someone must extract which software and which versions are affected. FewVuln fine-tunes a pre-trained language model for this extraction. It was presented at MLHat 2021, a workshop on machine learning for security held with KDD 2021, a major data mining conference.</p><p>Trained on 576 labeled sentences, FewVuln reaches an F1 score of 0.97 (on a 0-to-1 scale) for software names. The best earlier system reached 0.98 with 5,758 labeled sentences. In other words, FewVuln needs about 90% less labeling work, at a small cost in accuracy. <a href="https://arxiv.org/abs/2108.06590">Read the FewVuln paper</a>. <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-fewvuln-security-reports-fewer-labels">Read the full post on FewVuln</a>.</p><h3>2.2 Step 2, Writing Tests: TestAug and HateModerate</h3><p>A model can score well on a standard test set and still fail simple cases. In the TestAug study, text classifiers that were wrong on only 5.7% to 12.6% of a standard test set failed 14.2% to 49.8% of targeted tests, such as sentences with negation. Teams usually write targeted tests by hand, and TestAug lets GPT-3 write them instead. People wrote 29 test templates, and GPT-3 produced the other 1,441, about 98% of the 1,470 in total. TestAug was published at COLING 2022, the International Conference on Computational Linguistics. <a href="https://aclanthology.org/2022.coling-1.307/">Read the TestAug paper</a>.</p><p>HateModerate applies the same idea to content moderation. It appeared in Findings of NAACL 2024, a peer-reviewed volume of the North American conference of the Association for Computational Linguistics (ACL). The main results:</p><ul><li><p>HateModerate matches 7,704 test cases to the 41 hate speech policies that Facebook had published by November 2022.</p></li><li><p>Training a hate speech detector on these cases cut its failure rate on policy tests it had not seen from 0.423 to 0.275, where lower is better.</p></li><li><p>On HateCheck, an independent test set, the same training cut the failure rate from 0.365 to 0.199.</p></li></ul><p><a href="https://aclanthology.org/2024.findings-naacl.172/">Read the HateModerate paper</a>. <a href="https://newsletter.guanqun-yang.com/p/2026-10-16-testaug-hatemoderate-ai-written-tests">Read the full post on TestAug and HateModerate</a>.</p><h3>2.3 Step 3, Acting With Tools: PatchHolmes</h3><p>Security databases, and the alert tools built on them, need to know which commit fixed each vulnerability, for example to tell users which version is safe. That link is often missing. A 2024 study, <a href="https://arxiv.org/abs/2311.01532">VFCFinder</a>, found no link to the fixing commit for 63% of the advisories in the GitHub Advisory Database.</p><p>PatchHolmes gives a language model four tools that can read a code repository but not change it. For each vulnerability, it works in three steps:</p><ol><li><p>A search engine picks 100 candidate commits from the repository.</p></li><li><p>The agent compares the candidates side by side and opens 3 to 10 of them.</p></li><li><p>The agent names one commit as the fix.</p></li></ol><p>On a benchmark of 809 vulnerabilities, PatchHolmes ranks the true fix first for 60.0% of them, against 34.6% for <a href="https://arxiv.org/abs/2602.12500">Favia</a>, the strongest earlier method. Each search costs about USD 0.007 at the public price of the model it uses. PatchHolmes has been accepted at AACL-IJCNLP 2026, a joint Asia-Pacific conference on natural language processing. <a href="https://arxiv.org/abs/2609.38807">Read the PatchHolmes paper</a>. <a href="https://newsletter.guanqun-yang.com/p/2026-10-17-patchholmes-ai-agent-finds-vulnerability-fixes">Read the full post on PatchHolmes</a>.</p><h3>2.4 Step 4, Working With Another Agent: WatchPoint</h3><p>As of 2026, AI coding agents can build whole web applications. When their code fails a test, they usually see only the error message. A human developer would open the application, click its buttons, and inspect it. WatchPoint simulates that developer. When a task fails:</p><ol><li><p>A second language model writes small diagnostic scripts for the failed task.</p></li><li><p>It runs the scripts against the live application and reports what it observes.</p></li><li><p>The coding agent uses the report to repair its work.</p></li></ol><p>WatchPoint is a 2026 arXiv preprint. Its evaluation uses <a href="https://arxiv.org/abs/2505.07473">Web-Bench</a>, a benchmark of 50 multi-file web projects:</p><ul><li><p>Of the failed tasks that WatchPoint diagnosed, 57.6% passed on the retry (38 of 66).</p></li><li><p>When the coding agent saw only the test errors, 45.1% passed on the retry.</p></li><li><p>In a small study on a different set of tasks, two human testers fixed 12 of 22 tasks (54.5%), so the comparison with people is preliminary.</p></li></ul><p><a href="https://arxiv.org/abs/2609.26204">Read the WatchPoint paper</a>. <a href="https://newsletter.guanqun-yang.com/p/2026-10-18-watchpoint-measured-feedback-ai-coding-agents">Read the full post on WatchPoint</a>.</p><h2>3. What It Means for Your Team</h2><p>The four studies suggest two lessons for any team that uses AI models. The figure after the lessons shows the numbers behind both.</p><h3>3.1 The Design Around a Model Matters More Than Its Size</h3><p>In PatchHolmes, on the benchmark of 809 vulnerabilities:</p><ul><li><p>The search engine alone ranked the true fix first for 32.6% of vulnerabilities.</p></li><li><p>Adding the agent raised that share to 60.0%.</p></li><li><p>Switching to a model with about 7 times fewer active parameters lowered the share by only 0.86 percentage points, to 59.1%.</p></li></ul><p>Before paying for a bigger model, check whether a better workflow around the current model gives the same gain.</p><h3>3.2 A Helper&#8217;s Value Depends on the Model That Uses It</h3><p>In WatchPoint, the measure is the share of coding tasks passed within two attempts:</p><ul><li><p>With WatchPoint&#8217;s feedback, a weaker coding model rose from 26.6% to 28.1%.</p></li><li><p>With the same kind of feedback, a stronger coding model fell from 44.5% to 37.7%.</p></li><li><p>The drop for the stronger model is within the normal variation between repeated runs, so it needs more tests before anyone draws a firm conclusion.</p></li></ul><p>Measure a helper on your own model before enabling it everywhere.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RLwn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 424w, https://substackcdn.com/image/fetch/$s_!RLwn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 848w, https://substackcdn.com/image/fetch/$s_!RLwn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 1272w, https://substackcdn.com/image/fetch/$s_!RLwn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!RLwn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two lessons. Lesson 1, from PatchHolmes: the search engine alone ranks the true fix first for 32.6% of vulnerabilities, the agent with a small model for 59.1%, and the agent with a large model for 60.0%. Lesson 2, from WatchPoint: a weaker coding model passes 26.6% of tasks within two attempts alone and 28.1% with WatchPoint feedback, while a stronger coding model passes 44.5% alone and 37.7% with feedback.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two lessons. Lesson 1, from PatchHolmes: the search engine alone ranks the true fix first for 32.6% of vulnerabilities, the agent with a small model for 59.1%, and the agent with a large model for 60.0%. Lesson 2, from WatchPoint: a weaker coding model passes 26.6% of tasks within two attempts alone and 28.1% with WatchPoint feedback, while a stronger coding model passes 44.5% alone and 37.7% with feedback." title="Two lessons. Lesson 1, from PatchHolmes: the search engine alone ranks the true fix first for 32.6% of vulnerabilities, the agent with a small model for 59.1%, and the agent with a large model for 60.0%. Lesson 2, from WatchPoint: a weaker coding model passes 26.6% of tasks within two attempts alone and 28.1% with WatchPoint feedback, while a stronger coding model passes 44.5% alone and 37.7% with feedback." srcset="https://substackcdn.com/image/fetch/$s_!RLwn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 424w, https://substackcdn.com/image/fetch/$s_!RLwn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 848w, https://substackcdn.com/image/fetch/$s_!RLwn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 1272w, https://substackcdn.com/image/fetch/$s_!RLwn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e2b67bc-dc65-4cab-ac1f-cd9f2139d45c_1456x2020.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><h2>4. Research Beyond the Dissertation</h2><p>Later papers follow the same approach. Each one tests a small, cheap change carefully against a costlier one:</p><ul><li><p><a href="https://arxiv.org/abs/2608.10240">Sequential Modality Dropout (SMD)</a>, accepted at CIKM 2026, an international conference on information and knowledge management, keeps recommender systems accurate when product photos or descriptions are missing.</p></li><li><p><a href="https://arxiv.org/abs/2608.10247">DualSpectralCF</a>, also accepted at CIKM 2026, lets a recommender use dislikes without costly training. <a href="https://newsletter.guanqun-yang.com/p/2026-10-19-recommenders-missing-content-dislikes">Read the full post on SMD and DualSpectralCF</a>.</p></li><li><p><a href="https://arxiv.org/abs/2609.38822">SkillSeek</a>, accepted at AACL-IJCNLP 2026, shows that plain search selects skills for AI agents about as well as a search loop run by a large language model, at a much lower cost.</p></li></ul><h2>5. Questions Readers Ask</h2><p><strong>Can language models replace security analysts and test engineers?</strong></p><p>They can do the repetitive parts of the work. In these studies, people still:</p><ul><li><p>define each task,</p></li><li><p>write a few hundred labels or a few dozen test templates,</p></li><li><p>check the results.</p></li></ul><p>PatchHolmes, for example, cannot find a fix that its search engine leaves out of the top 100 candidates, which happened for 12.4% of vulnerabilities.</p><p><strong>Do these methods need the largest and most expensive models?</strong></p><p>The studies suggest they do not. PatchHolmes lost less than 1 percentage point with a model about 7 times smaller, and each search costs about USD 0.007. The gains came mainly from how the model was used.</p><p><strong>Where can I read the full dissertation?</strong></p><p>The full text is free on <a href="https://guanqun-yang.com/paper/dissertation.pdf">guanqun-yang.com</a>. The official record is in <a href="https://www.proquest.com/docview/3386606282">ProQuest</a>, the main database of dissertations from US universities.</p><h2>6. Read and Cite the Work</h2><p>The dissertation:</p><ul><li><p>Guanqun Yang. <em>Scalable Software Testing and Analysis with NLP Methods.</em> Doctoral dissertation, Stevens Institute of Technology, 2026. <a href="https://guanqun-yang.com/paper/dissertation.pdf">Full text (PDF)</a>, <a href="https://www.proquest.com/docview/3386606282">ProQuest record</a>.</p></li></ul><p>A two-page summary of the research plan, from the doctoral consortium of AAAI-25, the 2025 conference of the Association for the Advancement of Artificial Intelligence:</p><ul><li><p>Guanqun Yang. &#8220;Advancing Intelligent Software Development and Trustworthy Models Through the Synergy of Software Engineering and LLMs.&#8221; <em>Proceedings of the AAAI Conference on Artificial Intelligence</em> 39(28), 2025. <a href="https://doi.org/10.1609/aaai.v39i28.35234">Publisher page</a>.</p></li></ul><p>The papers behind each step:</p><ol><li><p>&#8220;Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-Trained Language Models&#8221; (FewVuln), MLHat 2021. <a href="https://arxiv.org/abs/2108.06590">arXiv</a>, <a href="https://doi.org/10.1007/978-3-030-87839-9_3">Springer</a>.</p></li><li><p>&#8220;TestAug: A Framework for Augmenting Capability-based NLP Tests&#8221;, COLING 2022. <a href="https://aclanthology.org/2022.coling-1.307/">ACL Anthology</a>.</p></li><li><p>&#8220;HateModerate: Testing Hate Speech Detectors against Content Moderation Policies&#8221;, Findings of NAACL 2024. <a href="https://aclanthology.org/2024.findings-naacl.172/">ACL Anthology</a>.</p></li><li><p>&#8220;PatchHolmes: Agentic Patch Retrieval via Listwise Selection&#8221;, AACL-IJCNLP 2026. <a href="https://arxiv.org/abs/2609.38807">arXiv</a>.</p></li><li><p>&#8220;WatchPoint: Executable User Feedback for Real-World Agentic Web Development&#8221;, arXiv preprint, 2026. <a href="https://arxiv.org/abs/2609.26204">arXiv</a>.</p></li></ol><p>To cite the dissertation:</p><pre><code>@phdthesis{yang2026scalable,
  author = {Guanqun Yang},
  title  = {Scalable Software Testing and Analysis with NLP Methods},
  school = {Stevens Institute of Technology},
  year   = {2026},
  url    = {https://guanqun-yang.com/paper/dissertation.pdf}
}
</code></pre><div><hr></div><p><em>Guanqun Yang, LLM Researcher. More work at <a href="https://guanqun-yang.com">guanqun-yang.com</a> and on <a href="https://scholar.google.com/citations?user=4XUIHW0AAAAJ">Google Scholar</a>.</em></p>]]></content:encoded></item></channel></rss>