When an AI coding agent’s code fails a test, the agent usually sees only the test’s error message. A human developer would open the app, click around, and check what the page actually does. WatchPoint, a system I led, gives the coding agent those measurements. With them, a coding agent fixed 57.6% of its failed tasks on its one retry, against 45.1% with test errors alone.
The same feedback did not help a stronger coding agent. Its share of tasks passed within two attempts fell from 44.5% to 37.7%. WatchPoint is an arXiv preprint posted on 10 August 2026 by Guanqun Yang, Wei Yang of the University of Texas at Dallas, and Xueqing Liu of Stevens Institute of Technology. It is the fourth and last step of my dissertation, where a language model works alongside another agent.

In this post
Why Test Errors Are Poor Feedback for Coding Agents
How WatchPoint Measures the Running App
2.1 Two Scripts Run Once After Each Failure
2.2 Raw Measurements Go Straight to the Coding Agent
What the Results Show
3.1 More Failed Tasks Passed on the Retry
3.2 The Gains Came From State-Management Projects
3.3 The Stronger Coding Model Did Worse With the Feedback
3.4 A Model That Interpreted the Results Invented Errors
How WatchPoint Compared With Two Human Testers
Where the Results Stop
What It Means for Your Team
Questions Readers Ask
Read and Cite the Paper
1. Why Test Errors Are Poor Feedback for Coding Agents
A coding agent is a language model that runs tools, such as a terminal and a file editor, in a loop until a task is done. Some teams split software work across several such agents, with one that plans, one that writes code, and one that tests. The messages between them often go wrong. Across 1,642 annotated runs of multi-agent systems, Cemri and colleagues traced 32.3% of the failures to agents that misunderstood each other.
According to the WatchPoint paper, the feedback that coding agents usually receive takes four main forms:
the raw error message from a failed test
a screenshot, sometimes scored by a vision model
a score or opinion from another language model, often called LLM-as-a-judge
a correction written in plain language
The authors argue that few of these approaches use the live application the way a developer does, and that most were tested on single-page demo sites.
WatchPoint was tested on Web-Bench, a 2025 benchmark of 50 real web projects with 20 tasks each, 1,000 tasks in total. According to its authors, engineers with 5 to 10 years of experience designed the projects, and one project takes a senior engineer 4 to 8 hours. Each task builds on the earlier ones and is checked by end-to-end tests, which open the real app in a browser and check what it does. The best agent in the WatchPoint paper passes 25.0% of tasks on the first attempt.
2. How WatchPoint Measures the Running App
2.1 Two Scripts Run Once After Each Failure
WatchPoint works with a coding agent built on OpenHands, an open-source platform for software agents. The paper calls WatchPoint’s role a simulated user, a model that plays the person testing the app. The loop has five steps:
The coding agent writes the next feature, using up to 30 model calls.
Web-Bench’s end-to-end tests run. If they pass, the next task begins.
If they fail, a browser agent and a terminal agent each write one script with a single model call. Both scripts run once, in parallel.
The coding agent retries once, in the same conversation, with the test errors plus one line per finding from the scripts.
If the retry also fails, the rest of the project stops, because later tasks depend on this one.
The two scripts see different failures. The browser script uses Playwright, Microsoft’s library for controlling a browser from code, to open the page, click, fill forms, and read styles, text, and page structure. The terminal script reads files, starts the server and sends it requests, and runs a type check. As the authors note, a browser cannot detect a missing software dependency, and a terminal cannot detect a broken layout.
WatchPoint switches on only for three kinds of project: plain web-standard code, user interface frameworks such as React, and state-management libraries such as Redux. It decides from each project’s list of software dependencies, the same signal that hosting services such as Vercel and Netlify use to detect frameworks. Across the 50 projects, it made the right on-or-off choice 92% of the time. Each diagnosis takes about 15 seconds, and the diagnostic calls added less than 2% to the total API cost.

2.2 Raw Measurements Go Straight to the Coding Agent
Each script prints one line per finding, with a name and a measured value. WatchPoint passes these CHECK lines to the coding agent unchanged, and no model summarizes them. The paper states the rule as “the LLM decides what to measure but never interprets what was measured.”
Here is a logged case from a project that builds a code editor with syntax highlighting. The first attempt failed this test:
Error: expect(locator).toHaveText(expected)
Locator: locator('#editor')
Expected: "foo AND bar"
Received: "fooANDbar "
The browser script then reported these lines, among others:
CHECK editor_exists: True
CHECK after_fill_text: foo AND bar
CHECK highlight_function: 3
CHECK identifier_color: #0000ff
CHECK operator_color: #00ff00
On the retry, the coding agent compared its code with what the page showed. It noticed that it had built the highlighted text as one HTML string, which dropped the spaces between words. It switched to creating a separate page element for each word, and the task passed.
3. What the Results Show
3.1 More Failed Tasks Passed on the Retry
The main measure is the recovery rate: of the tasks that failed the first attempt and got a retry, the share that passed on the retry. Higher is better. The coding model here is MiniMax M2.7, and GLM-5 writes the diagnostic scripts.
With WatchPoint, 38 of 66 retried tasks passed, a recovery rate of 57.6%.
With test errors alone, the same coding model recovered 45.1% on the same 28 projects.
On the 20 projects where WatchPoint never runs, recovery stayed the same, 45.9% against 46.2%.
At least one script produced a valid measurement for 59 of the 66 retried tasks.
3.2 The Gains Came From State-Management Projects
A second measure is Pass@2, the share of all tasks passed within two attempts, where higher is better. Because a twice-failed task stops its project, one recovered task can let many later tasks run. In one project, a single recovered early task allowed 8 later tasks to run.
In state-management projects, Pass@2 rose from 11.2% to 18.8%.
Across all 28 projects where WatchPoint runs, it rose from 21.8% to 24.5%.
Over all 1,000 tasks, including projects where WatchPoint never runs, it rose from 26.6% to 28.1%.
3.3 The Stronger Coding Model Did Worse With the Feedback
The authors repeated the test with GLM-5 as the coding model, which passes far more tasks on its own. Pass@2 over all 1,000 tasks, where higher is better:
MiniMax M2.7 reached 26.6% alone, 28.1% with GLM-5 as the tester, and 27.9% with GPT-5.4 as the tester.
GLM-5 reached 44.5% alone and 37.7% with GPT-5.4 as the tester.
The authors caution that the drop for GLM-5 overlaps the normal variation between identical runs. Their reading is that the feedback adds clear value when the coding model is weak. A model that can already fix its own mistakes from test errors may get only noise from the extra input. A stronger tester did not help more either, since GPT-5.4 and GLM-5 gave the weaker coder similar results.

3.4 A Model That Interpreted the Results Invented Errors
An earlier version of WatchPoint ran a fixed browser script and asked GLM-5 to interpret its output for the coding agent. On a calculator project, the interpreter reported a page titled “Blog” and a 504 server error, neither of which existed. The coding agent spent more than 40 steps chasing those invented problems, and tasks passed on the first attempt fell from 11 of 20 to 4 of 20. Passing the raw CHECK lines removed the failure.

4. How WatchPoint Compared With Two Human Testers
The authors also ran a small user study. Two graduate students with web development coursework took WatchPoint’s place. When a task failed, they opened the app in a browser and typed free-form feedback for the coding agent. They covered five small projects in plain HTML, CSS, and JavaScript, in sessions of about two hours each.
Tester A recovered 4 of 9 failed tasks, 44%.
Tester B recovered 8 of 13, 62%.
WatchPoint recovered 57.6%, measured on a different and larger set of tasks.
Some human feedback was vague, such as “I think it’s fine”. The authors call the comparison preliminary evidence, given two testers and the easiest kind of project for browser checks.
5. Where the Results Stop
The results have these limits:
They come from one benchmark, Web-Bench, with one run per setting. Between two identical runs of the same coding model, 113 of 980 tasks changed outcome. The authors estimate that totals can differ by about 60 of the 1,000 tasks from chance alone.
The user study had two testers and five small projects.
The scripts can mislead. In one project, the terminal script checked the wrong network ports, and the coding agent spent 30 steps changing a build file. All 20 tasks in that project were skipped.
For 7 of the 66 retried tasks, the scripts produced no usable measurement.
WatchPoint is an arXiv preprint, posted on 10 August 2026.
6. What It Means for Your Team
The study suggests five practices for teams that run AI coding agents:
Give coding agents facts measured from the running app, next to the test errors.
Let a model write the measurement script, and pass its raw output to the coding agent. Do not let a second model summarize it.
Switch the diagnostics on only for project types where they help.
Test with and without the feedback on your own coding model. A model that already fixes most failures from test errors may do worse with more input.
Run each comparison more than once. Identical runs here differed by up to about 60 of 1,000 tasks.
7. Questions Readers Ask
How can an AI coding agent test the web app it just wrote?
It can run a short script against the live app, in a browser and in a terminal, and read the measured results. In WatchPoint, a second model writes those scripts after each failed test. The coding agent then fixed 57.6% of its failed tasks on the retry, against 45.1% with test errors alone.
Does more feedback always make an AI coding agent better?
It did not in this study. With a stronger coding model, GLM-5, the share of tasks passed within two attempts fell from 44.5% to 37.7% when WatchPoint’s feedback was added. The authors note that the drop overlaps the normal variation between identical runs.
Can an automated tester replace a human quality assurance tester for web apps?
The evidence so far is preliminary. In a small study, two human testers recovered 44% and 62% of failed tasks. WatchPoint recovered 57.6% on a different, larger set of tasks.
How much does it cost to add diagnostics like WatchPoint’s to a coding agent?
In the paper, the diagnostics added less than 2% to the total API cost and about 15 seconds per failed task.
What is Web-Bench?
Web-Bench is a 2025 benchmark of 50 real web projects with 20 dependent tasks each, checked by end-to-end tests in a real browser. The best agent in the WatchPoint paper passes 25.0% of tasks on the first attempt.
8. Read and Cite the Paper
Guanqun Yang, Wei Yang, and Xueqing Liu. “WatchPoint: Executable User Feedback for Real-World Agentic Web Development.” arXiv preprint, 2026. arXiv.
WatchPoint is the fourth of four steps in my dissertation. The overview post explains how the steps fit together. The earlier steps each have a post: FewVuln on reading vulnerability reports, TestAug and HateModerate on writing tests, and PatchHolmes on finding vulnerability fixes.
To cite the paper:
@misc{yang2026watchpointexecutableuserfeedback,
title = {WatchPoint: Executable User Feedback for Real-World Agentic Web Development},
author = {Guanqun Yang and Wei Yang and Xueqing Liu},
year = {2026},
eprint = {2609.26204},
archivePrefix = {arXiv},
primaryClass = {cs.SE},
url = {https://arxiv.org/abs/2609.26204}
}
Guanqun Yang, LLM Researcher. More work at guanqun-yang.com and on Google Scholar.

