Goal. A vision-based verification fallback for click_index when DOM-diff has nothing to read (canvas, WebGL, + "
The new DOM-diff verifier (#vision-verify) covers most click checks but is silent on pages where state changes don't reflect into the DOM. A "screenshot before / after + ask the model" fallback would fill that gap.
Sketch.
- Take a screenshot of the bounding rect before and after a click.
- If
+ "domChanged" + is false but the click target is on a canvas-y region, attach both screenshots to the next model turn with an explicit "did this click do anything?" prompt.
- Cost gate: only do this when DOM-diff is silent AND target is non-DOM-y.
Open questions. Token cost. Whether to ship as a separate tool or fold into + "click_index" + .
Goal. A vision-based verification fallback for click_index when DOM-diff has nothing to read (canvas, WebGL,
+ "The new DOM-diff verifier (#vision-verify) covers most click checks but is silent on pages where state changes don't reflect into the DOM. A "screenshot before / after + ask the model" fallback would fill that gap.
Sketch.
+ "domChanged" +is false but the click target is on a canvas-y region, attach both screenshots to the next model turn with an explicit "did this click do anything?" prompt.Open questions. Token cost. Whether to ship as a separate tool or fold into
+ "click_index" +.