SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

The Problem: Coding Agents Fail When Users Edit Code

Existing benchmarks for coding agents assume a solo workspace—the agent works alone toward a goal. Real software development isn't like that. Users inspect, modify, and sometimes contradict the agent's work in progress. Current evaluation frameworks don't test this. The authors ask a simple but important question: when a user edits code that conflicts with what an agent is trying to do, does the agent notice and adapt?

The answer is no, not well enough.

How SWE-Touch Works

The authors built a framework that deliberately creates friction. They inject "Counter-Edits"—realistic code changes that obstruct task completion—into an agent's workflow while it's running.

The process has three steps. First, they mine task-critical code regions by analyzing multiple repair trajectories (different solution paths) on existing benchmarks. This identifies which code segments matter most for solving each task. Second, a separate "User Patch Generator" creates plausible edits to those segments—changes that conflict with the agent's goal but look like something a real user might do. Third, when an agent reaches the relevant code during task execution, they inject the counter-edit alongside a contextual message explaining what the user did.

This simulates a shared workspace where the user has moved the goalposts mid-task.

The Results: State Awareness Is Missing

Testing nine coding models on SWE-bench Verified (a dataset of validated real GitHub issues), counter-edits reduced the average resolve rate by 7.7 percentage points. The effect persists on longer-horizon tasks from SWE-Bench Pro and DeepSWE. That's substantial—a real degradation in performance under realistic conditions.

Why do agents fail? Trajectory analysis reveals the core issue: agents lack workspace state awareness. They either retain conflicting code (ignoring the user's edit) or replace it without sufficiently re-inspecting the repository to understand what changed. They fail to run targeted tests to validate whether the revised code still works. Essentially, agents treat user edits as noise rather than signals that the task context has shifted.

Why This Matters

Current benchmarks let agents appear more capable than they are. A 7.7 percentage point drop may sound modest until you consider the baseline: these agents already succeed on a minority of tasks. A 7.7 point hit on top of an already-challenging benchmark is significant.

More fundamentally, the findings expose a gap between narrow task completion and real collaborative development. Strong autonomous performance—high accuracy on solo benchmarks—doesn't translate to the adaptability needed when humans are in the loop. Agents need three capabilities they mostly lack: detecting when the workspace has changed, reconciling conflicting edits with the original task goal, and verifying affected code through testing.

SWE-Touch creates a harder, more realistic evaluation protocol. It suggests future research should focus on building agents that monitor and respond to external changes rather than just executing a fixed plan from start to finish.