Second Thought: Reasoning in Parallel as LLM Agents Act and Observe

The Core Insight

Large language model (LLM) agents using the ReAct framework—which cycles through thinking, acting, and observing—waste reasoning capacity. While the agent waits for environmental feedback after taking an action, its reasoning engine sits idle. Second Thought recovers this dead time by spawning parallel reasoning threads during the wait, then integrating those thoughts back when the observation arrives. The result: fewer turns to solve problems and up to 43% less sequential decoding on the main reasoning path, with no drop in solution quality.

How It Works

The ReAct loop normally runs: generate a Thought → generate an Action → send it to the environment → wait for Observation → repeat. That wait period is pure dead time for reasoning.

Second Thought intercepts the moment a Thought concludes and forks four auxiliary reasoning branches. These branches decode in parallel with the main loop while the agent awaits the environment's response. When the observation arrives, Second Thought merges the auxiliary thoughts back into context, enriching what the agent knows before the next reasoning step. All this happens off the main sequential decoding path—the expensive bottleneck for LLMs. The framework requires no retraining; it's a pure inference-time technique.

Think of it as the agent thinking ahead while waiting for an answer, then incorporating those preparatory thoughts when it's time to act again.

What Changed

The authors tested Second Thought on three agentic benchmarks using three different reasoning LLMs. Across all nine (model, benchmark) combinations:

• Turn count dropped consistently—the agent needs fewer steps to reach a solution.
• Sequential decoding fell by up to 43% in six of nine cases (averaging ~20% reduction where it applied); in one case it barely moved.
• Pass@1 (first-attempt success rate) held steady—seven of nine pairs showed no significant change. The other two improved by +12.4 and +10.2 percentage points. No degradation.

Against a fair baseline that allocates the same compute budget to the main reasoning thread instead of auxiliary branches, Second Thought still won: it achieved higher success rates while performing 1.3 to 3.2× less sequential decoding, across all four applicable settings.

Why It Matters

LLM inference is bottlenecked by the sequential generation of tokens—you can't parallelize computing one token until the previous one exists. Second Thought doesn't break that constraint; instead, it sidesteps it by moving extra reasoning into time that's already wasted. The gains are real: fewer turns means faster agent execution in wall-clock time, and less main-thread decoding means lower latency for users.

For practitioners, this is a free upgrade. No retraining, no architectural changes—just a smarter scheduling strategy at inference. The consistency across different models and benchmarks suggests the insight generalizes: any ReAct-style agent will find idle windows, and any model can use that time productively.