Chain-of-Experience for Continual LLM Improvement

The Central Finding

Large language models can improve dramatically during inference—not just at training time—by learning from their own mistakes through iterative feedback loops. A study across eight state-of-the-art models (GPT-4o, Gemini-2.5 Pro, Claude-4.5 Sonnet) shows that Chain-of-Experience (CoE), a method where models iteratively refine answers using self-feedback or environmental signals, achieves 5.6% overall accuracy gains and cuts API costs by 19%, outperforming single-shot inference across math, coding, and knowledge tasks.

How It Works

CoE operates like a human learning cycle. Instead of asking an LLM a question once and accepting its answer, the model is given a chance to reconsider based on feedback signals. These signals come in two flavors:

Self-feedback: The model reflects on its own reasoning. For instance, it might check its own math work or verify a coding solution's logic without external validation.

Environmental feedback: The model receives concrete signals—whether an answer is correct, or whether code passes automated test cases. This ground truth helps steer the model toward better solutions.

The model accumulates these "experiential traces" across multiple iterations, treating each round as a learning opportunity. Rather than generating one response and stopping, the system runs multiple refinement cycles, allowing the model to spot errors and adjust.

Why It Matters

The work challenges a fundamental assumption in LLM evaluation: that a model's capability is fixed at inference time. By showing models can improve through interaction, it reframes how we think about their utility. A model that generates better answers after two or three attempts—while still costing less in total API calls—is more valuable than metrics suggest.

The 5.6% overall gain is substantial for a technique requiring no retraining. More striking is the efficiency: iterative refinement yields higher accuracy per token consumed, meaning the model achieves better results without proportional cost increases. This matters in deployed systems where API costs scale with token volume.

The authors find that:

• Gains emerge early in iterations (most improvement in the first few rounds), not spread evenly across cycles.
• Stronger base models improve more—a model's ability to self-correct correlates with its raw capability.
• Models remain robust even under weak or misleading feedback, suggesting the mechanism isn't fragile.
• Different feedback types contribute to different aspects of improvement (self-feedback excels at logical refinement; correctness signals excel at pattern recognition).

Practical Implications

CoE doesn't require changing model weights or fine-tuning. It works with existing off-the-shelf LLMs today. The approach is modular: practitioners can combine feedback channels (self + correctness + test pass rates) to match their domain, and gains compound across channels.

For applications like code generation or mathematical problem-solving where ground truth is computable, environmental feedback unlocks systematic improvement. For open-ended tasks, self-feedback alone provides consistent wins. The authors' observation that most gains arrive in the first few iterations means practitioners don't need to iterate indefinitely—a practical constraint for real-world deployment.

This work demonstrates that current LLM benchmarks, which measure one-shot performance, may underestimate model capability in realistic interactive settings where feedback and refinement are possible.